reports on charter school performance were released this week. On
December 14, Harvard University economist Caroline Hoxby announced
results showing that charter school students are more likely to be
proficient in math and reading than students in the nearest
comparable public school.
And on December 15, the U.S. Department of Education's National
Center for Education Statistics (NCES) released a study showing
that charter school students in the aggregate lag behind
traditional public school students in fourth grade reading and
math, though these differences become statistically insignificant
when the numbers are broken down by race.
Media stories have
largely overlooked the Hoxby study. The New York Times, for
example, completely ignored her results in its December 16 article
"A Second Report Shows Charter School Students Not Performing as
Well as Other Students."
Instead, the first report to which the Times alludes is a
corroborating study by the American Federation of Teachers (AFT)
based on the same data as the NCES study.
solely on the NCES report and ignoring Hoxby's results paints a
distorted picture of charter school performance. NCES itself is
quick to warn that its pilot study of charter school students has a
number of shortcomings. Context is critical in comparing student
performance across different types of schools, and charter schools
in one state may radically differ from those in another. Thus,
state-based and even more localized comparisons of charter schools
and public schools, such as undertaken in the Hoxby study, are a
better way to analyze charter schools' performance.
What the NCES Study Says
According to the
new NCES study based on data from the National Assessment of
Educational Progress, students at traditional public schools
outperformed charter school students in math and reading. While
these differences tend to disappear when the results are broken
down by race, they are statistically significant among students
eligible for free and reduced price lunches and among students in
is quick to caution about interpretation of its results. "While
charter schools are similar to other public schools in many
respects, they differ in several important ways, including the
makeup of the student population and their location."
charter schools are unique because they are chosen, unlike
traditional public schools, which are usually assigned. Because of
this difference, NCES warns, "when comparing the performance of
charter and other public school students, it is important to
compare students who share a common characteristic."
these caveats, Dr. Peggy Carr, Associate Commissioner for
Assessment at NCES, said, "This pilot study cannot take into
account the varying purposes for which charter schools have been
created and the conditions under which they function in individual
Comparing the Two Studies
The NCES study
relies on averages of student's academic achievement at a sample of
charter schools from across the country. It compares these averages
with averages of academic achievement among traditional public
school students, also from across the country.
effectively neglects the major ways in which charter schools and
public schools differ. For example, some charter schools cater to
low-performing students who have languished in traditional public
schools. Additionally, many charter schools locate in low-income
and minority communities. As Manhattan Institute Senior Fellow Jay
P. Greene notes, "Because so many charter schools are specifically
targeted to struggling students, a large percentage of their
minority and poor students face obstacles greater than students of
similar demographics in regular public schools." Thus, charter schools
and traditional public schools often serve very different student
bodies, and comparing them using blanket averages is problematic at
In contrast, the
Hoxby study looks at virtually all students who attend elementary
charter schools and compares them to students who attend the
nearest equivalent public school, yielding "matched pairs" of
student bodies. In this way, Hoxby effectively addresses many of
the problems associated with the NCES's methodology.
And in contrast
with the NCES study's results, the Hoxby study shows that charter
school students are more likely to be proficient in math and
reading than students in the nearest comparable public school.
Overall, charter students' reading proficiency rates are 5.2
percent higher than those of their public school counterparts, and
their math proficiency rates are 3.2 percent higher.
Hoxby's study shows that charter school students' gains in academic
achievement, relative to their public-school counterparts, tend to
increase as their charter schools mature. In other words, charter
schools are pulling away from public schools in terms of
performance. For example, children at charter schools that have
been operating for 1 to 4 years are 2.5 percent more proficient in
reading than students in the nearest comparable public school. That
proficiency advantage increases to 5.2 percent for charter schools
open 5 to 8 years, and to 10.1 percent for charter schools
operating 9 to 11 years. When comparing charters with long-lived
traditional public schools, then, it is only reasonable to evaluate
charter schools after several years of operation. Many charter
schools included in the NCES study, however, had only been open for
a year or two.
schools in minority and high-poverty areas show even greater
achievement gains, relative to their public school counterparts,
than charter schools in general. For example, students at charter
schools in Hispanic areas have a 7.6 percent advantage in reading,
and charters in African-American areas have a 4.5 percent
advantage, compared to a 4.2 percent advantage for students in
charter schools in other areas. Similarly, students at charters in
high-poverty areas have a reading proficiency advantage of 6.5
percent, compared with a 2.6 percent advantage for students in
other charter schools. This suggests that charter schools have the
most promise in the areas that have the greatest need for
innovative educational reforms.
Which Study is Superior?
above, the NCES study suffers from several deficiencies that limit
its usefulness. At the December 15 press conference announcing the
NCES results, Darvin Winick from the National Assessment Governing
Board argued that the proper tool for analyzing charter school
results is state assessment data, rather than national aggregate
data of the sort used by NCES, because knowing something about
"common characteristics" is key.
Dr. Peggy Carr of
NCES went a step further. When asked about how the NCES study
compared to the Hoxby study, she was very clear: "Matching [i.e.,
the Hoxby methodology] is a more superior design."
results in the NCES study, as well as the related American
Federation of Teachers study that is based on NCES data, should be taken with
a grain of salt. Even Bella Rosenberg of the AFT was unable to
offer any substantial critique of the Hoxby study when asked.
Rather than address its methodology, she weakly responded, "No one
has been able to replicate her results."
Both the NCES and
Hoxby studies of charter schools provide information about the
academic achievement of students at traditional public schools and
at charter schools. On balance, however, the Hoxby study is
superior because it uses better methodology. Even the NCES's own
associate commissioner of assessment agrees that Hoxby's
methodology is more appropriate than the NCES's.
results, Congress and state legislators should continue to support
charter schools. Educating more than 1 million students nationwide,
charter schools represent America's greatest experiment in school
choice to date.
Marshall is Director of Domestic Policy Studies, and Kirk A.
Johnson, Ph.D., is Senior Policy Analyst in the Center for Data
Analysis, at the Heritage Foundation.
Jay P. Greene, "No Comparison" New
York Sun, August 19, 2004, Editorial & Opinion, pg. 9. This
op-ed is based on a more extensive research report that also uses a
matched-schools methodology to analyze the differences between
charter schools and traditional public schools. See Jay P. Greene,
Greg Forster, and Marcus Winters, "Apples to Apples: An Evaluation of
Charter Schools Serving General Student Populations," Manhattan
Institute Education Research Report No. 1, July 2003