A recent study published by the National Center for Education
Statistics (NCES) contains some surprising results based on a
snapshot of student achievement data. According to the study,
public school students are performing better than private school
students in fourth grade mathematics and at the same level as
private school students in fourth grade reading and eighth
grade math. Indeed, the report says that private school students
have an advantage over public school students only in eighth
grade reading. These results should be handled very
Policymakers and journalists need to know that the NCES findings
that public schools outperform private schools employ significantly
limited data. Some commentators on the NCES report appear to
believe that this study describes causal relationships-that public
school attendance causes better student achievement and that
private school attendance causes students to have lower math
and reading achievement.
The NCES study analyzes the 2003 National Assessment of
Educational Progress (NAEP) data that, due to a major limitation,
are ill-suited for making any causal inferences. The NAEP data
assess achievement only at one point in time, providing a
snapshot of how American students are performing in math and
reading at that specific time. The NAEP data are not suitable for
evaluating the effectiveness of private or public school attendance
in raising academic achievement. In fact, the NCES authors
explicitly warn against this in two sections of the report that are
appropriately titled "Cautions in Interpretation."
Furthermore, various individuals and organizations have
used the report to discredit private school voucher programs,
asserting that government education funds should be used only
to fund public education, where they can be put to better use.
However, the results of several more sophisticated and
conclusive studies of voucher program effectiveness overwhelmingly
point toward greater math and reading achievement for students who
attend private schools through voucher programs.
As noted, some commentators appear to believe that the NCES
study describes causal relationships-that public school
attendance causes better student achievement and private
school attendance causes students to have lower math and
reading achievement. This is surprising because the authors of the
report explicitly warn against causal interpretations. They
also acknowledge that comparisons of public and private schools
(which amounts essentially to comparing apples to oranges) may
not be useful because of the differences that exist between and
within the public and private school populations.
The NCES report analyzes the 2003 NAEP data, a nationally
representative assessment of academic performance in math, reading,
and science (among other things). Researchers frequently use the
NAEP data to gauge student academic performance in the United
States, even though the cross-sectional nature of these data limits
what these researchers can do. Analysis of NAEP data can only be
used to generally describe the characteristics of students who took
the NAEP tests. These data are not suitable for use in
evaluating the effectiveness of public and private schooling over
In fact, the NCES study would not even meet the standards of
effective research set forth by the Institute of Education
Sciences-the very people responsible for releasing the study.
The Institute of Education Sciences, which is part of the
Department of Education, established the What Works
Clearinghouse (WWC) in 2002 to provide researchers,
policymakers, and the general public with standards by which
to evaluate research on educational interventions. The WWC
evidence standards require that research studies include either
random assignment or a pretest measure for the research results to
be considered effective at establishing causality.
The NAEP assessments do not meet either of these criteria. The
students who take the NAEP tests each year are not randomly
assigned into public or private schools. In addition, the NAEP does
not sample the same students or schools during each successive
administration. This means that the NAEP data do not include any
pretest or baseline measures that can be used to adjust for
differences between the public and private school populations.
The best method to draw causal inferences from research is to
collect data as part of a random study. In this type of study,
research participants are randomly assigned to either an
experimental group or a control group. The experimental group
receives some treatment-their environment is changed or manipulated
in some manner. The control group, on the other hand, is not
exposed to anything new or different and serves as a source of
comparison for the experimental group.
Random assignment helps to ensure that the experimental and
control groups contain similar mixes of participants and are
generally representative of the larger population to which
they belong. The variable of interest is measured at regular
intervals throughout the duration of the study (including
at the beginning and end) to see whether there is any difference in
outcomes between the experimental and control groups. In this
case, causal inferences can be drawn because any difference in
outcomes between the experimental and control groups can be
attributed to the treatment experienced by the experimental
Education studies that include measurement over time are much
more useful for drawing conclusions about school quality.
Education researchers have repeatedly pointed out that a
student's low test score at a single specific time may indicate
only that she is not a good student or that some external
circumstance influenced her bad performance on the test that day.
However, if that student's test scores rise over time, it indicates
that she is being well-served and well-educated by the school that
she is attending.
The NAEP data do not sample the same students or schools each
year. Thus, researchers cannot study how achievement has changed
over time for either individual students or a cohort of students
attending the same school, whether public or private. As a
result, the NAEP data are not well-suited to establishing whether a
specific math or reading achievement outcome is associated with
attending either a private or public school.
Evidence on Voucher Programs
Voucher programs specifically target the academic needs of
low-income (frequently minority) students, who often live and go to
school in high-poverty areas. These children are frequently stuck
in persistently low-performing public schools that are not meeting
their educational needs.
Currently, six states and the District of Columbia offer
government-sponsored voucher programs to their students. In
addition, several other states offer privately funded voucher
programs. These programs allocate government or
private funds directly to parents who then use that money to pay
tuition costs at a private school of their choosing. Voucher
programs are a powerful school choice tool because they give
low-income families the opportunity to send their children to
private, tuition-based schools that they could not afford without
Several random assignment studies have been conducted on the
effects of private school voucher programs on math and reading
achievement. These studies include evaluations of public and
privately funded voucher programs in Wisconsin, North Carolina,
Ohio, New York, and the District of
Columbia. The studies compared the achievement
outcomes of students who were randomly awarded vouchers to attend a
private school to the outcomes of those students who did not
receive vouchers and remained in their regular public schools. They
all reached the same conclusion: Students who received vouchers
experienced greater math and/or reading achievement gains than did
the students who remained in the public school system.
Since these studies included random design and measured
differences in achievement scores over time, it can be said
with confidence that attending a private school resulted in
significantly greater math and/or reading achievement gains for the
students who were awarded vouchers.
The results of a recent NCES study that finds that public school
students are performing better then private school students should
be interpreted cautiously. The NAEP data used in the study
provide a snapshot of student achievement at the time the NAEP
tests were administered. As a result, the NAEP data can be used
only to describe the characteristics of the student population in
the United States at the time when students took the NAEP
The NAEP data are certainly not suitable for establishing
whether a specific math or reading achievement outcome is
associated with attending either a private or public school.
Despite this fact, the results of the NCES study are being
interpreted inappropriately to imply that voucher programs, which
include private schools, are a bad idea.
The research literature that addresses the effectiveness of
school voucher programs in raising math and reading achievement,
based on more sophisticated methodology, is much more
convincing and conclusive. Students who attend a private
school through a voucher program experience greater gains in math
and reading than do their public school counterparts.
Voucher programs are a powerful school choice tool that results
in many positive outcomes for the students who participate in them.
More families should have the opportunity and flexibility to choose
the most effective school for their children to achieve
Shanea Watkins is Policy Analyst in
Empirical Studies in the Center for Data Analysis at The Heritage
 Henry Braun, Frank Jenkins, and Wendy
Grigg, "Comparing Private Schools and Public Schools Using
Hierarchical Linear Modeling," NCES 2006-461, U.S. Department of
Education, National Center for Education Statistics, Institute of
Education Sciences, July 2006, at /static/reportimages/138647316ED8FB156FBA7DA4338A6FEC.pdf
(August 31, 2006).
 National Education Association,
"Education Department Reports Private Schools on Par with Public
Schools," July 20, 2006, at
www.nea.org/newsreleases/2006/nr060720.html (August 31,
2006); Lois Romano, "GOP Unveils School Voucher Plan," The
Washington Post, July 19, 2006, p. A17, at http://www.washingtonpost.com/wpdyn/
content/article/2006/07/18/AR2006071801305.html (August 31,
2006); Diana Jean Schemo, "Public Schools Perform Near Private Ones
in Study," The New York Times, July 15, 2006, p. A1; Zachary
M. Seward, "Long-Delayed Education Study Casts Doubt on Value of
Vouchers," The Wall Street Journal, July 15, 2006; Edward J.
McElroy, President, American Federation of Teachers, "On the
Introduction of School Voucher Legislation," July 18, 2006, at
(August 31, 2006); Edward J. McElroy, "On Vouchers and Private vs.
Public Education," July 19, 2006, at http://www.aft.org/presscenter/releases/2006/071906.htm
(August 31, 2006).
 U.S. Department of Education, Institute
of Education Sciences, What Works Clearinghouse, Web site, at http://www.whatworks.ed.gov (August 31,
 Brian Gill, P. Mike Timpane, Karen E.
Ross, and Dominic J. Brewer, Rhetoric Versus Reality: What We
Know and What We Need to Know About Vouchers and Charter
Schools (Santa Monica, Calif.: Rand Education, 2001), at http://www.rand.org/pubs/monograph_reports/
MR1118/index.html (August 31, 2006), and Paul E. Peterson
and David E. Campbell, eds., Charters, Vouchers, and Public
Education (Washington, D.C.: Brookings Institution Press,
 Jay P. Greene, Paul E. Peterson, and
Jiangtao Du, "School Choice in Milwaukee: A Randomized Experiment,"
in Paul E. Peterson and Bryan C. Hassel, eds., Learning from
School Choice (Washington, D.C.: Brookings Institution Press,
1998), and Cecilia Elena Rouse, "Private School Vouchers and
Student Achievement," Quarterly Journal of Economics, Vol.
113, No. 2 (May 1998).
 William G. Howell and Paul E. Peterson,
The Education Gap: Vouchers and Urban Schools, revised ed.
(Washington, D.C.: Brookings Institution Press, 2006).
 John Barnard, Constantine E. Frangakis,
Jennifer L. Hill, and Donald B. Rubin, "Principal Stratification
Approach to Broken Randomized Experiments: A Case Study of School
Choice Vouchers in New York City," Journal of the American
Statistical Association, Vol. 98, No. 462 (June 2003); Howell
and Peterson, The Education Gap; Alan B. Krueger and Pei
Zhu, "Another Look at the New York City School Voucher Experiment,"
Princeton University, Woodrow Wilson School of Public and
International Affairs Policy Brief, April 2003, at /static/reportimages/C8EADCD44E5F1E788BAFC9DE14AA51F1.pdf
(August 31, 2006); and Paul E. Peterson and William G. Howell,
"Latest Results from the New York City Voucher Experiment," paper
prepared for presentation before the Association of Public Policy
and Management, Washington, D.C., November 2003, at /static/reportimages/836BE730D17EA1A5698A766A7E0BB96D.pdf
(August 31, 2006).
 Howell and Peterson, The Education
 Math and reading achievement gains were
analyzed over a period of one to four years.
 The Milwaukee voucher program is
publicly funded; the other studies listed here are all of privately
funded voucher programs. For a good overview of all of these
studies, see Jay P. Greene, Education Myths: What Special
Interest Groups Want You to Believe About Our Schools-and Why It
Isn't So (Lanham, Md.: Rowman & Littlefield Publishers,
Inc., 2005), pp. 147-156.