A recent study published by the National Center for Education Statistics (NCES) contains some surprising results based on a snapshot of student achievement data. According to the study, public school students are performing better than private school students in fourth grade mathematics and at the same level as private school students in fourth grade reading and eighth grade math. Indeed, the report says that private school students have an advantage over public school students only in eighth grade reading.[1] These results should be handled very carefully.
Policymakers and journalists need to know that the NCES findings that public schools outperform private schools employ significantly limited data. Some commentators on the NCES report appear to believe that this study describes causal relationships-that public school attendance causes better student achievement and that private school attendance causes students to have lower math and reading achievement.
The NCES study analyzes the 2003 National Assessment of Educational Progress (NAEP) data that, due to a major limitation, are ill-suited for making any causal inferences. The NAEP data assess achievement only at one point in time, providing a snapshot of how American students are performing in math and reading at that specific time. The NAEP data are not suitable for evaluating the effectiveness of private or public school attendance in raising academic achievement. In fact, the NCES authors explicitly warn against this in two sections of the report that are appropriately titled "Cautions in Interpretation."[2]
Furthermore, various individuals and organizations have used the report to discredit private school voucher programs,[3] asserting that government education funds should be used only to fund public education, where they can be put to better use. However, the results of several more sophisticated and conclusive studies of voucher program effectiveness overwhelmingly point toward greater math and reading achievement for students who attend private schools through voucher programs.
Causal Interpretation
As noted, some commentators appear to believe that the NCES study describes causal relationships-that public school attendance causes better student achievement and private school attendance causes students to have lower math and reading achievement. This is surprising because the authors of the report explicitly warn against causal interpretations. They also acknowledge that comparisons of public and private schools (which amounts essentially to comparing apples to oranges) may not be useful because of the differences that exist between and within the public and private school populations.
The NCES report analyzes the 2003 NAEP data, a nationally representative assessment of academic performance in math, reading, and science (among other things). Researchers frequently use the NAEP data to gauge student academic performance in the United States, even though the cross-sectional nature of these data limits what these researchers can do. Analysis of NAEP data can only be used to generally describe the characteristics of students who took the NAEP tests. These data are not suitable for use in evaluating the effectiveness of public and private schooling over time.
In fact, the NCES study would not even meet the standards of effective research set forth by the Institute of Education Sciences-the very people responsible for releasing the study. The Institute of Education Sciences, which is part of the Department of Education, established the What Works Clearinghouse (WWC) in 2002[4] to provide researchers, policymakers, and the general public with standards by which to evaluate research on educational interventions. The WWC evidence standards require that research studies include either random assignment or a pretest measure for the research results to be considered effective at establishing causality.
The NAEP assessments do not meet either of these criteria. The students who take the NAEP tests each year are not randomly assigned into public or private schools. In addition, the NAEP does not sample the same students or schools during each successive administration. This means that the NAEP data do not include any pretest or baseline measures that can be used to adjust for differences between the public and private school populations.
The best method to draw causal inferences from research is to collect data as part of a random study. In this type of study, research participants are randomly assigned to either an experimental group or a control group. The experimental group receives some treatment-their environment is changed or manipulated in some manner. The control group, on the other hand, is not exposed to anything new or different and serves as a source of comparison for the experimental group.
Random assignment helps to ensure that the experimental and control groups contain similar mixes of participants and are generally representative of the larger population to which they belong. The variable of interest is measured at regular intervals throughout the duration of the study (including at the beginning and end) to see whether there is any difference in outcomes between the experimental and control groups. In this case, causal inferences can be drawn because any difference in outcomes between the experimental and control groups can be attributed to the treatment experienced by the experimental group.
Education studies that include measurement over time are much more useful for drawing conclusions about school quality. Education researchers have repeatedly pointed out that a student's low test score at a single specific time may indicate only that she is not a good student or that some external circumstance influenced her bad performance on the test that day. However, if that student's test scores rise over time, it indicates that she is being well-served and well-educated by the school that she is attending.
The NAEP data do not sample the same students or schools each year. Thus, researchers cannot study how achievement has changed over time for either individual students or a cohort of students attending the same school, whether public or private. As a result, the NAEP data are not well-suited to establishing whether a specific math or reading achievement outcome is associated with attending either a private or public school.
Evidence on Voucher Programs
Voucher programs specifically target the academic needs of low-income (frequently minority) students, who often live and go to school in high-poverty areas. These children are frequently stuck in persistently low-performing public schools that are not meeting their educational needs.
Currently, six states and the District of Columbia offer government-sponsored voucher programs to their students. In addition, several other states offer privately funded voucher programs.[5] These programs allocate government or private funds directly to parents who then use that money to pay tuition costs at a private school of their choosing. Voucher programs are a powerful school choice tool because they give low-income families the opportunity to send their children to private, tuition-based schools that they could not afford without financial assistance.[6]
Several random assignment studies have been conducted on the effects of private school voucher programs on math and reading achievement. These studies include evaluations of public and privately funded voucher programs in Wisconsin,[7] North Carolina,[8] Ohio,[9] New York,[10] and the District of Columbia.[11] The studies compared the achievement outcomes of students who were randomly awarded vouchers to attend a private school to the outcomes of those students who did not receive vouchers and remained in their regular public schools. They all reached the same conclusion: Students who received vouchers experienced greater math and/or reading achievement gains than did the students who remained in the public school system.
Since these studies included random design and measured differences in achievement scores over time,[12] it can be said with confidence that attending a private school resulted in significantly greater math and/or reading achievement gains for the students who were awarded vouchers.[13]
Conclusion
The results of a recent NCES study that finds that public school students are performing better then private school students should be interpreted cautiously. The NAEP data used in the study provide a snapshot of student achievement at the time the NAEP tests were administered. As a result, the NAEP data can be used only to describe the characteristics of the student population in the United States at the time when students took the NAEP tests.
The NAEP data are certainly not suitable for establishing whether a specific math or reading achievement outcome is associated with attending either a private or public school. Despite this fact, the results of the NCES study are being interpreted inappropriately to imply that voucher programs, which include private schools, are a bad idea.
The research literature that addresses the effectiveness of school voucher programs in raising math and reading achievement, based on more sophisticated methodology, is much more convincing and conclusive. Students who attend a private school through a voucher program experience greater gains in math and reading than do their public school counterparts.
Voucher programs are a powerful school choice tool that results in many positive outcomes for the students who participate in them. More families should have the opportunity and flexibility to choose the most effective school for their children to achieve success.
Shanea Watkins is Policy Analyst in Empirical Studies in the Center for Data Analysis at The Heritage Foundation.
[1] Henry Braun, Frank Jenkins, and Wendy Grigg, "Comparing Private Schools and Public Schools Using Hierarchical Linear Modeling," NCES 2006-461, U.S. Department of Education, National Center for Education Statistics, Institute of Education Sciences, July 2006, at /static/reportimages/138647316ED8FB156FBA7DA4338A6FEC.pdf (August 31, 2006).
[2] Ibid., pp. v and 4.
[3] National Education Association,
"Education Department Reports Private Schools on Par with Public
Schools," July 20, 2006, at
www.nea.org/newsreleases/2006/nr060720.html (August 31,
2006); Lois Romano, "GOP Unveils School Voucher Plan," The
Washington Post, July 19, 2006, p. A17, at http://www.washingtonpost.com/wpdyn/
content/article/2006/07/18/AR2006071801305.html (August 31,
2006); Diana Jean Schemo, "Public Schools Perform Near Private Ones
in Study," The New York Times, July 15, 2006, p. A1; Zachary
M. Seward, "Long-Delayed Education Study Casts Doubt on Value of
Vouchers," The Wall Street Journal, July 15, 2006; Edward J.
McElroy, President, American Federation of Teachers, "On the
Introduction of School Voucher Legislation," July 18, 2006, at
http://www.aft.org/presscenter/releases/2006/071806a.htm
(August 31, 2006); Edward J. McElroy, "On Vouchers and Private vs.
Public Education," July 19, 2006, at http://www.aft.org/presscenter/releases/2006/071906.htm
(August 31, 2006).
[4] U.S. Department of Education, Institute of Education Sciences, What Works Clearinghouse, Web site, at http://www.whatworks.ed.gov (August 31, 2006).
[5] For more information on school choice
initiatives, including voucher programs, see The Heritage
Foundation, "Types of School Choice," at
http://www.heritage.org/Research/Education/S
choolChoice/typesofschoolchoice.cfm (August 31, 2006).
[6] Brian Gill, P. Mike Timpane, Karen E.
Ross, and Dominic J. Brewer, Rhetoric Versus Reality: What We
Know and What We Need to Know About Vouchers and Charter
Schools (Santa Monica, Calif.: Rand Education, 2001), at http://www.rand.org/pubs/monograph_reports/
MR1118/index.html (August 31, 2006), and Paul E. Peterson
and David E. Campbell, eds., Charters, Vouchers, and Public
Education (Washington, D.C.: Brookings Institution Press,
2001).
[7] Jay P. Greene, Paul E. Peterson, and Jiangtao Du, "School Choice in Milwaukee: A Randomized Experiment," in Paul E. Peterson and Bryan C. Hassel, eds., Learning from School Choice (Washington, D.C.: Brookings Institution Press, 1998), and Cecilia Elena Rouse, "Private School Vouchers and Student Achievement," Quarterly Journal of Economics, Vol. 113, No. 2 (May 1998).
[8] Jay P. Greene, "Vouchers in Charlotte," Education Matters, Summer 2001, at /static/reportimages/40719E6D99C47B453CD37459E1027343.pdf (August 31, 2006).
[9] William G. Howell and Paul E. Peterson, The Education Gap: Vouchers and Urban Schools, revised ed. (Washington, D.C.: Brookings Institution Press, 2006).
[10] John Barnard, Constantine E. Frangakis, Jennifer L. Hill, and Donald B. Rubin, "Principal Stratification Approach to Broken Randomized Experiments: A Case Study of School Choice Vouchers in New York City," Journal of the American Statistical Association, Vol. 98, No. 462 (June 2003); Howell and Peterson, The Education Gap; Alan B. Krueger and Pei Zhu, "Another Look at the New York City School Voucher Experiment," Princeton University, Woodrow Wilson School of Public and International Affairs Policy Brief, April 2003, at /static/reportimages/C8EADCD44E5F1E788BAFC9DE14AA51F1.pdf (August 31, 2006); and Paul E. Peterson and William G. Howell, "Latest Results from the New York City Voucher Experiment," paper prepared for presentation before the Association of Public Policy and Management, Washington, D.C., November 2003, at /static/reportimages/836BE730D17EA1A5698A766A7E0BB96D.pdf (August 31, 2006).
[11] Howell and Peterson, The Education Gap.
[12] Math and reading achievement gains were analyzed over a period of one to four years.
[13] The Milwaukee voucher program is publicly funded; the other studies listed here are all of privately funded voucher programs. For a good overview of all of these studies, see Jay P. Greene, Education Myths: What Special Interest Groups Want You to Believe About Our Schools-and Why It Isn't So (Lanham, Md.: Rowman & Littlefield Publishers, Inc., 2005), pp. 147-156.
