A recent study published by the National Center for Education Statistics (NCES) contains some surprising results based on a snapshot of student achievement data. According to the study, public school students are performing better than private school students in fourth grade mathematics and at the same level as pri­vate school students in fourth grade reading and eighth grade math. Indeed, the report says that private school students have an advantage over public school stu­dents only in eighth grade reading.[1] These results should be handled very carefully.

Policymakers and journalists need to know that the NCES findings that public schools outperform private schools employ significantly limited data. Some com­mentators on the NCES report appear to believe that this study describes causal relationships-that public school attendance causes better student achievement and that private school attendance causes students to have lower math and reading achievement.

The NCES study analyzes the 2003 National Assessment of Educational Progress (NAEP) data that, due to a major limitation, are ill-suited for making any causal inferences. The NAEP data assess achieve­ment only at one point in time, providing a snapshot of how American students are performing in math and reading at that specific time. The NAEP data are not suitable for evaluating the effectiveness of private or public school attendance in raising academic achievement. In fact, the NCES authors explicitly warn against this in two sections of the report that are appropriately titled "Cautions in Interpretation."[2]

Furthermore, various individuals and organiza­tions have used the report to discredit private school voucher programs,[3] asserting that govern­ment education funds should be used only to fund public education, where they can be put to better use. However, the results of several more sophisti­cated and conclusive studies of voucher program effectiveness overwhelmingly point toward greater math and reading achievement for students who attend private schools through voucher programs.

Causal Interpretation

As noted, some commentators appear to believe that the NCES study describes causal relation­ships-that public school attendance causes better student achievement and private school attendance causes students to have lower math and reading achievement. This is surprising because the authors of the report explicitly warn against causal interpre­tations. They also acknowledge that comparisons of public and private schools (which amounts essen­tially to comparing apples to oranges) may not be useful because of the differences that exist between and within the public and private school populations.

The NCES report analyzes the 2003 NAEP data, a nationally representative assessment of academic performance in math, reading, and science (among other things). Researchers frequently use the NAEP data to gauge student academic performance in the United States, even though the cross-sectional nature of these data limits what these researchers can do. Analysis of NAEP data can only be used to generally describe the characteristics of students who took the NAEP tests. These data are not suit­able for use in evaluating the effectiveness of public and private schooling over time.

In fact, the NCES study would not even meet the standards of effective research set forth by the Insti­tute of Education Sciences-the very people respon­sible for releasing the study. The Institute of Education Sciences, which is part of the Department of Education, established the What Works Clearing­house (WWC) in 2002[4] to provide researchers, pol­icymakers, and the general public with standards by which to evaluate research on educational interven­tions. The WWC evidence standards require that research studies include either random assignment or a pretest measure for the research results to be considered effective at establishing causality.

The NAEP assessments do not meet either of these criteria. The students who take the NAEP tests each year are not randomly assigned into public or private schools. In addition, the NAEP does not sample the same students or schools during each successive administration. This means that the NAEP data do not include any pretest or baseline measures that can be used to adjust for differences between the public and private school populations.

The best method to draw causal inferences from research is to collect data as part of a random study. In this type of study, research participants are ran­domly assigned to either an experimental group or a control group. The experimental group receives some treatment-their environment is changed or manipulated in some manner. The control group, on the other hand, is not exposed to anything new or different and serves as a source of comparison for the experimental group.

Random assignment helps to ensure that the experimental and control groups contain similar mixes of participants and are generally representa­tive of the larger population to which they belong. The variable of interest is measured at regular inter­vals throughout the duration of the study (includ­ing at the beginning and end) to see whether there is any difference in outcomes between the experi­mental and control groups. In this case, causal inferences can be drawn because any difference in outcomes between the experimental and control groups can be attributed to the treatment experi­enced by the experimental group.

Education studies that include measurement over time are much more useful for drawing con­clusions about school quality. Education research­ers have repeatedly pointed out that a student's low test score at a single specific time may indicate only that she is not a good student or that some external circumstance influenced her bad performance on the test that day. However, if that student's test scores rise over time, it indicates that she is being well-served and well-educated by the school that she is attending.

The NAEP data do not sample the same students or schools each year. Thus, researchers cannot study how achievement has changed over time for either individual students or a cohort of students attending the same school, whether public or pri­vate. As a result, the NAEP data are not well-suited to establishing whether a specific math or reading achievement outcome is associated with attending either a private or public school.

Evidence on Voucher Programs

Voucher programs specifically target the aca­demic needs of low-income (frequently minority) students, who often live and go to school in high-poverty areas. These children are frequently stuck in persistently low-performing public schools that are not meeting their educational needs.

Currently, six states and the District of Columbia offer government-sponsored voucher programs to their students. In addition, several other states offer privately funded voucher programs.[5] These pro­grams allocate government or private funds directly to parents who then use that money to pay tuition costs at a private school of their choosing. Voucher programs are a powerful school choice tool because they give low-income families the opportunity to send their children to private, tuition-based schools that they could not afford without financial assistance.[6]

Several random assignment studies have been conducted on the effects of private school voucher programs on math and reading achievement. These studies include evaluations of public and privately funded voucher programs in Wisconsin,[7] North Carolina,[8] Ohio,[9] New York,[10] and the District of Columbia.[11] The studies compared the achievement outcomes of students who were randomly awarded vouchers to attend a private school to the outcomes of those students who did not receive vouchers and remained in their regular public schools. They all reached the same conclusion: Students who received vouchers experienced greater math and/or reading achievement gains than did the students who remained in the public school system.

Since these studies included random design and measured differences in achievement scores over time,[12] it can be said with confidence that attend­ing a private school resulted in significantly greater math and/or reading achievement gains for the stu­dents who were awarded vouchers.[13]

Conclusion

The results of a recent NCES study that finds that public school students are performing better then private school students should be interpreted cau­tiously. The NAEP data used in the study provide a snapshot of student achievement at the time the NAEP tests were administered. As a result, the NAEP data can be used only to describe the characteristics of the student population in the United States at the time when students took the NAEP tests.

The NAEP data are certainly not suitable for establishing whether a specific math or reading achievement outcome is associated with attending either a private or public school. Despite this fact, the results of the NCES study are being interpreted inappropriately to imply that voucher programs, which include private schools, are a bad idea.

The research literature that addresses the effec­tiveness of school voucher programs in raising math and reading achievement, based on more sophisticated methodology, is much more convinc­ing and conclusive. Students who attend a private school through a voucher program experience greater gains in math and reading than do their public school counterparts.

Voucher programs are a powerful school choice tool that results in many positive outcomes for the students who participate in them. More families should have the opportunity and flexibility to choose the most effective school for their children to achieve success.

Shanea Watkins is Policy Analyst in Empirical Studies in the Center for Data Analysis at The Heritage Foundation.