Congress put school vouchers to the test in 2004 when it authorized the D.C. Opportunity Scholarship Program (DCOSP), a federally funded voucher program serving low-income students in the nation’s capital. It has awarded $7,500 scholarships to more than 3,700 students over the past six years.
Congress mandated a formal evaluation of the program, and researchers hired by the Department of Education have now released their latest report. Its content should help determine the future of DCOSP. Without reauthorization from Congress, the program will expire after next year.
What the Facts Demonstrate
Among the report’s key findings:
Parental satisfaction. School satisfaction was higher among parents of voucher students.
School safety. Parents of voucher students were more likely to describe their children’s schools as safe and orderly.
Graduation rates. Voucher-using students achieved a graduation rate of 91 percent, compared to 70 percent for non-voucher students.
Test scores. On reading tests, voucher students scored slightly higher (by 0.13 standard deviations) compared to non-voucher students, but the difference is not statistically significant. DCOSP did not produce any gains in mathematics scores.
These results should be considered in light of the study’s quality of methodology and consistency with past findings.
Quality of Methodology
Because parents, teachers, or the students themselves must elect to participate in programs like DCOSP, participants tend to be different from non-participants in terms of ability, motivation, family background, and many other variables. An essential part of any program evaluation is to avoid mistaking these initial differences for the effect of the program itself. To do this, evaluators need a control group that is as similar as possible to the students who participate in the program.
The DCOSP evaluation uses the best possible control group, which is constructed from a random lottery. Among 5,547 eligible applicants, 3,738 were randomly selected to receive a voucher. The DCOSP evaluation then compared students who used a voucher versus those who were denied the voucher by random chance.
A lottery is the “gold standard” method of evaluation, which produces results deserving the most attention. If statistically significant differences between participants and non-participants emerge from this strict comparison, policymakers can be sure that the program in question has had an impact.
Without a lottery, the next most desirable evaluation method is careful matching of participants and non-participants on as many background variables as possible. Since researchers can never account for every variable, matching is less reliable than the lottery method, but it can still be informative when performed carefully. Recent examples of effective matching include the evaluations of the Milwaukee Parental Choice Program and the Edgewood Voucher Program in Texas. Each came to similar conclusions as the DCOSP report.
Less scholarly studies use raw comparisons or insufficient matching of participants and non-participants. These evaluations are rarely informative. A recent example is a comparison of voucher and non-voucher students in New Orleans published by the non-profit group Educate Now. The comparison involved no matching at all, making the results uninterpretable.
Consistency with Past Findings
Increases in graduation rates and parental satisfaction are frequent findings in the school choice literature. Recent examples are the aforementioned Edgewood and Milwaukee evaluations, which used matching techniques. As the most rigorous evaluation, the DCOSP study can be viewed as confirmation of the positive effects on graduation and parental satisfaction observed in other voucher studies.
In prior years of this evaluation series, DCOSP did produce small reading gains that reached statistical significance, but the gains were insignificant in this round. Test scores are notoriously hard to raise through intervention. However, increasing funding for public schools—through class size reduction, teacher training, stricter certification requirements, etc.—also rarely results in significant test score improvement.
Because Congress funded the DCOSP and mandated the evaluation, the results outlined above—greater parental satisfaction, safer schools, and higher graduation rates—should inform upcoming policy debates. But if scholars and policymakers focus on DCOSP’s modest test score effects, they may overlook the broader benefits of school choice.
Even holding test scores constant, improving high school graduation rates has a substantial positive impact on students’ earnings later in life. And given the higher levels of parental satisfaction produced by DCOSP, test scores are clearly only one factor parents consider in evaluating schools.
In fact, parents probably understand the limitations of social policy better than most academics and policymakers do. Rather than obsessing over elusive test score gains, parents seem to have a more nuanced and child-specific set of criteria: They want schools that are safe, cultivate a positive attitude about learning, and best fit their children’s abilities and interests. Only school choice programs can satisfy these diverse preferences and expectations.
The Big Picture
In summary, the DCOSP evaluation uses the gold standard of scholarly rigor and reliability, and its findings corroborate past school choice studies. Among the positive outcomes of DCOSP are greater parental satisfaction—especially regarding school safety—and a significant increase in high school graduation rates. As the DCOSP evaluation makes clear, school choice offers real benefits to students and their families.
Jason Richwine is Senior Policy Analyst in the Center for Data Analysis at The Heritage Foundation.