Introduction

Lindsey M. Burke, PhD

Two years ago, Heritage Foundation Senior Research Fellow Jay Greene was retained by the state of New York to write an expert report as part of its defense in New Yorkers for Students’ Educational Rights, et al. v. The State of New York.REF The Heritage Foundation is now able to publish that report, enabling Americans to see the evidence debunking the claim that increasing education spending generally leads to improved student outcomes.

This claim has become almost a matter of consensus among education policy researchers, more than 450 of whom signed a group letter stating, “Research is abundantly clear that money matters for student achievement and other important life outcomes, and this is especially the case for low-income students.”REF That sentence contains four citations, all of which refer to research conducted by Kirabo Jackson, an economics professor at Northwestern University. Jackson also served as an expert witness in New Yorkers for Students’ Educational Rights, et al. v. The State of New York, but on behalf of the plaintiffs. The report reproduced below by Jay Greene is a rebuttal of Jackson’s claims about the effects of increasing school spending.

Serious Flaws in School Spending Research

Greene’s expert report reveals that Jackson’s evidence is plagued by errors and characterized by p-hacking, or specification-shopping, techniques with which researchers alter their analyses, deliberately or unconsciously, to ensure that they yield desired results. The practice of p-hacking or specification-shopping renders the results unreliable.

Because Jackson produced six different versions of his meta-analysis, or systematic review, of the evidence on the causal relationship between school spending and student outcomes for different legal proceedings, it is possible to see how Jackson changed the analysis to engineer more favorable results. For example, Greene’s expert report documents:

There are 37 studies that Dr. Jackson at one time or another thought appropriate to consider when reviewing the evidence on how additional spending affects student outcomes. Dr. Jackson has changed his mind about whether to include the study or how to characterize the direction or significance of the results for 22 of those 37 studies. For some of those studies, he has changed his characterization of the results more than once across these six reviews in the last two years.

Since most studies report multiple results, meta-analyses or systematic reviews must apply consistent rules for selecting which results are considered or how multiple results should be combined. On more than one occasion, Jackson changed those rules and then inconsistently applied them to select positive outcomes and avoid negative ones.

In a flagrant example of p-hacking observed in real time, Jackson switched an analysis that produced a null finding reported in the November 2020 version of his meta-analysis so that in the March 2021 version he found a statistically significant effect with a p-value of .0497, just below the standard threshold of .05 for classifying a result as statistically significant. Changing analyses to achieve p values below a critical threshold is the very definition of p-hacking.

Jackson also failed to follow standard procedures for identifying the full set of studies he should include in his meta-analysis. Instead, he simply did a Google search and asked followers on Twitter for suggestions. Not surprisingly, the set of studies he found was skewed to exclude those with negative results.

In another study, Jackson sought to demonstrate that cuts in education spending during the Great Recession led to worse outcomes for students in states where education funding is more dependent on state revenues that are more sensitive to economic cycles than local property taxes. There are multiple data sources for information on the share of education spending that comes from state revenues. There is also a difficulty in deciding what the “state” share of education spending is in the District of Columbia, where the state and local government are one and the same.

Jackson made the indefensible decision to assign the District a value of 0 for the state share of education revenue even though his data source listed the information for DC as “not applicable.” He also chose to rely solely on Census data for the state share of education spending rather than also considering data from the U.S. Department of Education’s National Center for Education Statistics. Had he made different decisions, he would have found a null result in most alternative specifications. Had he excluded DC from the analysis, which seems like the most reasonable solution, 23 of 30 possible permutations of data sources for his two outcomes would have yielded null results.

In addition, Jackson chose to divide states into three groups based on whether they have high, medium, or low shares of education spending from state revenue. Had he chosen slightly different cut-offs for classifying states into these groups, he would have found null results in almost all cases. In fact, of the 120 permutations that Greene examined of how one might classify the states and which data sources to use, Jackson would have found null effects in 104 of those analyses. Only by making very specific and insupportable decisions about model specification and data sources is he able to find significant, positive effects for school spending. This is the very definition of specification-shopping.

As Greene’s expert report states, “The practice of making or altering research decisions after results are known in a way that supports one’s argument is known as cherry-picking, p-hacking, or specification-shopping. These practices are fundamentally inconsistent with generally accepted scientific principles. It is false to claim a bull’s-eye if the circles are drawn after the arrow lands.”

Intellectual Corruption in Social Science

Despite all the obvious defects of Jackson’s research, his meta-analysis was recently accepted for publication in a leading economics journalREF and his analysis of the effects of cuts in education spending during the Great Recession was earlier published in another leading economics journal.REF The Biden Administration also just announced the appointment of Jackson to the White House’s Council of Economic Advisors in part because of his research on the benefits of increased school spending.REF

How did the journal editors, the articles’ peer reviewers, more than 450 education policy researchers, and the staff who vet presidential appointments all miss the academic dishonesty in Jackson’s research on this issue? Scientific processes, such as peer review, are intended to detect these kinds of problems and ensure the quality of published research. In addition, well-trained social scientists should approach research with skepticism and should not collectively embrace findings as “settled science” without overwhelming evidence and careful scrutiny.

Social scientists have abandoned the standards of their disciplines and betrayed the rigor of their training for the allure of advancing political goals and receiving social acceptance and career advancement within their professional communities.

Many social scientists want to believe that additional school spending would improve student outcomes, regardless of what the evidence shows. In addition, Jackson is a high-status academic who is an editor at two top economics journals. In fact, one of those journals published his Great Recession paper. He may have followed the formal process of recusing himself from the decision to publish that article, but the close personal connections and the dangers of crossing someone in a powerful position would nevertheless undermine the quality control that editors and the peer-reviewers they select are supposed to ensure.

Economics, in particular, once stood out among the social sciences for having greater methodological rigor and a combative culture that would facilitate careful scrutiny of research. But like many institutions that once served as checks on concentrated power, from journalism to professional associations, social science has been corrupted by the allure of that power.

That is why it is so important that The Heritage Foundation publish Greene’s expert report. It not only documents the falsehood of specific claims about the benefits of additional school spending, it reveals the ways in which social science was unable to detect and reject shoddy research. Nothing about this report is out of date and no subsequent revisions in Jackson’s research can undo the defects revealed here. Once a study is marred by p-hacking, that bell cannot be un-rung. The results have been compromised by selecting favorable results after doing too many analyses, which cannot be corrected by doing yet more analyses.

Now that Greene’s report is available to the public, everyone can see those defects and how they were ignored.