GAO Critiques Research Touting COPS Program Effectiveness

A new General Accounting Office (GAO) study, reviewing an Office of Community Oriented Policing Services (COPS) funded self-evaluation, casts doubt on claims on the effectiveness of COPS grants.

Among the flaws identified in the COPS study's methodology -- all of which put into question claims regarding the program's effectiveness -- were:

1. Its failure to account for factors that may have significantly influenced crime rates,
2. Its use of outdated control variables, and
3. The exclusion of a control for the efforts of local law-enforcement entities.

The COPS program has a long history of poor performance. It has failed to achieve its goals, and has assigned to the federal government functions that fall within the expertise, jurisdiction, and constitutional responsibilities of state and local governments.

Flawed Methodology
The COPS study -- conducted by researchers at the University of Nebraska at Omaha, COPS, and Southwest Texas State University -- concluded that the program had "a strong impact on making communities safer places to live."[1] This finding was not surprising since the study was funded by the COPS program through an agreement that gave the agency control over the study's finding and conclusions.[2]

In an assessment similar to that previously produced by The Heritage Foundation's Center for Data Analysis,[3] the GAO severely criticized the methodology used by the authors of the "COPS study" and concluded that "We cannot agree...that their 2001 study shows that some COPS grants (hiring and innovative) significantly reduced crime because, among other things, important variables were omitted from their analyses, the analytic models were misspecified, and the sample of cities included in the study was limited."[4]

Further, the GAO stated "Our review revealed several problems with the 2001 [COPS] study that cast doubt on the validity of the conclusions about the effectiveness of COPS grants."[5]

Serious Failures in Methodology
In sum, the GAO critique of the COPS study is based on the following five points:

1. The results of the COPS study can be regarded as equivocal, inconsistent, and inconclusive;
2. The omission of important variables may have biased the study's results;
3. The analytic model used in the COPS study was misspecified;
4. The COPS study excluded relevant law enforcement agencies from its analysis; and
5. The COPS study relied on outdated census data.

Equivocal, inconsistent, and inconclusive findings. While the COPS study's authors emphasized that COPS grants for hiring and innovation had been effective in reducing crime in cities with populations greater than 10,000, the authors downplayed findings that COPS grant had no effect-or were even correlated with increased crime rates-in cities with populations between 1,000 and 10,000.[6] For all of these cities in the COPS study, the authors found that COPS hiring grants and MORE grants failed to have an impact on crime rates.[7] This inconsistency in the COPS study's findings led the GAO to declare, "One can also conclude that the study's findings are equivocal, inconsistent, and inconclusive."[8]

Omission of significant variables. Another highly questionable aspect of the COPS study is its assumption that state and local law-enforcement efforts do not influence crime rates. Noting this major shortcoming of the COPS study, the GAO stated, "We believe the absence of any control for state and local expenditures to be a serious weakness."[9] 

As The Heritage Foundation reported in 2002, while the COPS program had a nationwide budget of $6.9 billion during 1994-1999, state and local governments allocated more than $280 billion for police agencies.[10] In other words, for every $1 spent on COPS initiatives, over $40 was spent by state and local governments for police protection.

A misspecified analytic model. The GAO concluded that the model used in the COPS study was misspecified.[11] The COPS study used county-level dummy variables to control unmeasured variability in factors that may influence crime rates across counties, while the unit of analysis was cities.[12] The GAO asserted, "Since the data on crime rates and COPS funds were measured at the city level, that unmeasured variability would have been more effectively controlled had dummy variables been used to distinguish cities, instead of the counties in which the cities were located."[13] Moreover, the GAO found that the inability of the COPS study's county-level dummy variable approach to control for "unmeasured and systematic variability across cities within the same county" introduced "a potential source of bias in the parameters representing the effects of the COPS grants estimated in the models."[14]

The exclusion of relevant law enforcement agencies. The COPS study only analyzed crime rates in cities that received COPS grants, even though COPS grants went to county policy departments, sheriff's officers, university police departments, and other specialized law-enforcement agencies. The agencies that were excluded from the COPS study account for more 40 percent of all the agencies that received COPS grants from 1994 to 1998.[15]

Outdated Census Data. Although COPS researchers had been sharply critical of prior research that did not "control for extraneous factors that may be correlated with both increases in the number of police officers and increases in crime rates, such as local politics or fluctuations in the local economies of cities," they ignored important contributing factors in their own study.[16] According to the GAO, the COPS study used "outdated census data for control variables."[17] A 2002 Heritage Foundation CDA report criticized the COPS study for being based mainly on data from 1990 and failing to take into account many significant subsequent demographic changes that may have influenced crime rates, such as fluctuations in minority and youth populations.[18] As far-fetched as it may seem, the COPS study used 1990 data in an attempt to account for changes in crime rates from 1995 to 1999.

A Shadow of Doubt
Given the five failures in the COPS study's methodology described above, the GAO is correct in questioning the validity of the study's conclusions. Although the COPS study was presented to Congress and the public as independent research, the Office of Management and Budget has rightly called for COPS to take "additional steps to guarantee the independence of [its] external evaluations."[19]

COPS funded its current evaluation through a cooperative agreement that gave the agency control over the study's findings and conclusions. The study's authors' lack of independence has cast considerable doubt on the objectivity of the COPS study.[20] It should be noted that COPS and the study's authors initially refused to release the data used for their study and complied only when a congressional inquiry forced them to do so.[21]