December 21, 2004

December 21, 2004 | Commentary on

Finding the Truth in Federal Data

Imagine trying to work out the family budget with two different sets of books when they show a dramatically different bottom line. Which could you rely on?

Welcome to the world of federal data - which can drive researchers and policy-makers crazy.

For example, on Nov. 30, the Bureau of Labor Statistics released the new consumer expenditures figures for 2003. The Labor Department says more than 115 million households spent an average of $40,817 on goods and services last year, totaling some $4.7 trillion.

But hold the phone. Earlier this year, the Commerce Department released its own personal consumption figures for 2003. These numbers were different, to say the least. Commerce said spending amounted to $7.76 trillion, about $3 trillion more than BLS announced. So what gives?

Unfortunately, this is an all-too-common occurrence. Not only do we have different statistical agencies releasing sometimes substantially different numbers on key public policy indicators, but we also have cases where the same federal agency releases conflicting numbers.

Just look at the Bureau of Labor Statistics again. Out of one side of its mouth, the BLS says that a net 283,000 new jobs were created between June and July of this year, as measured by revised figures from the agency's payroll survey. Out of the other, BLS claims that 577,000 - more than twice as many people - became employed in July, as measured by its household survey.

Confused yet? So are federal officials. Upon release of the jobs and employment report last summer, Treasury Secretary John Snow commented, "The divergence between the household and the payrolls survey is very striking, and I'll leave it to statisticians to try to reconcile those numbers."

He then concluded, "I suppose that the real number lies somewhere in between."

Unfortunately, federal number crunchers don't have the answers either. Tom Nardone, chief of the Division of Labor Force Statistics, was remarkably candid: "We just don't know why there's a difference between the surveys."

While the jobs and employment report is a high-profile example of the federal data problem, it's certainly not the only one. In a June 2004 report, Census Bureau statistician Shailesh Bhandari delved into why two Census surveys that both measure health insurance coverage rates came to different conclusions.

In 2001, the Current Population Survey showed that 85.4 percent of Americans had health insurance. However, that's substantially lower than the 93.2 percent estimated by the Survey of Income and Program Participation. This translates into a 22 million-person discrepancy. The numbers matter, because federal policy-makers use them when passing laws or issuing orders. How can our elected officials make the right decisions if they aren't working from the right data?

After much effort, though, the Census Bureau couldn't reconcile the two numbers. Instead, the oft-heard "Further research is needed" position was offered.

And what about the $3 trillion difference between expenditure data reported by the Commerce Department and the BLS? That much money ought to be easy to track down. But it isn't.

Several years ago, the General Accounting Office tried to rationalize the two estimates. While some of it could be explained by "definitional differences," the GAO concluded in 1996, "Differences between estimates of consumer spending in the PCE [Commerce] and CEX [BLS] data cannot be fully reconciled."

These examples only scratch the surface of the problem. There are dozens of cases where different federal surveys yield different results on important policy questions. Some, not surprisingly, are more egregious than others.

The point of this is not to attack the Bureau of Labor Statistics, the Census Bureau, or other federal statistical agencies. I know from my own first-hand experience at the Census Bureau that such agencies are filled with hard-working and diligent survey statisticians. But these different federal agencies should work to reconcile their inconsistent numbers. Only through coordinating their data efforts can they end their perpetual discrepancies.

Additionally, policy-makers and the general public ought to be aware that the information they're seeing isn't exact by any means. We all need to review, assess and investigate all of the available data in order to make truly informed decisions on important public policy issues.

Only after policy-makers begin to consider all of the relevant data, and agencies stem the flow of inconsistent information, will Americas truly be able to have an intelligent debate on the issues of the day.

Stephen Johnson is senior policy analyst for Latin America in the Davis Institute for International Studies at The Heritage Foundation.

About the Author

Kirk A. Johnson, Ph.D. Visiting Fellow
Center for Data Analysis

First Appeared on