A Baghdad Statistician's Perspective on the Positives and Negativesof Polling in Iraq
Opinion polls can provide some insight into the thoughts of
broad populations and have been used to gauge public perception for
decades in the United States and other Western countries. Since
shortly after the start of the conflict in Iraq, household surveys
have been conducted by a variety of news outlets, non-governmental
organizations, and other groups in order to ascertain what Iraqis
think about the major issues of the day. Recently, some of these
polls have been introduced in the Iraq debate on Capitol Hill.
I was Deputy Director for Assessments at the U.S. Embassy in
Baghdad between July 2006 and August 2007. In that position, I
analyzed a great number of opinion surveys for Ambassador Crocker,
senior leaders at the embassy, and other interested personnel. Over
the course of those many months, I came to appreciate the
difficulties of polling in such an environment. Policymakers should
approach such polls with caution and should use them to measure
trends in the population rather than to conduct policy.
Issues with Polling in Iraq
The following six issues must be considered when analyzing the
results of any Iraqi survey.
- Iraqis typically have negative reactions and opinions
when asked about the American occupation. The reasoning
for this is simple: Iraq is a sovereign country, and the Iraqis
want to be able to determine their own affairs. Large segments of
Iraq have, since the start of the conflict, been opposed to the
Recently, The New York Times captured this
sentiment when a city worker in Baquba, in Diyala Province,
forcefully said, "I want them to withdraw all their troops in one
That notwithstanding, the Sunni worker continued:There
is something that I want to say although I hate to say it. The
Americans forces, which are an ugly occupation force, have become
something important to us, the Sunnis. We are a minority and we do
not having a force to face the militias. If the Americans leave, it
will mean a total elimination of the Sunnis in Iraq.
There is a sense in Iraq that the security situation would worsen
if Coalition Forces (CF) leave too soon. Ambassador Crocker noted
during the September 10, 2007, hearing, "All of Iraq's principal
leaders…don't want to see any marked precipitous reduction
in how those [CF military] forces are deployed until conditions
sustain it." If the Iraqi people truly wanted U.S. and CF troops to
leave Iraq immediately, their leaders would force the issue.
Instead, its leaders have encouraged CF troops to stay.
I know I said I want them to leave, but if we think about it, then
I have to say I want them to stay for a while until we end all the
suspicions we have of each other and have a strong national
- Security is a lagging indicator and tends to be better
within neighborhoods as compared to outside neighborhoods.
People will not feel a broad sense of security until sufficient
time passes for perspective. Better security has generally been a
recent phenomenon. During the House and Senate hearings, General
Petraeus indicated that security incidents were down in eight of
the last 12 weeks. It may take time for this improvement to be
reflected in polling data.
The security data shows that Baghdad became more secure earlier in
2007 as compared to the rest of Iraq. The surge first focused on
setting up roughly 30 joint security stations around the city
staffed with additional CF troops. As a result, the capital became
more secure sooner. Ambassador Crocker noted during the House
hearing, "I have seen other national polling data that shows, for
example, that the number of Iraqis who now feel secure in their own
neighborhoods and indeed feel secure moving around the city has
gone up significantly." In fairness, Ambassador Crocker indicated
thereafter that he does not know of the accuracy of the polling.
That said, he understands that it is difficult to poll in conflict
areas such as Baghdad unless it is done very carefully.
- Opinions tend to vary significantly across locations
and ethno-sectarian groups. Iraq, by its very nature, is
not a monolith. Public opinion tends to differ not only by the
three major ethno-sectarian groups (Shia, Sunni, and Kurd), but
also by geography. Take, for example, some polling reported by the
military in their latest "Measuring Security and Stability in Iraq"
report. The military reported some results of a
nationwide poll taken in April 2007. One question asked respondents
to agree or disagree with the following statement: "I feel safe and
secure in my neighborhood." While most Iraqis in the Kurdish areas
and in most of the Shia-dominated south agreed with that statement,
there was less agreement in the Sunni provinces just north and west
of Baghdad. Even though 77 percent of Iraqis nationwide agreed with
the statement, the distribution of answers clearly varies by
Poll responses differ by ethno-sectarian group as well. Different
communities often have varying views on a wide range of subjects.
Obtaining separate results for Kurdish, Shia, and Sunni populations
would paint a better picture of opinions than simply polling Iraqis
at large. One such example of ethno-sectarian differences is shown
in a recent BBC/ABC poll. Shia Arabs are more likely to believe
that the security situation in their neighborhoods is good, as
compared with Sunni Arabs, by a margin of 46 percent to 21 percent.
(There were no figures for Kurds, although the polling methodology
suggests that they were surveyed.)
- Iraqi polls should be analyzed via trends rather than
snapshots in time. Because of the difficulties in polling
in unstable areas such as Iraq (see below), static percentages or
other results may not have the kind of reliability that Western
polls enjoy. Therefore, most analysts look at trends of polls. The
actual level of opinion is secondary to whether opinion is staying
the same, increasing, or decreasing over time.
- No matter how good the polling methodology is, there
can be problems with polls when there is distance between the
organization commissioning the poll and the interviewers
themselves. Many polls that are fielded in dangerous
places are already difficult to complete because of the security
situation. In addition, the actual interviewers may not be
conducting the poll as the methodology would dictate. In Western
countries, after the polling contractor conducts the poll, the
organization that commissioned the poll can check to be sure that
the polling company completed the interviews as specified by the
survey methodology. This is not possible for polls where there is
such a firewall between the polling company and the organization
that commissioned the poll.
By way of example, the BBC/ABC News poll was not actually
conducted by either of the news organizations themselves. Rather,
the interviews were conducted by Iraqis. This is appropriate for
security reasons. The downside, however, is that basic polling
validity checks of the data cannot be undertaken. In the United
States, a good survey methodology would allow for the originating
organization to contact households to verify that someone came by
to ask questions. In a conflict environment, the poll becomes a
"black box" whereby the organization commissioning the poll must
trust the results without verification.
At least one organization, the International Republican Institute,
cancelled a polling contract in Iraq in 2006 because (in part) of a
lack of trust and internal validity of polling results. Naturally,
there are only a small handful of firms that have the capacity to
conduct polling in Iraq today. If accountability is lacking in some
of these firms, that can lead to validity issues in the
- Response rates for polls become more important in
dangerous areas. The purpose of polling is to get a
representative sample of opinion from a broad cross section of the
population. If a large proportion of the population refuses to be
surveyed for one reason or another, this can substantially affect
the reliability of poll results. The BBC/ABC News poll shows how.
Roughly 2,200 Iraqis were surveyed, which itself is a reasonable
number of people to poll; however, the polling company had to
contact many more households than that in order to get the 2,200.
According to the methodological note, there was a response rate of
only about 60 percent on the poll. Therefore, nearly 1,500
households that were contacted did not participate, mostly because
they refused to do so.
It is impossible to tell how the 60 percent who participated with
the poll are similar or different to the 40 percent who did not
participate. For example, are the 60 percent more likely to
complain about the situation in Iraq? Are they more likely to be
Shia (or conversely Sunni)? Were non-responders adequately replaced
by individuals with similar characteristics? Again, it is
impossible to tell. This introduces error into the results that
usually cannot be adequately fixed via data analysis, especially
when data is from unstable areas like Iraq.
While opinion polls can be a useful tool in gauging public
opinion, polling in areas such as Iraq presents a host of
difficulties-especially the uncertainty inherent in the polls and
the near-impossibility of basic checks for validity. Such polls
should not be used to conduct policy, but rather to inform on
trends in the population. Policymakers should, therefore, approach
such polls with caution during the upcoming debate on Iraq.
Johnson, Ph.D., is a Visiting Fellow in the Center for Data
Analysis at The Heritage Foundation and served as Deputy
Director for Assessments in the Joint Strategic Planning and
Assessment office at the U.S. Embassy in Baghdad in
Alissa J. Rubin, "For Iraqis, General's Report Offers Bitter Truth"
The New York Times
, September 11, 2007, at www.nytimes.com/2007/09/11/world/middleeast/11cnd-reax.html
Ambassador Ryan C. Crocker, United States Ambassador to the
Republic of Iraq, "Transcript of the Hearing to the Committee on
Foreign Affairs and the Committee on Armed Services of the U.S.
House of Representatives," September 10, 2007.
Department of Defense, "Measuring Security and Stability in Iraq,"
Report to Congress in accordance with the Department of Defense
Appropriations Act of 2007 (Section 9010, P.L. 109-289), June 2007,
pg. 26, at www.defenselink.mil/pubs/pdfs/9010-Final-20070608.pdf.
The military will likely submit an update to this report by the end
of September, as required by law.
BBC/ABC News Iraq Poll, released September 10, 2007.