Opinion polls can provide some insight into the thoughts of broad populations and have been used to gauge public perception for decades in the United States and other Western countries. Since shortly after the start of the conflict in Iraq, household surveys have been conducted by a variety of news outlets, non-governmental organizations, and other groups in order to ascertain what Iraqis think about the major issues of the day. Recently, some of these polls have been introduced in the Iraq debate on Capitol Hill.
I was Deputy Director for Assessments at the U.S. Embassy in Baghdad between July 2006 and August 2007. In that position, I analyzed a great number of opinion surveys for Ambassador Crocker, senior leaders at the embassy, and other interested personnel. Over the course of those many months, I came to appreciate the difficulties of polling in such an environment. Policymakers should approach such polls with caution and should use them to measure trends in the population rather than to conduct policy.
Issues with Polling in Iraq
The following six issues must be considered when analyzing the results of any Iraqi survey.
- Iraqis typically have negative reactions and opinions
when asked about the American occupation. The reasoning
for this is simple: Iraq is a sovereign country, and the Iraqis
want to be able to determine their own affairs. Large segments of
Iraq have, since the start of the conflict, been opposed to the
Recently, The New York Times captured this sentiment when a city worker in Baquba, in Diyala Province, forcefully said, "I want them to withdraw all their troops in one day."
That notwithstanding, the Sunni worker continued:There is something that I want to say although I hate to say it. The Americans forces, which are an ugly occupation force, have become something important to us, the Sunnis. We are a minority and we do not having a force to face the militias. If the Americans leave, it will mean a total elimination of the Sunnis in Iraq.There is a sense in Iraq that the security situation would worsen if Coalition Forces (CF) leave too soon. Ambassador Crocker noted during the September 10, 2007, hearing, "All of Iraq's principal leaders…don't want to see any marked precipitous reduction in how those [CF military] forces are deployed until conditions sustain it." If the Iraqi people truly wanted U.S. and CF troops to leave Iraq immediately, their leaders would force the issue. Instead, its leaders have encouraged CF troops to stay.
I know I said I want them to leave, but if we think about it, then I have to say I want them to stay for a while until we end all the suspicions we have of each other and have a strong national government.
- Security is a lagging indicator and tends to be better
within neighborhoods as compared to outside neighborhoods.
People will not feel a broad sense of security until sufficient
time passes for perspective. Better security has generally been a
recent phenomenon. During the House and Senate hearings, General
Petraeus indicated that security incidents were down in eight of
the last 12 weeks. It may take time for this improvement to be
reflected in polling data.
The security data shows that Baghdad became more secure earlier in 2007 as compared to the rest of Iraq. The surge first focused on setting up roughly 30 joint security stations around the city staffed with additional CF troops. As a result, the capital became more secure sooner. Ambassador Crocker noted during the House hearing, "I have seen other national polling data that shows, for example, that the number of Iraqis who now feel secure in their own neighborhoods and indeed feel secure moving around the city has gone up significantly." In fairness, Ambassador Crocker indicated thereafter that he does not know of the accuracy of the polling. That said, he understands that it is difficult to poll in conflict areas such as Baghdad unless it is done very carefully.
- Opinions tend to vary significantly across locations
and ethno-sectarian groups. Iraq, by its very nature, is
not a monolith. Public opinion tends to differ not only by the
three major ethno-sectarian groups (Shia, Sunni, and Kurd), but
also by geography. Take, for example, some polling reported by the
military in their latest "Measuring Security and Stability in Iraq"
report. The military reported some results of a
nationwide poll taken in April 2007. One question asked respondents
to agree or disagree with the following statement: "I feel safe and
secure in my neighborhood." While most Iraqis in the Kurdish areas
and in most of the Shia-dominated south agreed with that statement,
there was less agreement in the Sunni provinces just north and west
of Baghdad. Even though 77 percent of Iraqis nationwide agreed with
the statement, the distribution of answers clearly varies by
Poll responses differ by ethno-sectarian group as well. Different communities often have varying views on a wide range of subjects. Obtaining separate results for Kurdish, Shia, and Sunni populations would paint a better picture of opinions than simply polling Iraqis at large. One such example of ethno-sectarian differences is shown in a recent BBC/ABC poll. Shia Arabs are more likely to believe that the security situation in their neighborhoods is good, as compared with Sunni Arabs, by a margin of 46 percent to 21 percent. (There were no figures for Kurds, although the polling methodology suggests that they were surveyed.)
- Iraqi polls should be analyzed via trends rather than
snapshots in time. Because of the difficulties in polling
in unstable areas such as Iraq (see below), static percentages or
other results may not have the kind of reliability that Western
polls enjoy. Therefore, most analysts look at trends of polls. The
actual level of opinion is secondary to whether opinion is staying
the same, increasing, or decreasing over time.
- No matter how good the polling methodology is, there
can be problems with polls when there is distance between the
organization commissioning the poll and the interviewers
themselves. Many polls that are fielded in dangerous
places are already difficult to complete because of the security
situation. In addition, the actual interviewers may not be
conducting the poll as the methodology would dictate. In Western
countries, after the polling contractor conducts the poll, the
organization that commissioned the poll can check to be sure that
the polling company completed the interviews as specified by the
survey methodology. This is not possible for polls where there is
such a firewall between the polling company and the organization
that commissioned the poll.
By way of example, the BBC/ABC News poll was not actually conducted by either of the news organizations themselves. Rather, the interviews were conducted by Iraqis. This is appropriate for security reasons. The downside, however, is that basic polling validity checks of the data cannot be undertaken. In the United States, a good survey methodology would allow for the originating organization to contact households to verify that someone came by to ask questions. In a conflict environment, the poll becomes a "black box" whereby the organization commissioning the poll must trust the results without verification.
At least one organization, the International Republican Institute, cancelled a polling contract in Iraq in 2006 because (in part) of a lack of trust and internal validity of polling results. Naturally, there are only a small handful of firms that have the capacity to conduct polling in Iraq today. If accountability is lacking in some of these firms, that can lead to validity issues in the polls.
- Response rates for polls become more important in
dangerous areas. The purpose of polling is to get a
representative sample of opinion from a broad cross section of the
population. If a large proportion of the population refuses to be
surveyed for one reason or another, this can substantially affect
the reliability of poll results. The BBC/ABC News poll shows how.
Roughly 2,200 Iraqis were surveyed, which itself is a reasonable
number of people to poll; however, the polling company had to
contact many more households than that in order to get the 2,200.
According to the methodological note, there was a response rate of
only about 60 percent on the poll. Therefore, nearly 1,500
households that were contacted did not participate, mostly because
they refused to do so.
It is impossible to tell how the 60 percent who participated with the poll are similar or different to the 40 percent who did not participate. For example, are the 60 percent more likely to complain about the situation in Iraq? Are they more likely to be Shia (or conversely Sunni)? Were non-responders adequately replaced by individuals with similar characteristics? Again, it is impossible to tell. This introduces error into the results that usually cannot be adequately fixed via data analysis, especially when data is from unstable areas like Iraq.
While opinion polls can be a useful tool in gauging public opinion, polling in areas such as Iraq presents a host of difficulties-especially the uncertainty inherent in the polls and the near-impossibility of basic checks for validity. Such polls should not be used to conduct policy, but rather to inform on trends in the population. Policymakers should, therefore, approach such polls with caution during the upcoming debate on Iraq.
Kirk A. Johnson, Ph.D., is a Visiting Fellow in the Center for Data Analysis at The Heritage Foundation and served as Deputy Director for Assessments in the Joint Strategic Planning and Assessment office at the U.S. Embassy in Baghdad in 2006-2007.
 Alissa J. Rubin, "For Iraqis, General's Report Offers Bitter Truth" The New York Times, September 11, 2007, at www.nytimes.com/2007/09/11/world/middleeast/11cnd-reax.html
 Ambassador Ryan C. Crocker, United States Ambassador to the Republic of Iraq, "Transcript of the Hearing to the Committee on Foreign Affairs and the Committee on Armed Services of the U.S. House of Representatives," September 10, 2007.
 Department of Defense, "Measuring Security and Stability in Iraq," Report to Congress in accordance with the Department of Defense Appropriations Act of 2007 (Section 9010, P.L. 109-289), June 2007, pg. 26, at www.defenselink.mil/pubs/pdfs/9010-Final-20070608.pdf. The military will likely submit an update to this report by the end of September, as required by law.
 BBC/ABC News Iraq Poll, released September 10, 2007.