Opinion polls can provide some insight into the thoughts of broad populations and have been used to gauge public perception for decades in the United States and other Western countries. Since shortly after the start of the conflict in Iraq, household surveys have been conducted by a variety of news outlets, non-governmental organizations, and other groups in order to ascertain what Iraqis think about the major issues of the day. Recently, some of these polls have been introduced in the Iraq debate on Capitol Hill.

I was Deputy Director for Assessments at the U.S. Embassy in Baghdad between July 2006 and August 2007. In that position, I analyzed a great number of opinion surveys for Ambassador Crocker, senior leaders at the embassy, and other interested personnel. Over the course of those many months, I came to appreciate the difficulties of polling in such an environment. Policymakers should approach such polls with caution and should use them to measure trends in the population rather than to conduct policy.  

Issues with Polling in Iraq
The following six issues must be considered when analyzing the results of any Iraqi survey.

1. Iraqis typically have negative reactions and opinions when asked about the American occupation. The reasoning for this is simple: Iraq is a sovereign country, and the Iraqis want to be able to determine their own affairs. Large segments of Iraq have, since the start of the conflict, been opposed to the troop presence.
   
   Recently, The New York Times captured this sentiment when a city worker in Baquba, in Diyala Province, forcefully said, "I want them to withdraw all their troops in one day."
   
   

   That notwithstanding, the Sunni worker continued:There is something that I want to say although I hate to say it. The Americans forces, which are an ugly occupation force, have become something important to us, the Sunnis. We are a minority and we do not having a force to face the militias. If the Americans leave, it will mean a total elimination of the Sunnis in Iraq.
   
   I know I said I want them to leave, but if we think about it, then I have to say I want them to stay for a while until we end all the suspicions we have of each other and have a strong national government.[1]

   There is a sense in Iraq that the security situation would worsen if Coalition Forces (CF) leave too soon. Ambassador Crocker noted during the September 10, 2007, hearing, "All of Iraq's principal leaders…don't want to see any marked precipitous reduction in how those [CF military] forces are deployed until conditions sustain it." If the Iraqi people truly wanted U.S. and CF troops to leave Iraq immediately, their leaders would force the issue. Instead, its leaders have encouraged CF troops to stay.
   
   
2. Security is a lagging indicator and tends to be better within neighborhoods as compared to outside neighborhoods. People will not feel a broad sense of security until sufficient time passes for perspective. Better security has generally been a recent phenomenon. During the House and Senate hearings, General Petraeus indicated that security incidents were down in eight of the last 12 weeks. It may take time for this improvement to be reflected in polling data.
   
   The security data shows that Baghdad became more secure earlier in 2007 as compared to the rest of Iraq. The surge first focused on setting up roughly 30 joint security stations around the city staffed with additional CF troops. As a result, the capital became more secure sooner. Ambassador Crocker noted during the House hearing, "I have seen other national polling data that shows, for example, that the number of Iraqis who now feel secure in their own neighborhoods and indeed feel secure moving around the city has gone up significantly."[2] In fairness, Ambassador Crocker indicated thereafter that he does not know of the accuracy of the polling. That said, he understands that it is difficult to poll in conflict areas such as Baghdad unless it is done very carefully.
   
   
3. Opinions tend to vary significantly across locations and ethno-sectarian groups. Iraq, by its very nature, is not a monolith. Public opinion tends to differ not only by the three major ethno-sectarian groups (Shia, Sunni, and Kurd), but also by geography. Take, for example, some polling reported by the military in their latest "Measuring Security and Stability in Iraq" report.[3] The military reported some results of a nationwide poll taken in April 2007. One question asked respondents to agree or disagree with the following statement: "I feel safe and secure in my neighborhood." While most Iraqis in the Kurdish areas and in most of the Shia-dominated south agreed with that statement, there was less agreement in the Sunni provinces just north and west of Baghdad. Even though 77 percent of Iraqis nationwide agreed with the statement, the distribution of answers clearly varies by geography.
   
   Poll responses differ by ethno-sectarian group as well. Different communities often have varying views on a wide range of subjects. Obtaining separate results for Kurdish, Shia, and Sunni populations would paint a better picture of opinions than simply polling Iraqis at large. One such example of ethno-sectarian differences is shown in a recent BBC/ABC poll. Shia Arabs are more likely to believe that the security situation in their neighborhoods is good, as compared with Sunni Arabs, by a margin of 46 percent to 21 percent. (There were no figures for Kurds, although the polling methodology suggests that they were surveyed.)[4]
   
   
4. Iraqi polls should be analyzed via trends rather than snapshots in time. Because of the difficulties in polling in unstable areas such as Iraq (see below), static percentages or other results may not have the kind of reliability that Western polls enjoy. Therefore, most analysts look at trends of polls. The actual level of opinion is secondary to whether opinion is staying the same, increasing, or decreasing over time.
   
   
5. No matter how good the polling methodology is, there can be problems with polls when there is distance between the organization commissioning the poll and the interviewers themselves. Many polls that are fielded in dangerous places are already difficult to complete because of the security situation. In addition, the actual interviewers may not be conducting the poll as the methodology would dictate. In Western countries, after the polling contractor conducts the poll, the organization that commissioned the poll can check to be sure that the polling company completed the interviews as specified by the survey methodology. This is not possible for polls where there is such a firewall between the polling company and the organization that commissioned the poll.
   
   By way of example, the BBC/ABC News poll was not actually conducted by either of the news organizations themselves. Rather, the interviews were conducted by Iraqis. This is appropriate for security reasons. The downside, however, is that basic polling validity checks of the data cannot be undertaken. In the United States, a good survey methodology would allow for the originating organization to contact households to verify that someone came by to ask questions. In a conflict environment, the poll becomes a "black box" whereby the organization commissioning the poll must trust the results without verification.
   
   At least one organization, the International Republican Institute, cancelled a polling contract in Iraq in 2006 because (in part) of a lack of trust and internal validity of polling results. Naturally, there are only a small handful of firms that have the capacity to conduct polling in Iraq today. If accountability is lacking in some of these firms, that can lead to validity issues in the polls.
   
   
6. Response rates for polls become more important in dangerous areas. The purpose of polling is to get a representative sample of opinion from a broad cross section of the population. If a large proportion of the population refuses to be surveyed for one reason or another, this can substantially affect the reliability of poll results. The BBC/ABC News poll shows how. Roughly 2,200 Iraqis were surveyed, which itself is a reasonable number of people to poll; however, the polling company had to contact many more households than that in order to get the 2,200. According to the methodological note, there was a response rate of only about 60 percent on the poll.[5] Therefore, nearly 1,500 households that were contacted did not participate, mostly because they refused to do so.
   
   It is impossible to tell how the 60 percent who participated with the poll are similar or different to the 40 percent who did not participate. For example, are the 60 percent more likely to complain about the situation in Iraq? Are they more likely to be Shia (or conversely Sunni)? Were non-responders adequately replaced by individuals with similar characteristics? Again, it is impossible to tell. This introduces error into the results that usually cannot be adequately fixed via data analysis, especially when data is from unstable areas like Iraq.

Conclusion
While opinion polls can be a useful tool in gauging public opinion, polling in areas such as Iraq presents a host of difficulties-especially the uncertainty inherent in the polls and the near-impossibility of basic checks for validity. Such polls should not be used to conduct policy, but rather to inform on trends in the population. Policymakers should, therefore, approach such polls with caution during the upcoming debate on Iraq.

Kirk A. Johnson, Ph.D., is a Visiting Fellow in the Center for Data Analysis at The Heritage Foundation and served as Deputy Director for Assessments in the Joint Strategic Planning and Assessment office at the U.S. Embassy in Baghdad in 2006-2007.