Poorly Defined Questions Lead to Meaningless Poll Results

Poorly Defined Questions Lead to Meaningless Poll Results, by Indur Goklany
Master Resource, January 23, 2009

The American Geophysical Union’s house organ, Eos, has an article entitled “Examining the Scientific Consensus on Climate Change,” written by Peter Doran and Kendall Zimmerman of the University of Illinois at Chicago. (h/t Roger Pielke, Sr.)

The paper explains that the two “primary questions” asked were:

1. When compared with pre-1800s levels, do you think that mean global temperatures have generally risen, fallen, or remained relatively constant?
2. Do you think human activity is a significant contributing factor in changing mean global temperatures?

The article reports that over 90 percent of the responders agreed with the statement that mean global temperatures have “risen” relative to the pre-1800 levels (Question 1). But this is unremarkable. Everyone knows that in the 1700s and early 1800s the world was in the grip of the Little Ice Age, so a recovery toward normalcy alone would lead to global warming. No one I know — and I know plenty of skeptics (not yet a criminal offense, I hope) — has ever disputed this point. So the real surprise to me is that it wasn’t 99 percent–plus!

With respect to the second question, the authors report that 82 percent agreed that “human activity is a significant contributing factor” in the increased global temperature.

But what precisely does this mean? How did respondents interpret the terms “human activity” and “significant contributing factor”?

I am reminded that under the Clean Air Act, an area that has relatively clean air is deemed to be “significantly deteriorated” if the concentration of an air pollutant in that area increases by 2–8% of the National Ambient Air Quality Standard. Under this interpretation, a contribution of 2–8% would be significant! So the question is: What instructions or guidance, if any, was provided to respondents in interpreting what precisely was meant by “significant contributing factor” in the second question? Is a 2% contribution significant? Why not 20%, or 40%, or 60%?

In addition, what exactly is the “human activity” that is supposed to be significantly contributing to global warming? Is it land clearance, paving over surfaces, producing black-carbon emissions, generating well-mixed greenhouse gases, or a combination of all these activities? I claim that most people would interpret “human activity” to encompass “all of the above.”

Moreover, how do the researchers know that there is a uniform understanding of the question, and what precisely is that uniform understanding?

Given that the two key terms are imprecisely defined, a person who believed that well-mixed greenhouse gases have contributed no more than, say, 10% to the global warming since 1800 might be compelled to agree that “human activity is a significant contributing factor” to post-1800 global warming.

From the perspective of policy makers, to whom this article is partly addressed, identifying the precise kind of human activity that contributes significantly to global warming (however defined) is absolutely critical. If well-mixed greenhouse gases (GHGs), for example, carbon dioxide, do not contribute significantly, then no amount of GHG reductions will make a significant difference (however defined) to the degree of future warming. Thus, the imprecise wording of Question 2 precludes this poll from closing the debate among policy makers.

The fact that the article doesn’t raise, let alone address, the issues posed by its inexact question indicates that the Eos article is premature, if not erroneous, in its claim that “the debate on the authenticity of global warming and the role played by human activity is largely nonexistent among those who understand the nuances and scientific basis of long-term climate processes.” It also reflects poorly on the pollsters and Eos reviewers — assuming it was reviewed — that they missed such obvious difficulties with the study.

The article also claims that ‘[t]he challenge, rather, appears to be how to effectively communicate this fact to policy makers and to a public that continues to mistakenly perceive debate among scientists.”

But the real challenge is: How can pollsters accurately communicate their questions to potential respondents so that there is a common and precise understanding on everyone’s part regarding the question posed. Without this, one is condemned to obtain meaningless poll results. And the rest of the world is subjected to reading (and reading about) one more poorly designed study.