Framing poll questions & results

July 10, 2013 – Research has shown that metaphors shape the way people reason about social & political issues – with most folk having no awareness that metaphors are influencing their thinking. This is relevant to polling, of course.

For example, one study found that if crime is framed metaphorically as a “virus”, survey respondents proposed “investigating the root causes… eradicating poverty and improving education (etc)”. But, when crime is framed as a “beast”, participants prefer enforcement and punishment.

Notably, in this study, there was only a one word difference (“virus”/”beast”) in the questions asked. Most participants said the crime statistics (which were included in the question, and the same in both cases) influenced their reasoning most. The authors of the study remarked: “These findings suggest that metaphors can act covertly in reasoning.”

“Majority say X”
“Majority say NOT X”

YouGov tested how a question’s wording shapes responses by asking different groups essentially the “same” question (but with different wording). For example:

• “The BBC licence fee costs £145.50 a year. Do you think this is good or bad value for money?”
• “The cost of the BBC licence works out at 40p a day. Do think this is good or bad value for money?”

Since 40p x 365 = £146, you’d expect roughly similar responses. In fact there was a massive difference. The poll asking the first question found that twice as many people thought the BBC was bad value (27% good, 54% bad). The poll using the second question found a majority saying the BBC was good value (44% good, 36% bad).

There’s no obvious difference in terms of metaphor here, but the large shift in response suggests that different cognitive frames are activated in each case – perhaps the larger (yearly) sum “reminds” people of money they need (eg to pay utilities bills). Work in ‘behavioural economics’, by the likes of Dan Ariely, has catalogued similar examples.

Here’s another example, reported by the New York Times, of a simple change in poll wording that dramatically changed the responses:

“Seventy-nine percent of Democrats said they support permitting gay men and lesbians to serve openly [in the military]. Fewer Democrats however, just 43 percent, said they were in favor of allowing homosexuals to serve openly.” (NYT, 11/2/2010)

As has been commented, this example probably isn’t surprising, as the wording evokes different frames, one about human rights, and the other about sex.

Framing poll results

So, small changes in wording can produce very different responses. And that’s just in the question asked. What about different framings of the results (eg by the news media)? Peter Kellner, the journalist & President of YouGov, makes the following comment:

The results frequently arouse media interest. Indeed we are often commissioned to ask stark questions in order to generate bold headlines and stark findings […]. It’s not that these headlines or allegations are wrong, but they are often too crude. A single question, or even a short sequence of questions, will seldom tell us all we need to know. (Peter Kellner, 24/10/2011)

But it’s not just the mass media which promotes simplistic conclusions based on crude polling. The “public interest” website, Spinwatch (of all people) recently did something similar…

Even SpinWatchers spin?

A Spinwatch blog commented on a poll which asked people in the UK to estimate the number of Iraqis who “died as a consequence of the war that began in Iraq in 2003”. The poll itself seems genuinely shocking: 59% of the respondents estimated that fewer than 10,001 Iraqis died as a result of the war.

An obvious question: Where did these low estimates originate? – since they are far lower than figures reported from Iraq Body Count or the Lancet-published surveys, etc. (Or were they just ignorant guesses from people too embarrassed to select the “Don’t know” option?)

Unfortunately, the poll doesn’t provide any answers to these questions, as it was limited (for cost reasons) to just two poll questions, neither of which indicates sources of estimates or media preferences of respondents, etc. But this didn’t stop the Spinwatch blogger from making a sweeping conclusion:

“The poll results are a striking illustration of how a ‘free press’ imposes ignorance on the public in order to promote war.” (Spinwatch, 4/6/2013)

Of course, it doesn’t follow. The poll says precisely nothing about the press. The blogger’s conclusion that the press “imposes ignorance” is based on his own presumptions about the effects of the press – not on the poll findings.

I return to notion that the press “imposes ignorance” below.

(Spinwatch published a follow-up piece with some media searches, apparently showing unbelievably few mentions of the Lancet Iraq studies – eg only 13 results for “All English Language News”, since 1/12/04, from a Lexis-Nexis search. This is clearly wrong, and, in fact, the last paragraph – of an addendum to the piece – briefly notes that “searching ‘Lancet AND Iraq’ with Lexis Nexis turns up 2602 articles since December 1, 2004”. But the Spinwatch author doesn’t present this as a correction to his earlier seemingly botched search-term format which yielded just 13 articles. Rather, he writes: “As with any search, the results can be tweaked by modifying search terms slightly”!)

Causal metaphors – a digression

Reports of poll results (in common with headlines in general) often use direct causation metaphors to frame complex social issues. All such metaphors have their own logic, which is transferred from the physical realm of force to the more abstract social realms of institutions, politics, beliefs, etc. The effect is inescapably “reductive”, but not necessarily illegitimate (some metaphors – and their imported logics – are more appropriate than others). Here are some examples of such metaphorical causal expressions:

• Public generosity hit by immigrant wave
• 72% believe Iraq on path to democracy
• Obama’s leadership brought the country out of despair
• Majority fear Vietnam will fall to communism

Each of the causal logics here is different – for example, the notion that one country “falls” to communism, while another takes the right “path” (to democracy). Of “falling to communism”, Lakoff & Johnson remark (Philosophy in the Flesh, p172) that the ‘domino effect’ theory was used to justify going to war with Vietnam: when one country “falls”, the next will, and the next – unless force (military might) is applied to stop the “falling”. The metaphor of taking a “path” has very different political entailments. A nation might not even resemble a democracy, but if it chooses the “right path”, it “deserves” US military and economic “aid”, to help overcome any obstacles put in its “way”. (Incidentally, rightwing ideologues regard any “move” towards “free market” economics as taking the “path” to democracy).

The different types of causal logic resulting from each metaphor may seem obvious when spelt out like this. But the point is that the reasoning in each case is evoked automatically by the metaphorical frame; it takes effect without being spelt out, without being “made conscious”. Rather, the logic – including political inferences – is an entailment of a frame that’s simply activated by the language used.

“Imposes ignorance”

The notion that the press “imposes ignorance on the public” is also metaphorical (although this perhaps isn’t as obvious as in the above examples). The question is whether we regard it as valid and appropriate for 21st century media – given the increasing levels of information access. It takes less than a minute, for example, for anyone with an internet connection to google “Iraq war deaths”. Such a search immediately returns the BBC article, Iraq war in figures, which cites Iraq Body Count, UN-backed IFHS, and Lancet studies, and their figures.

(BBC headlined with the 2006 Lancet study – on BBC1 News and BBC2 Newsnight – on the day of its publication, published a “question and answer” piece with one of the study’s authors (Les Roberts) and conducted an investigation – using a Freedom Of Information request – showing that the government’s scientific advisers privately stated that “The study design is robust and employs methods that are regarded as close to ‘best practice’ in this area”.)

None of this fits the notion of a media which “imposes ignorance on the public”. That’s not to say that the “news” media isn’t a determining factor in “public ignorance” (in various complex ways – several of them explored in the cognitive framing literature and in the work of Tversky, Kahneman and others on heuristics and biases, etc). But to conceive of it as forceful restriction (“imposes”) seems a fundamental misunderstanding of how the media works in the 21st century – not to mention how people acquire knowledge and form opinions in an information-saturated world of competing frames.

The Spinwatch piece notes that “Rumsfeld AND Iraq” yielded more search results in a 3-week period than “Lancet AND Iraq” did over 8.5 years – and the author concludes that, “There is simply no honest way to absolve the establishment media for imposing ignorance on the public”. But if there were a simple (inverse) correlation between number of media mentions and public ignorance, you’d expect the “public” to be relatively knowledgable about what Donald Rumsfeld said and did regarding Iraq.

That would take another poll to determine, but I suspect that public indifference/ignorance on Iraq (if that’s what the above poll illustrated) extends to what Rumsfeld said and did – regardless of the media’s apparent over-representation of Rumsfeld.

Graphics by NewsFrames