By Spencer Walrath Activists are out with a new survey that claims nearly half of Americans believe energy companies should be forced to pay for climate change-related damages. With funding from the Union of Concerned Scientists (UCS), the Yale Program on Climate Change Communication published the results of the survey earlier this week. But a review of their data and analysis reveals that numerous flawed methodological practices were built into the survey, such as priming participants to arrive at a particular conclusion. UCS has played a lead role in marshalling other activists in a broader campaign pushing climate liability litigation. Funded by the Rockefellers, the campaign traces its origins back to a 2012 conference in La Jolla, Calif., where activists plotted how to use targeted research to ascribe specific climate impacts and blame to energy companies. The UCS-commissioned survey found that a majority of Americans believe their local communities are being impacted by climate change and fossil fuel companies should be held financially liable for these impacts. The survey also claims that 50 percent of Americans, or 163 million citizens, support their local officials filing liability suits against fossil fuel companies. If those data points seem incredible, it’s because they are – in the literal sense of the word. In the context of widely-accepted best practices, the methodology employed by the survey’s architects is inherently flawed, making these claims dubious. Maps Not Based on Survey Data Yale published an interactive map alongside their survey, which they claim depicts “the percentage of Americans in each state, congressional district, metro area, and county who hold fossil fuel companies responsible for the local damage of global warming.” The only problem is, those maps aren’t actually based on local polling. UCS and Yale simply took the responses from the 5,000 people they polled nationally and fed it into an algorithm that basically took an educated guess at how people in a given state or district would have responded based on demographics. So while the media rushed to report that 56 percent of Texans, for example, supported forcing energy companies to pay for climate change, unless UCS and Yale release their data it’s impossible to tell how many Texans were actually polled and whether their answers differed substantially from the national results, especially on a niche issue like climate liability. Relying on their formula instead of actual local polling data increased the margin of error to as much as +/- 11 points, which is enormous. Steering Respondents to the Answers They Wanted Participants were also “primed” or conditioned to answer in favor of holding energy companies responsible for climate costs. When designing a survey, it is critical to consider the language being employed and the order that the questions are presented because these factors can directly impact the gathered responses. This phenomenon is called priming, and if a survey either intentionally or unintentionally fails to properly account for this effect, the results of the survey become questionable:  if the respondents are primed to consider one issue while answering all the subsequent questions, then their answers are going to be skewed. Even more curious is that, outside of the graphs provided in the article, the researchers have refused to publish the complete results of the survey, at least as of publication of this blog. Although the question language is provided on the “Survey Questions” tab, the full survey questionnaire is not. However, even with the limited information that was provided, one key question is a clear-cut example of priming the respondents, and naturally it was the very first question. In the research findings, the surveyors state that respondents were asked “How much responsibility do you think fossil fuel companies have for the damages caused by global warming?” However, this is not actually the full question that was asked. In its totality, participants were asked:

“Because the impacts of global warming are costly for local communities, some cities and states have filed lawsuits against fossil fuel companies to make them pay for protective measures and/or a share of the damages. How much responsibility do you think fossil fuel companies have for the damages caused by global warming?”

Before asking respondents how much they think fossil fuel companies are responsible for damages related to climate change, the questionnaire plainly asserts that global warming is costly and directly links those costs to fossil fuel companies. By framing the question like this, respondents are conditioned to view energy companies as being responsible for the damages associated with climate change – as opposed to, say, consumers who use those products, or even governments at the state and federal level that have sanctioned the production, transport, and use of energy resources. This primed question could have an even larger effect on the overall survey results because it is the first question asked to the respondents. Having established that these controversial claims are facts, any subsequent answers could be expected to yield similarly skewed results. Aggregated Data Muddies Signal Additionally, the data presented were not from a single poll. Instead, they were aggregated results from multiple rounds of polling spread across three months. Specifically, the data were collected between November 2018 and January 2019. When conducting a public opinion poll, current events can play a major role in a participant’s answers. In the context of this specific survey data, the current events playing out in November 2018 vary greatly from January 2019. Additionally, the pollsters changed the survey questions over the course of those three months, noting that “the question about who should pay for most or all of the costs of damages from global warming was included in only one survey wave.” Furthermore, the survey does not present any demographic breakdowns for the polls individually or aggregated. Referencing and making available the demographic make-up of the results is an accepted best practice, so it is concerning that this data is not easily accessible. Given the UCS funding and questionable methodology used to produce the survey, the results should be viewed with skepticism. For a topic as nuanced and complex as addressing climate change, reporters and policymakers should be wary of activists bearing polling “data.”