National Climate Assessment: A crisis of epistemic overconfidence
4th National Climate Assessment (NCA4) is published in two volumes:
- Vol I: Climate Science Special Report
- Vol II: Impacts, Risks, and Adaptation in the United States
- provide a more rational assessment of the confidence that should be placed in these findings
- provide motivation and a framework for doing a better job on the next assessment report.
“Longer-term climate records over past centuries and millennia indicate that average temperatures in recent decades over much of the world have been much higher, and have risen faster during this time period, than at any time in the past 1,700 years or more, the time period for which the global distribution of surface temperatures can be reconstructed. (High confidence)” This statement really struck me, since it is at odds with the conclusion from the IPCC AR5 WG1 Chapter 5 on paleoclimate: “For average annual NH temperatures, the period 1983–2012 was very likely the warmest 30-year period of the last 800 years (high confidence) and likely the warmest 30-year period of the last 1400 years (medium confidence). While my knowledge of paleoclimate is relatively limited, I don’t find the AR5 conclusion to be unreasonable, but it seems rather overconfident with the conclusion regarding the last 1400 years. The NCA4 conclusion, which is stronger than the AR5 conclusion and with greater confidence, made me wonder whether there was some new research that I was unaware of, and whether the authors included young scientists with a new perspective. Fortunately, the NCA includes a section at the end of each Chapter that provides a traceability analysis for each of the key conclusions: “Traceable Accounts for each Key Finding: 1) document the process and rationale the authors used in reaching the conclusions in their Key Finding, 2) provide additional information to readers about the quality of the information used, 3) allow traceability to resources and data, and 4) describe the level of likelihood and confidence in the Key Finding. Thus, the Traceable Accounts represent a synthesis of the chapter author team’s judgment of the validity of findings, as determined through evaluation of evidence and agreement in the scientific literature.” Here is text from the traceability account for the paleoclimate conclusion: “Description of evidence base. The Key Finding and supporting text summarizes extensive evidence documented in the climate science literature and are similar to statements made in previous national (NCA3) and international assessments. There are many recent studies of the paleoclimate leading to this conclusion including those cited in the report (e.g., Mann et al. 2008; PAGES 2k Consortium 2013).” “Major uncertainties: Despite the extensive increase in knowledge in the last few decades, there are still many uncertainties in understanding the hemispheric and global changes in climate over Earth’s history, including that of the last few millennia. Additional research efforts in this direction can help reduce those uncertainties.” “Assessment of confidence based on evidence and agreement, including short description of nature of evidence and level of agreement : There is high confidence for current temperatures to be higher than they have been in at least 1,700 years and perhaps much longer. I read all this with acute cognitive dissonance. Apart from Steve McIntyre’s takedown of Mann et al. 2008 and PAGES 2K Consortium (for the latest, see PAGES2K: North American Tree Ring Proxies), how can you ‘square’ high confidence with “there are still many uncertainties in understanding the hemispheric and global changes in climate over Earth’s history, including that of the last few millennia”? Further, Chapter 5 of the AR5 includes 1+ pages on uncertainties in temperature reconstructions for the past 200o years (section 188.8.131.52), a few choice quotes:I checked the relevant references in the NCA4 Chapter 1; only two (Mann et al., 2008; PAGES 2013), both of which were referenced by the AR5. The one figure from this section was from — you guessed it — Mann et al. (2008). I next wondered: exactly who were the paleoclimate experts that came up with this stuff? Here is the author list for Chapter 1: Wuebbles, D.J., D.R. Easterling, K. Hayhoe, T. Knutson, R.E. Kopp, J.P. Kossin, K.E. Kunkel, A.N. LeGrande, C. Mears, W.V. Sweet, P.C. Taylor, R.S. Vose, and M.F. Wehner I am fairly familiar with half of these scientists (a few of them I have a great deal of respect for), somewhat familiar with another 25%, and unfamiliar with the rest. I looked these up to see which of them were the paleoclimate experts. There are only two authors (Kopp and LeGrande) that appear to have any expertise in paleoclimate, albeit on topics that don’t directly relate to the Key Finding. This is in contrast to an entire chapter in the IPCC AR5 being devoted to paleoclimate, with substantial expertise among the authors. A pretty big lapse, not having an expert on your author team related to one of 6 key findings. This isn’t to say that a non-expert can’t do a good job of assessing this topic with a sufficient level of effort. However the level of effort here didn’t seem to extend to reading the IPCC AR5 Chapter 5, particularly section 184.108.40.206. Why wasn’t this caught by the reviewers? The NCA4 advertises an extensive in house and external review process, including the National Academies. I took some heat for my Report On Sea Level Rise and Climate Change, since it had only a single author and wasn’t peer reviewed. Well, the NCA provides a good example of how multiple authors and peer review is no panacea for providing a useful assessment report. And finally, does this issue related to whether current temperatures were warmer than the medieval warm period really matter? Well yes, it is very important in context of detection and attribution arguments (which will be the subject of forthcoming posts). This is but one example of overconfidence in the NCA4. What is going on here? Confidence guidance in the NCA4 Exactly what does the NCA4 mean by ‘high confidence’? The confidence assessment used in the NCA4 is essentially the same as that used in the IPCC AR5. From the NCA4: “Confidence in the validity of a finding based on the type, amount, quality, strength, and consistency of evidence (such as mechanistic understanding, theory, data, models, and expert judgment); the skill, range, and consistency of model projections; and the degree of agreement within the body of literature.” “Assessments of confidence in the Key Findings are based on the expert judgment of the author team. Confidence should not be interpreted probabilistically, as it is distinct from statistical likelihood. “These descriptions for each confidence category don’t make sense to me; the words ‘low’, ‘medium’ etc. seem at odds with the descriptions of the categories. Also, I thought I recalled a ‘very low’ confidence category from the IPCC AR5 (which is correct link). The AR5 uncertainty guidance doesn’t give verbal descriptions of the confidence categories, although it does include the following figure:
“Reconstructing NH, SH or global-mean temperature variations over the last 2000 years remains a challenge due to limitations of spatial sampling, uncertainties in individual proxy records and challenges associated with the statistical methods used to calibrate and integrate multi-proxy information” “A key finding is that the methods used for many published reconstructions can underestimate the amplitude of the low-frequency variability”
“data are still sparse in the tropics, SH and over the oceans”
“Limitations in proxy data and reconstruction methods suggest that published uncertainties will underestimate the full range of uncertainties of large-scale temperature reconstructions.” Heck, does all this even justify the AR5’s ‘medium’ confidence level?
- ‘High confidence’ is described as ‘Moderate evidence, medium consensus.’ The words ‘moderate’ and ‘medium’ sound like ‘medium confidence’ to me.
- ‘Medium confidence’ is described as ‘Suggestive evidence (a few sources, limited consistency, models incomplete, methods emerging); competing schools of thought.’ Sounds like ‘low confidence’ to me.
- ‘Low confidence’ is described as inconclusive evidence, disagreement or lack of opinions among experts. Sounds like ‘no confidence’ to me.
- ‘Very high confidence’ should be reserved for evidence where there is very little chance of the conclusion being reversed or whittled down by future research; findings that have stood the test of time and a number of different challenges.
- Personal explanations attribute error to the personal qualities of individuals or groups of individuals. Carelessness, gullibility, closed-mindedness, dogmatism, and prejudice and wishful thinking are examples of such qualities. These qualities are epistemic vices.
- Sub-personal explanations attribute error to the automatic, involuntary, and non-conscious operation of hard-wired cognitive mechanisms. These explanations are mechanistic in a way that personal explanations are not, and the mechanisms are universal rather than person-specific.
- Situational explanations attribute error to contingent situational factors such as time pressure, distraction, overwork or fatigue.
- Systemic explanations attribute error to organizational or systemic factors such as lack of resources, poor training, or professional culture.
I have written multiple blog posts previously on strategies for addressing overconfidence, including:
- Red Teams
- The method of multiple working hypotheses
- Cognitive bias – how petroleum scientists deal with it
- I know I’m right(?) – the best cure for overprecision is continual challenges of “How could I be wrong?”
- Certainly not! – on cultivating doubt and finding pleasure in mystery.
- Italian Flag – three-valued logic that explicitly includes unknowns.
- The lure of incredible certitude
- We are all confident idiots.
- It pays to be overconfident, even if you have no idea what you are doing
- the general overconfidence issue
- additional examples (with documentation) of unjustified, overconfident conclusions (e.g. relative to the AR5)