We welcome the initiatives within the global financial system to examine acute risks associated with physical climate change and we strongly concur that acute risks associated with weather and climate threaten elements of the financial system. Using physical climate models to examine large-scale risks or as guides for scenario or storyline planning is useful, and using reduced complexity models such as IAMs to develop large ensembles of how GMT responds to emission scenarios is well established. Our analysis is not examining whether acute risks are material, rather we examine the assumption, within methodologies including but not limited to NGFS, that large ensembles of GMT can be used to inform acute climate risk at spatial scales well below the sub-regional scale.

The NGFS methodology links large ensembles of GMT, via ISIMIP, to local and regional-scale climate risk. The methods used by NGFS to create large ensembles of GMT are not in question, nor are the climate models used in ISIMIP which have considerable validity for the large-scale assessment of impacts of climate change. The issue is the link implied within the NGFS methodology that translates GMT, through ISIMIP, to a granular level of physical climate risk which, in reality, is generated through climate-induced weather-scales and weather-related extremes. This link depends on the patterns simulated by the ISIMIP models, balancing the thermodynamic and dynamic responses, and their capacity to reflect the correlations between GMT and material extremes at a granular scale.

Our results show that irrespective of the capacity to derive a distribution for possible changes in GMT, and however well this distribution samples uncertainty, the methods used to link GMT to local, i.e. city-scale, annual extremes of rainfall and wind, or the return periods of two compound events, or the 1 in 20, 1 in 50 and 1 in 100 year rainfall or temperature extremes is deeply uncertain. Whether the ISIMIP models, or CMIP models are used, the translation of GMT into spatial expressions of extremes leads to uncertainty not merely in the magnitude of change, but in the sign of many changes. The uncertainty dwarfs any signal from emission scenarios, at least over the next 50 year. There are strategies to reduce the apparent uncertainty in projected extremes by sampling climate models according to skill or independence, but whether this reduces actual uncertainty, thereby enabling more robust decisions on managing risk, is unknown. Before we continue, we emphasise that the conclusion that there is no useful link between GMT and material risks does not mean that climate models have no role to play in assessing the impact of climate change on financial risk.

One of the advantages of physical climate models, including those used in ISIMIP, is that they provide easily accessible and quantitative information. Within CMIP6 for example, which include newer models than ISIMIP, a multi-petabyte store of open access climate change information exists. This is obviously very attractive to groups seeking to build approaches, or undertake analyses, that can be applied anywhere in the world. However, there are two fundamental principles to consider in using any physical climate modelling system. First, accuracy and precision are not the same thing; physical climate models are very precise, but not necessarily accurate and may not be accurate for problems they were not designed for. Second, uncertainty cannot be ignored; deep uncertainty exists in climate projections (Lempert et al 2013) and affects both the magnitude and sign of the change in most physical risks and very probably most material risks. This cannot be ignored because the consequences are not easy to predict. Ranger et al (2022) describe, for example, the stress testing run by the Bank of England (2020), noting that the input data is largely sourced via the NGFS methodology and that no uncertainty information is provided. From a physical climate projections perspective this is simply flawed. Refer to figures 2(d) and 3(a) and take any value of GMT and select the associated wind speed change or return period. Depending on which CMIP6 model and emission scenario is selected, increases, decreases or no change can be obtained. It is deeply misleading to select a single value from the ranges shown in figures 2 and 3 without also accounting for the uncertainty. Further, despite claims within NGFS (Bertram et al 2021) that the IAM used (MAGICC6) is designed 'to capture the full GMT uncertainty for different emissions scenarios', and accepting MAGICC6 is a legitimate tool to use, it is misleading to suggest it captures the full range of uncertainty. It is not known, and it is probably unknowable, to what degree any IAM captures the full range of uncertainty. The ISIMIP project is not designed to select global models that capture uncertainty, or independence (Abramowitz and Bishop 2014), or particularly good or bad models. It is simply an ensemble of opportunity (Tebaldi and Knutti 2007) with strengths and weaknesses. The ISIMIP models are legitimate tools to use, but they are quite old model versions, quite coarse in terms of spatial resolution and only six models complete the ensemble. Referring to the uncertainty bars shown in figures 4–7, selecting six CMIP models would reduce the apparent uncertainty because of the smaller sample size, but it would not reduce the actual uncertainty. It is noteworthy here that even the full CMIP6 ensemble, which now includes over 50 models, samples an unknown fraction of the true uncertainty. We also note that assessing material risks using CMIP6 (with SSPs) is unlikely to lead to more robust conclusions that using CMIP5 (with RCPs). While climate models are improving, at the spatial scales of individual cities and on time scales of decades both CMIP5 and CMIP6 provide projections that cannot be clearly differentiated.

We acknowledge that many of these issues are clearly highlighted in the literature. Bertram et al (2021) notes that 'findings from the Climate Impact Explorer should thus be used to supplement rather than replace national or regional risk assessments'. They further note that 'uncertainty in the climate sensitivity is sampled by considering four different GCMs', and that several impact models are used to sample the uncertainty'. Bertram et al (2021) also notes:

Following established approaches in the scientific literature (see e.g. James et al 2017 ), we assess impact indicators as a function of the GMT level. This means we assume that a given GMT level will on average lead to the same change in that indicator even if it is reached at two different moments in time in two different emission scenarios. This assumption is generally well justified and differences are small compared to the spread across changes projected by different models (Herger et al 2015 ).

We strongly agree with these statements and emphasize the 'on average' and 'generally well justified'. The problem is, however, that while these approaches are well justified on average, the acute physical risks and the material extremes associated with regional-scale and finer scale climate change are not well described by averages. After all, the financial sector seeks to know which specific regions are most at risk, not that a fraction of the globe is at increased risk. If financial risk is aggregated to a continent, systematic errors associated with these assumptions might be averaged out, but the NGFS methodology is being used at a granularity well below that examined in this paper. This involves very significant uncertainties and determining whether climate change results in a material extreme is country, economy and business specific. At these scales, and in the context of material extremes associated with climate-induced weather-scale phenomenon, the ways in which the NGFS methodology are being employed is very likely misleading. There is a key implication here that is deeply concerning:

If all Central Banks (or the over 100 members of NGFS) use a methodology that is systemically biased, this could itself lead to a major systemic risk to the global financial system.

The current NGFS scenarios do not represent the range of plausible climate outcomes possible at a country level—a systematic bias—and most banks, insurers and investors are using these scenarios without fully accounting for uncertainty. Misuse or misunderstanding of what climate models tell us, and assumptions that products like NGFS have utility at sub-national scales could make the risks we are trying to avoid through the NGFS scenarios worse. Rectifying this is important and requires an open collaboration between banks and the scientific community to develop scenarios appropriate for stress testing.

The most fundamental issue with assessing financial risk associated with acute physical risk relates to the acknowledgement that these risks are associated with weather, usually locally, and usually (but not necessarily) statistically extreme. The use of global climate models, which do not resolve weather-scales, are not appropriate for local scales and may not capture material extremes, is highly questionable. While using the quantitative information from climate models is tempting and provides a considerable amount of apparently precise information, failure to fully represent uncertainty leads to false confidence. By contrast, there are well-known ways to decouple assessments of acute physical risks from climate model quantitative information. Using climate models to inform scenarios, storylines (Shepherd 2019, Jack et al 2020) and stress testing, or using climate models to modify the statistics represented in current-day catastrophe modelling can all help break the false assumption that the numerical precision in climate models equates to accuracy at a granular level. In many ways, this echoes guidance from Schinko et al (2017) to consider models as tools to explore a system as distinct from predicting a system, or Saravanan (2022) who explores the need to take climate models seriously, but not literally. Given the material risks from climate change are commonly the tail risks, more use of catastrophe modelling might lead to decision making that builds more resilient systems. However, some material risks are likely associated with long periods of drizzle, or of high cloud cover and still winds. These are events associated with persistence which climate models are known to capture with relatively low skills (see for example Kumar et al 2013).

The relative ease with which large ensembles using IAMs can be generated and linked to acute risk at sub-regional scales is understandably attractive for large financial institutions, central banks and financial regulators. It is therefore unlikely that these will be wholly replaced by an alternative approach. This relative ease, however, hides immense uncertainty that is likely material, and that risks misleading an institution or regulator, exposing entities to litigation, and directly challenging centuries of accounting and assurance practice. We suggest three immediate actions:

  • (a)  
    the NGFS method is likely misleading in determining granular level acute or material risks to the financial sector and we strongly advise that it is openly critiqued and does not become a de facto standard by default.
  • (b)  
    no products or methods should be employed that fail to properly account for uncertainty, and how uncertainty is estimated needs very carefully consideration. There is no evidence that merely adding more climate models, or more estimates of GMT reduces uncertainty.
  • (c)  
    there is a rich history of assessing risk at the local scale (Ranger et al 2022). This 'bottom-up' assessment can utilize historical climate data, existing risk estimates, analysis of the vulnerability of an entity to these acute physical risks, stress testing of investment portfolios and so on. The historical data can be perturbed using expert judgement based on multiple lines of evidence, including climate models. A financial institution should confront the 'top-down' methodologies proposed by regulators with bottom-up assessments of their acute physical risks and review how different the resulting estimates are.

Perhaps the single most important point here is that while the 'top-down' approach is likely to become the de facto standard for assessing a financial institution's exposure to climate change, this should only be done in conjunction with alternative 'bottom-up' methods.

Finally, we note that climate science and the science of climate projections is evolving rapidly. Further, regulation and disclosure linked with climate risk is developing rapidly. A company with the ability to undertake, at least to some degree, a bottom-up assessment of material risks, and to engage with external parties from a position of understanding, will be well positioned as climate projections change. A company with internal capability will be more able to ask the right questions, avoid buying risk advice that is misleading, and be able to identify opportunities associated with climate change more quickly. While building some internal capability might seem confronting and expensive, building future strategies on information that is not understood and is potentially misleading is likely more so, and quite possibly exposes the global financial system to systemic risks of its own making.