Are Rating Agencies Powerful? An Investigation into the Impact and Accuracy of Sovereign Ratings. By John Kiff, Sylwia Nowak, and Liliana Schumacher
IMF Working Paper WP/12/23
Jan 2012
http://www.imfbookstore.org/ProdDetails.asp?ID=WPIEA2012023
Abstract
IMF Working Paper WP/12/23
Jan 2012
http://www.imfbookstore.org/ProdDetails.asp?ID=WPIEA2012023
Abstract
We find that Credit Rating Agencies (CRA)’s opinions have an impact in the cost of funding of sovereign issuers and consequently ratings are a concern for financial stability. While ratings produced by the major CRAs perform reasonably well when it comes to rank ordering default risk among sovereigns, there is evidence of rating stability failure during the recent global financial crisis. These failures suggest that ratings should incorporate the obligor’s resilience to stress scenarios. The empirical evidence also supports: (i) reform initiatives to reduce the impact of CRAs’ certification services; (ii) more stringent validation requirements for ratings if they are to be used in capital regulations; and (iii) more transparency with regard to the quantitative parameters used in the rating process.
Excerpts
I. INTRODUCTION
1. Recent rating activities by the Credit Rating Agencies (CRAs) have induced some to ask whether ratings represent accurate risk assessments and to question how influential they are. The contention that ratings represent accurate default risk metrics was brought into question by the sheer volume and intensity of the multiple downgrades to U.S. mortgage-related structured finance securities in the wake of the crisis. Voices have also been raised against the timing of recent downgrades of European sovereigns amidst criticism that these downgrades promoted uncertainty in financial markets, leading to “cliff effects” and as a consequence affect their ability to funding themselves. Rating agencies have also been accused of behaving oligopolistically.
2. These criticisms are not new. CRAs’ downgrading actions have been accused before of not being timely but instead procyclical. It was argued that “the Mexican crisis of 1994-95 brought out that credit rating agencies, like almost anybody else, were reacting to events rather than anticipating them” (Reisen, 2003). During the late 1990s Asian crisis, CRAs were also blamed of downgrading East Asian countries too late and more than the worsening in these countries’ economic fundamentals justified, exacerbating the cost of borrowing.
3. The goal of this paper is to assess these concerns. We first examine CRAs’ role and whether CRAs are influential or just lag the market once new information is available and priced into fixed income securities. This is an important point. If CRAs influence the market, their opinions are important from a financial stability perspective. If they do not and just reflect information available to the market, their actions are not relevant and there is no policy concern. In this regard, we test three hypotheses regarding the services that CRAs provide to the market: information, certification and monitoring. We find evidence that CRAs’ opinions are influential and favor the information and certification role. We then attempt to determine what ratings actually measure and how accurate they are. We conclude with some policy recommendations based on these findings. This study is limited to sovereign ratings (of emerging markets and advanced economies) and covers the period January 2005-June 2010.
4. Our analysis of the interaction between the market and CRAs indicates that ratings have information value beyond the information already publicly available to the market. Specifically, the following results were evident:
- An event study shows that negative credit warnings (i.e., “reviews,” “watches,” and “outlooks”) have a significant impact on CDS spreads. This evidence is also supported by a Granger-causality test that finds that negative credit warnings Granger-cause changes in CDS spreads. These findings are consistent with the view that rating agencies do provide additional information to the markets, in addition to what is publicly available and used by markets to price fixed income securities.
- Although upgrades and downgrades in general do not have a significant impact on CDS spreads, upgrades and downgrades in and out of investment grade categories are statistically significant. This supports the view that the certification services provided by rating agencies do matter and likely create a purely liquidity effect (e.g. purchases and sales of assets forced by regulations or other formal mandates and not based on the additional information already in the market).
- We do not find evidence in favor of the most important testable implication of the monitoring services theory. The impact of downgrades preceded by an outlook review in the same direction is not statistically significant.
- From an informational point of view, the market appears to discriminate more than rating agencies among different kinds of issuers—in particular at lower rating grades and during crisis periods. This finding may indicate that ratings need to incorporate more granularity and leads to the second question of what ratings measure and how accurate they are.
- A common element among ratings by the major CRAs is that they represent a rank ordering of credit risk. This ordering is based on qualitative and quantitative inputs such as default probabilities, expected losses, and downgrade risk. However, there is no oneto- one mapping between any of these quantitative measures of credit risk and credit ratings. There is also no disclosure of the quantitative parameters that characterize each rating grade. For this reason, validation tests undertaken by outsiders can only apply to the ability of ratings to differentiate potential defaulters and non defaulters, but not to estimating cardinal measures such as default probabilities.
- The point highlighted above implies that—in spite of playing a similar role to internal ratings in the Basel II internal ratings-based (IRB) approach—ratings produced by the CRAs are subject to lower validation standards than are the banks using the IRB approach. In the Basel II IRB approach, financial institutions use measures of default probabilities (PD), losses given default (LGD) and exposure at default (EAD) to produce their internal ratings and are subject to calibration tests. Although validation is foremost the responsibility of banks, both bank risk managers and bank supervisors need to develop a thorough understanding of validation methods in evaluating whether banks’ rating systems comply with the operating standards set forth by Basel II.
- Ratings produced by the major CRAs perform reasonably well when it comes to rank ordering default risk among sovereigns, i.e. defaults tend to take place among the lowest rated issuers. Accuracy ratios (AR) indicate that agencies are more successful at sorting out potential defaulters among sovereign issuers (average ARs in the 80 to 90 percent range) than among corporate and structured finance issuers (average ARs in the 63 to 87 percent range), the latter ones having suffered a strong deterioration over the global financial crisis. For all classes of products though, the ARs indicate that sovereign rating accuracy deteriorates as the evaluation horizon increases.
- In general, long-term credit transition matrices show that higher ratings are more stable than lower ones, and tend to remain unchanged. But this was not the case during the global crisis period, when there has been a tendency to see heavier downgrade activity among higher-rated sovereigns than among lower-rated ones. There has been evidence of significant rating failure (defined here as three or more rating changes in one year) during the recent global financial crisis, although less than in the Asian crisis.
IV. SOME POLICY RECOMMENDATIONS
- Based on the evidence of the impact of the CRA’s certification services, the removal of the excessive reliance of regulations on ratings is warranted. This will not affect the information value of ratings—that appears to work mostly through outlook reviews—and will help lessen the additional liquidity impact due to the need to meet regulations, reducing potential cliff effects.
- To the extent that ratings continue to play a significant role in regulations, an issue arises as to whether CRAs should be more transparent about the quantitative measures they calibrate in the rating process (PDs, LGD, and stability assumptions), how these measures are mapped into ratings, and whether the final ratings can be used to infer the parameters used to obtain these measures. This is particularly relevant in the use of external ratings by banks employing the standardized approach in Basel II since internal ratings systems are subject to rigorous back testing.
- Moreover, recent heavy downgrade activity suggests that ratings should embed the notion that risk is a forward looking dimension conditional on the macroeconomic scenario. In this regard, ratings should be better tied to macroeconomic conditions, including their resilience to stress scenarios.