Each year, the Federal Reserve Board conducts two related supervisory tests, as required by the 2010 Dodd–Frank Wall Street Reform and Consumer Protection Act. These tests, the Dodd–Frank Act Stress Test (DFAST) and the Comprehensive Capital Analysis and Review (CCAR) are quantitative and qualitative assessments of large banks’ ability to cope with financial turmoil.REF In 2016, the Government Accountability Office (GAO) evaluated these tests and found that there were several ways to improve their effectiveness.REF Many of the 17 recommendations related in some way to transparency of these stress tests. While the Federal Reserve has acknowledged and attempted to make improvements over the past two years, the level of transparency and explanation is still lacking. Many GAO recommendations are still open. Others have been implemented but need improvement.
Federal Reserve Stress Testing
The Federal Reserve assesses whether large bank holding companies and U.S. intermediate holding companies are sufficiently capitalized to absorb losses during “stressful conditions,” while being able to continue normal daily function.REF This annual assessment includes two evaluations. First, “Dodd–Frank Act supervisory stress testing” is a quantitative evaluation of the impact of stressful economic and financial market conditions on firms’ capital. This evaluation is derived and performed under statute of the Dodd–Frank Act and under extra direction from the Federal Reserve Board.REF The DFAST is then utilized during the CCAR. This second assessment consists of a quantitative and qualitative component. Ultimately, the CCAR assesses whether firms have sufficient capital to continue making loans through tough economic times. Additionally, the Federal Reserve qualitatively assesses each firm’s capital planning processes. These evaluations are then used to determine if a firm’s capital plan is acceptable to the Federal Reserve.
The Federal Reserve claims that these assessments are important for ensuring the stability of U.S. financial markets. However, the scenarios and models employed throughout the DFAST are still unclear. First, the Federal Reserve Board still needs to provide more justification for the specification of supervisory scenarios. Second, while the DFAST models are generally described, there is only one specific model, listed in the Federal Register, for which there exists more detailed information.REF Even that model could provide more information and understanding to the public, as it is still quite a general explanation. Something as basic as the exact number and specification of models used throughout the DFAST is not readily available. Additionally, the links between the DFAST and CCAR’s qualitative evaluations still lack clarity. It is important for transparency to exist around the DFAST and CCAR so that their effect and the incentives they create for banks can be openly evaluated, not only by the Federal Reserve internally, but by independent researchers as well. There is no doubt that financial firms can perform various types of sensitivity analyses on their own, so transparency is needed to evaluate the true benefit of the Federal Reserve’s testing.
Understanding the Supervisory Scenarios
The Dodd–Frank Act requires the Federal Reserve Board to conduct a stress test of larger financial institutions each year.REF The first step of this stress test is building the scenarios the board will use to assess the strength of banking organizations and how well they can manage adverse economic conditions. The three scenarios are “baseline,” “adverse,” and “severely adverse.” The baseline scenario is an average of leading economic projections and is not a projection by the board. Each scenario contains 28 variables—16 variables describing the U.S. economy, and 12 describing four international country blocksREF (three variables for each country block).
- The baseline scenario has moderate economic growth in the U.S., as well as growth across all four country blocks.
- The adverse scenario weakens all economic activity. Long-term rates and yield curves decline rapidly. In this scenario, the U.S. experiences what the Federal Reserve calls a “moderate recession.” Real GDP falls two and one-quarter percent, while unemployment rises from under 4 percent to 7 percent. Outside the U.S., the Federal Reserve assumes a moderate recession in the euro area, a more pronounced recession in Japan, and slower growth in developing regions of Asia.
- The severely adverse scenario assumes a severe global recession and global aversion to long-term fixed-income assets. This hypothetical scenario induces large corrections in asset prices. This includes corporate bond and real estate markets.REF
In addition to the three scenarios, the Federal Reserve also requires the largest firms to include in their stress scenarios a “largest counterparty default” scenario. This component is intended to capture the largest potential losses and effects on capital of an unexpected counterparty default.REF Another add-on, starting in 2018, requires six firms to be subject to market risk components in “adverse” and “severely adverse” scenarios. This is another component to ensure that firms incorporate a forward-looking strategy in addressing market risk. Ultimately, these add-ons to the supervisory scenarios attempt to capture the dynamic and interconnected nature of financial firms in the U.S., but it is unclear whether these evaluations actually contribute to the analysis of the strength of the U.S. financial system.
The DFAST Models: Described, But Not Specific
In order to conduct the DFAST, the Federal Reserve employs several models to project stressed capital ratios and other necessary components for the firms subject to the supervisory testing. The Federal Reserve describes the general components they are modeling, with some brief discussion of the methods. Figuring out where each model is utilized is next to impossible, and, even more troubling, even making a specific count of models employed is not possible with the available information.REF One model, projecting the losses from corporate loan default, has a more detailed description in a rule proposed in 2017, but this standard is still in the comment period. The detailed description in the rule gives some formulas in the model, but still is not a full disclosure. After acknowledging that data and privacy concerns will force researchers to often seek or create useable, public data sets, this model specification is still not complete or replicable. However, an illustrative description of the models employed by the Federal Reserve might be the most telling when it comes to transparency:
In connection with DFAST 2018, and in addition to the models developed and data collected by federal banking regulators, the Federal Reserve used proprietary models or data licensed from the following providers: Andrew Davidson & Co., Inc.; ICE Data Services; Bloomberg L.P.; CB Richard Ellis, Inc.; CoreLogic Inc.; CoStar Group, Inc.; Equifax Information Services LLC; Fitch Ratings, Inc.; Haver Analytics; Kenneth French; IDC Financial Publishing, Inc.; Intex Solutions, Inc.; Black Knight McDash Data from Black Knight IP Holding Company, LLC; Markit Group; Moody’s Analytics, Inc.; Moody’s Investors Service, Inc.; Mergent, Inc.; Morningstar, Inc.; MSCI, Inc.; Municipal Securities Rulemaking Board; SNL Financial; StataCorp LP; S&P Global Market Intelligence: S&P Capital IQ Estimates; Standard & Poor’s Financial Services LLC; and World Bank Group. In addition, with respect to the global market shock component of the adverse and severely adverse scenarios, the Federal Reserve used proprietary data licensed from the following providers: Bloomberg L.P.; Intercontinental Exchange; JPMorgan Chase & Co.; Markit Group; and MSCI, Inc.REF
There are likely dozens of models utilized in the DFAST and dozens of sources of data.REF General descriptions of the purpose categories of these models are not sufficient for transparency and public accountability. The Federal Reserve states that it has an internal independent validation team for its models, but this process should be open to the public. As it stands currently, the DFAST is an obscure collection of unknown size of nondisclosed models and corporate contributions. It appears that the board is attempting to make this clearer to the public through its proposed rule, but even that description lacks important detail to facilitate public discourse of the models.
If the board were interested in full transparency, it would publish an exhaustive list of the models used. Up to the point of not violating privacy concerns or releasing proprietary models, there would be descriptions of the models that include all formulas. Data used as inputs would be made publicly available when possible. In the case of proprietary models and data, the specific uses of these resources would be published.
The GAO’s Recommendations for Transparency and How the Federal Reserve Has Performed thus Far
On November 15, 2016, the GAO evaluated the goals of the Dodd–Frank Act’s supervisory stress tests.REF The GAO found “[l]imitations in the Federal Reserve’s stress test programs that could hinder their effectiveness.” The GAO found limitations described loosely under three themes. The themes include scenario design, issues with transparency surrounding the more qualitative portions of the CCAR, and the lack of model risk assessment from the Federal Reserve. If the component models produce incorrect output, model risk is the possible adverse effect. The GAO goes on to make 17 recommendations for making the stress test process more effective. The GAO report includes the 17 recommendations, a description of their status (open or closed), what the Federal Reserve has done (if closed), and an evaluation of these actions.
The current status of the DFAST and CCAR, according to the GAO, could be described as mixed. The board has managed to improve internal standards of practice with increased meetings and communication across the entire process. One important change has been the Federal Reserve’s move to not solely use qualitative assessments to object to firms’ capital plans. Additionally, the Federal Reserve has increased communications with firms in regard to determinations around CCAR. These are welcome improvements. As noted, the board has also begun acknowledging transparency issues, but there is still room for vast improvement.
The areas where recommendations from the GAO have yet to be met are perhaps the most important. First, the GAO makes several recommendations about how the Federal Reserve forms supervisory scenarios. These range from questions of whether a single “adverse” scenario is sufficient, to deeper questions about the types of behavior the specific scenarios could induce as firms attempt to comply with Federal Reserve testing and decisions. In the case of the scenarios, the GAO fears that they are encouraging more intense business cycles, which could ultimately lead to greater turmoil.
Where Transparency and Policy Meet: Model Risk
One of the main issues with the current state of the DFAST and CCAR is the fact that the Federal Reserve still lacks a grip on model risk and how much this would affect the results of the stress tests. This technical point means that firms could be making decisions and presenting results that are skewed precisely because of the design of the stress test. In other words, attempts to comply and traverse regulations could, in fact, lead to unforeseen consequences. Currently, the Federal Reserve does not account for any of this behavior in their modelling, according to the GAO. This has been an open issue since January 2017 and seems to still be ongoing. Ultimately, this model risk issue is important because it implies that the Federal Reserve currently does not have the tools to determine the effectiveness and validity of the DFAST and the CCAR. Implementing the recommendations from the GAO on the issue of model risk is an important step as it is necessary for the Federal Reserve to consider the impact of its work. However, it is still unacceptable if the Federal Reserve continues to conduct all of this business internally. Creating a public release of all the models utilized in the DFAST would help the public and other interested parties to understand the purpose of stress testing and also bring to light any adverse incentives that could be created.
Conclusion
To this day, the supervisory stress tests created by the Dodd–Frank Act suffer from a lack disclosure. Without knowing the description of models employed, the subjective criteria under which the board occasionally operates, or whether the models themselves are the right models for the job, it is hard to know whether the DFAST is truly effective. The Federal Reserve should continue to implement the GAO’s 17 recommendations for transparency, but in a meaningful fashion. Models should be reviewable by the public, and general methodologies should be made specific. Without transparency for the world outside the Federal Reserve, the public cannot evaluate whether the Dodd–Frank Act stress testing is necessary—or even beneficial.
—Drew Gonshorowski is Senior Policy Analyst for Simulations in the Center for Data Analysis, of the Institute for Economic Freedom, at The Heritage Foundation.
