Important Aspects of Development and Validation of Stress Testing Models

The Federal Reserve expects Bank Holding Companies (BHCs) with assets greater than $50 billion to hold sufficient capital to maintain access to funding, to continue to serve as credit intermediaries, to meet their obligations to creditors and counter parties, and to continue operations, even in an adverse economy.

For this reason they have developed a process of Comprehensive Capital Analysis and Review (CCAR). CCAR stress tests are macroeconomic scenario-based stress tests, with three supervisory scenarios (Fed base, Fed adverse and Fed severely adverse) as well as additional internal scenarios (base, mild, inflation and deflation).

The capital plans rule applies to these BHCs and they have to develop and submit a capital plan to the Federal Reserve on an annual basis. Thus they request their prior approval before making a capital/dividend distribution to its shareholders.

In this article we explain key aspects which risk managers need to look into for modeling and validation of stress testing models.

First step will be understanding of the business (or company’s portfolio) for which stress testing model has to be developed. This understanding is required because stress testing framework requires projection of

1) Pre-Provision Net Revenue (PPNR)

2) Loan losses and changes in allowance for loan and lease losses (ALLL)

3) Losses on loans held for sale and measured under the fair-value option

4) Other-Than-Temporary Impairment (OTTI) losses on investment securities in the available-for-sale (AFS) and held-to-maturity (HTM) portfolios

5) For BHCs with large trading and private equity exposures, losses on those exposures resulting from a global market shock; and,

6) For BHCs with substantial trading, processing, or custodial operations, losses from the default of their largest counter party

Each of the above component has its own respective components and sources of data. Hence a top down understanding of the company’s business is critical. Reading and understanding form 10-K of the company will be an excellent starting point towards this direction.

Second step would be to understanding the data. Typical data types are

1) Loss driven data: Examples are historical Probability of Default (PD), Loss Given Default (LGD), Exposure of Default (EAD). PD data is modeled as time series data, LGD is modeled as cross sectional data and EAD modeling depends on the type of portfolio.

2) Valuation/return based data: Time series data, which is income, return, revenue forecasting. Based on given data sequence a time series model is developed which help in calculation of change in valuation with respect to the given stressed scenarios.

The combination of step 1 and step 2 will involve deep discussions between business leaders and modelers. The output of this step will be an agreement among them about what needs to be modeled. These two steps should take maximum investment of time and resources.

Third step is data development. This step is the most important step in the whole process because here the bridge is developed to transfer the documents and reports spread all around the organization into a format where a model can be developed using analytical methodology based on business sense.

Forth step development of a model which predicts the relevant dependent variable using external/internal independent variables. This will require business and statistical due diligence to find relevant independent variables. Generally linear regression based models are developed. Other methodologies which can also be implemented are Transition Probability Matrix (TPM) and Monte Carlo simulation based methodologies. TPM with be involved assigning conditional chances assigned to the scenarios resulting into most probable output. Monte Carlo simulation will involve introducing a random factor in the model and predicting the possible event with the choice of confidence.

It is important to ensure the robustness of the model by using appropriate statistical tests. Some suggestions:

1) If residuals of the model are having heteroskedasticity or auto-correlation, this implies that the relationship between the dependent and independent variables will not be valid when the independent variables are stressed.

2) The relationship strength can be checked by Granger Causality test. If the model relationship satisfies this relationship this means that the independent variables indeed impact the dependent variable and vice-versa.

Post model development analysis

Once the model has been developed, there are further analytical exercises which help in deriving more insight about the business using model output. These process when combined with business judgment also help us in authenticating the model output.

1) Back testing: Once model is designed, back testing checks the validity of the relationship in the past. The model is calibrated using (say) 80% of the data, for rest 20% the actual vs model values are compared.

2) Backcasting: Backcasting starts with defining a desirable future and then works backwards to identify policies and programs that will connect the future to the present. In the context of CCAR stress testing, future scenarios (values of the independent variables) are predefined. Using the model relationship future value of the dependent variables can be calculated. But for robust risk management, risk managers should not stop at this. The dependent variables as discussed above are composition of other variables. A qualitative (and possibly quantitative) business scenario based theory should be developed on how the those constituents are impacted by the change in external macro economic variables.

Other challenges

1) Often definition of data and its right source cannot be well defined. In vast organizations which have grown by mergers and acquisition, there may be multiple sources of data and it is a challenge to assemble best available level of granularity. It also needs to be ensured that there is no double counting of data/portfolio.

2) Finding robust validation framework, once the model is designed, it is important to find right statistical tests on which model needs to pass so that all the stake holders get comforted about its authenticity.

3) Strong model monitoring framework that ensures timely warnings and thus feedback from the stakeholders.

Challenge of Limited Data

Data is a natural phenomenon, but models are man-made. Consequently, there are no models that can completely capture all available data and predict all future behavior. This problem only gets worse when there is limited and unreliable data. There are a limited number of options available from a modeling perspective to deal with thin data sets. Consequently, modelers and business experts have to analyze business problems from alternative perspectives to achieve the best result. They need to compliment the model with alternative scenarios (through other more similar distributions), similar businesses (through expert judgments and independent studies) to support the output delivered by the model.

Parting thoughts

CCAR stress test models are last line of defense of the financial systems. This is a concept which is not just about actual model itself but also about the business understanding, understanding about the surrounding and articulating the impact of that surrounding on business in extreme times. All said, by no means we undermine the importance of statistical robustness, but rather emphasize that over and above that robustness further deeper assessment is required. The purpose of this article to list out key points which analyst should consider while modeling and validation of a stress testing model.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s