Common model related issues obseved by Model Validators

1) One of the most common issues raised by the model validators is not being able to replicate the model. The reasons being:

a) Many touch points in the model (operational risk)

b) Data discrepancies due to database updates. This is a serious on going monitoring issue if there is no disciplined approach in database updates.

c) Lack of robust SOP.

2) Insufficient analysis around variable selection process. The variable selection should not only be based on statistical justification but business justification as well. Many times regression is performed on lagged or transformed varaibles. Business and economic scenario based justification should be provided for their usage.

3) Model is calibrated based historical data. This is the only option available but in today’s dynamic business world past performance is not a guarantee for future performance. Also the portfolio concentration keeps on changing making the historical data less relevant.

It is recommended that model developers explain/document why past historical data is relevant for future scenarios and where this may deviate. They should also document that what are the actions management will take in case such deviations happen.

4) Feeder models not being valid: There are many models who take the output of the other models as their input. Many times it is observed that those feeder models are not been validated. It is also observed that those feeder models have been failed in a separate validation exercise. This makes the output of those feeder model invalid and hence that output is an invalid input to the model under consideration.

5) It is often observed that the parameter values are hard coded. They may be result of business judgement. But no explanation is provided.

6) In models which are based on regression, spurious regression is extremely common. Thumb rule: In any regression analysis if Rsquare is more than 80% then be cautious.

One example of “may be” spurious regression: Regressing one price series to another price series. The Rsquare may come high. The regression should be between the returns of two price series.

7) While validation of market risk models, replication of model is critical. If the validator is not able to replicate the model, then model should failed, unless there are some really  good explanations provided by the developers which may convince validator about the correctness of the model.

One interesting point is that valuations of simple model should match closely by the external model developed by the validator but sensitivity analysis results may not be replicable. That is because often sensitivity values will have error multiplied. Sensitivity measure is change in model value with change in some input. So the replicated model may have some slight difference, and that difference will be multiplied when sensitivities are calculated.

8) Typically in risk models development document, the developers/owners focus more on the model, instead of focusing on the business problem what model is trying to solve.

This page will be updated regularly.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s