Management challenges in Model Validation Exercise

In financial risk management small models are primarily those which cater low portfolio size in comparison to the total portfolio of the company. On the other side big models:

  1. Cater huge portfolio chunk of the company hence they have got high visibility in the senior management.
  2. They are generally network model, that is their inputs are outputs of other models. So those input models come in the validation scope of the main model.

When big models are to be validated, a team of validators is constructed. This team is an adhoc team constructed from personals from different teams. The members work together for the requisite time and then they go back to their respective teams. It is common that these members have their other respective responsibilities which they have to adhere while validation of the model in hand.

Generally team is composed of personals who are subject matter experts (SME) of the business which model caters, PHDs or masters in quantitative field with many years of business experience. Such a team even though composed of highly qualified and matured individuals has its own management challenges. In this article I would like to discuss some of them.

Deciding team structure: Selection of two members is extremely critical for the success of the validation exercise: lead validator and SME of the business which model caters.

The lead validator has to be not only a functional expert (quantitative and risk management concepts) but also a people manager. The SME will have to be a business expert and also an expert in quantitative techniques or in risk management. It is rare that SME is an expert in all three: business, quantitative techniques and risk management. Again SME should be a person who gels well with the team because in most of the times he would be the only goto person for every member for any business related understanding.

The team can be composed of employees of the bank and external consultants. The lead validator needs to ensure that roles and responsibilities of each individual are decided at the very beginning. The most important responsibility is documentation of the report. He may ask each analyst to document his work and finally coagulate or assign someone in the team to keep track of all the analysis.

Another challenge will be finding team members who have worked on same business and class of models. Some classes of models are similar models, for example suppose there is a portfolio “X”. One model is about its valuation, another is about stress testing. Ideally the validation team of these models should be same because of the similarity required in their understanding, but this is not always possible.

Limiting the number of meetings: Meetings are important to share the knowledge, analysis, ideas, and brainstorming. Meetings with model developers are also very important. But excess meetings result in wastage of time. Challenges are:

  1. Deciding who should attend which meeting: Involving every member in every meeting is a clear wastage of time and energy. On the other side it is important that every member should have clarity of the bigger picture so that they may provide relevant analysis and perspective.
  2. Lead validator needs to ensure that time of model developers should not be wasted by asking them same questions multiple times by different validators. Asking same question multiple times will not only waste the time of the developers but also create an impression that validation team lacks consensus thus reducing their credibility.
    • Discipline to maintain written record of all the communication needs to be followed.

Adherence of time lines: There are occasions when some other project/model comes with higher priority. That may require immediate attention of the validation team. This can create delay in analysis for the model in hand.

As discussed, the team members belong to different teams, so there can be a situation when some members may have to de-prioritize temporarily the model validation exercise in-hand. As a matter of fact the model in hand may not be of equal priority for each team member.

Reshuffling of resources is common. Back-up plans should be in hand.

Deciding the limit of analysis and number of findings: Every validator desires that his finding and analysis should find place in the final validation report. Also some validators in enthusiasm of performing deep down analysis tend to over analyze an issue.

Lead validator has to ensure that he stucks right balance on the number of findings related to the model and clarity of the validation team opinion. Fewer findings may appear that they have not gone into the details. Too many findings may leave the model developers distracted from the crucial issue.

Lead validator also needs to ensure that all the analysis and findings are iron clad accurate, to the point and consistent. Any flaw in their findings caught by the model owners will put the credibility of the whole model validation report into question. Such situation goes into a loop where model owners start validating the analysis and findings of the validators and vice versa.

Managing the ego of model validators: Model validation exercise is similar to mountain climbing. In mountain climbing target is to reach the top. Each mountaineer has to reach the top and on the way he has to find trails and treasures. Each one has his own tools and physical strengths. Some mountaineers may climb the mountain faster, some might be slower. Accordingly there would be variations in their findings of trails and treasures. On reaching the top the lead mountaineer has to decide which set of trails and treasures best describe the mountain.

Coming back to model validation, just replace the mountain with the model, trails with analysis and treasures with findings.

This creates a very competitive atmosphere in the team. Validators are often very emotional about their own analysis and findings but are critical of their peers’. This results in ego clashes. The lead validator needs to ensure that this competition remains healthy and productive. The lead validator has to make sure that each validator feels that he is given optimal of opportunity to present his analysis and given fair credit.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s