Benchmarking is important and mandatory exercise in the financial risk management. For any model which generates the risk numbers benchmarking is a mandatory requirement from the regulatory authorities.
It is important because through benchmarking the risk managers and the regulators will be comforting themselves regarding the risk numbers generated by the model.
But apart from drawing comfort from the numbers risk managers can draw comfort from the methodology perspectives. Please hold on to this thought we will come back to this in next to next paragraph.
Data challenges in risk management
Robust risk management requires robust risk measurement. Robust risk measurement require robust data collection and integration processes. It is because when we have right data then only we can have correct inputs and calibration process for the risk models in hand.
But robust data collection and integration process is easier said then done. Often definition of data and its right source cannot be well defined. In vast organizations which have grown by mergers and acquisition, there may be multiple sources of data and it is a challenge to assemble best available level of granularity. It also needs to be ensured that there is no double counting of data/portfolio.
This problem only gets worse when there is limited and unreliable data. There are a limited number of options available from a modeling perspective to deal with thin data sets.
Coming back to the thought of benchmarking methodologies, risk measurement has two aspects: right data and right methodology.
In case there is limitations in data, any implemented methodology should be benchmarked by an alternate source of data of similar risk. For example when developing credit risk model based on Transition Probability Matrix, if we have limited internal data, we should benchmark the methodology using external data, say Moody’s data assuming that data is adequately available. If the business problem which we want to solve using the methodology is effectively solved on Moody’s data at least we can find comfort that methodology is sound.
In case there is no limitation of data, even then benchmarking the methodology on external data will help the risk manager in finding comfort about the robustness of the methodology.
Even SR 11 7 guidelines of model risk support this idea “Benchmarking is the comparison of a given model’s inputs and outputs to estimates from alternative internal or external data or models.“