I attended recently a model risk panel discussion organized by PRMIA with sponsorship of Numerix in NYC.
It was really informative, but the personal from Numerix was not able to answer my question which is the key to vendor model validation.
I asked him why model vendors are not providing adequate (read complete) documentation so that the model validators may be comfortable in defending their understanding to the regulatory authorities.
Instead he dodged the question by saying …. try explaining the regulatory authorities that it is ok to treat the model as a blackbox. We can analyze the model through the outcomes analysis. Take some sample instruments to start with. Look into their outcomes analysis, look into the performance parameter. Check and verify that the vendor model is doing its job and hence defend the model in front of the regulators.
I told him that is easier said then done. Typically there is no doubt of the correctness of the implementation of the standalone models as such but there are challenges in
1) Understanding the aggregation process which vendor model takes when there are thousands of instruments, with different and varying cashflows.
2) In Monte Carlo simulation how is the simulated paths take into account the correlation among the risk factors?
3) How are the model parameters calibrated and what are the convergence criteria and what is the reason behind choosing those optimization strategy in the considered portfolio’s context?
I told him that as a software vendor, your strength is providing a software framework which makes the life of risk managers easy when they are looking multiple portfolios with varied risk factors. You are not using your in house developed risk models. If you are, as a validator I am anyway not passing it without thorough understanding which is impossible if your documentation is incomplete.
You must be using the well researched methodologies which are anyway available around in the internet. They why you have inhibitions in sharing the complete documentation.
To this he replied : It is a challenge to find people to document the models they developed.
As a validator, I was not comforted by his answer.