1) In vendor model documentation, model flow diagram is provided. These flow diagrams explain the conceptual framework but a more detailed model architecture, mapping outlining all model components including specific flow/transformation of data from input to final output should be provided.
2) Many models use multiple inputs from internal and external sources. There should be transparent evidence of their validity and authenticity.
3) A detailed document should be provided for calibration of pricing and risk models. An importance case is Value at Risk or Potential Future calculation using Monte Carlo simulation. There are various correlation assumption made in aggregation of the portfolio returns and scenario generation. These correlation assumptions should be clearly documented.
4) Validator should perform exhaustive sensitivity analysis. They should study the model output by changing each input variables and thus study variable’s linear or non linear relationship with the output.
5) When the validator is validating a vendor model which is replacing another vendor model, multiple subsets of input data should be created covering maximum possible business cases and output values of the new model should be benchmarked with the output values of the older model.
6) Validator should have clear understanding on how model treats the missing values. If documentation is not available then by running the model multiple times and thus make a judgment.
7) Often there will be times when vendor model output wont be intuitive to model owner or business owner. That will not mean that the model is conceptually wrong. Deep dive analysis by model or business owner will not be be possible because model methodology being proprietary. To manage model risk a process plan should be in place where detailed issue can be passed to the vendor to resolve the issue in efficient and minimum time and iteration.
For detailed discussion on Vendor Model Validation please refer to the below white paper from PSS Moorthy:
Vendor Model Validation: Challenges and Proposed Solutions
This post will be updated in future.