Probability & Partners: Model validation and IT risk management
This column was originally written in Dutch. This is an English translation.
By Erik Kooistra, Service Line Lead Model Validation, and Edward Roozenburg, Risk Management Consultant, both at Probability & Partners
In the financial sector, models play a crucial role in assessing risks. Credit, market, and liquidity risks are determined using models, and strategic decisions are based on the outcomes of these models.
Model validation and IT risk management are often seen as separate processes. This fragmentation can lead to confusion about where model validation ends and IT security begins, which can cause uncertainty about the reliability of model outcomes. An integrated approach is needed to ensure both the reliability of model outcomes and IT security.
Model validation: The backbone of reliable models
Model validation focuses on validating various elements that together ensure the reliability and applicability of a model. These elements include:
- Assumptions and model limitations: The starting points on which the model is based, including assumptions about market conditions and risk factors.
- Methodology: The mathematical and statistical techniques used to generate results.
- Outputs: The results produced by the model and their interpretation within the risk assessment.
- Model use: The way in which the model is used in practice compared to its intended use.
- Model governance: The processes and controls that ensure that the model is used in a consistent and controlled manner.
You can read more about this in the white paper Model Risk Management: Fundamental Concepts and Model Risk Management for Pension Funds: Increased model risk due to WTP transition. The purpose of model validation is to provide assurance that the model functions properly and is applied correctly in practice. In many cases, models are validated in their prototype form, with the focus on the assumptions, methodology, and outcomes. The IT security aspects of the model, such as robustness and data protection, are often only addressed in passing. IT security is usually only briefly considered when testing input parameters, while the broader IT security risks are largely ignored in the validations.
IT risks and invisible dangers for models
IT risks can range from data breaches and cyberattacks to system failures and internal threats. Protecting reliable data is key. Errors in IT control can lead to data being modified intentionally or accidentally. When a model relies on data that is not adequately protected, there is a risk that the data will become unreliable. For example, an error in assigning rights can result in unauthorized persons deliberately or accidentally modifying data, causing the model's outputs to lead to incorrect decisions.
What do we encounter in practice?
As mentioned above, the two worlds are generally separate in practice. IT controls are rarely considered in depth when models are validated. In some cases, a few components are taken into account in a validation. Often, these issues only receive attention when we point them out to the client. However, in almost all institutions known to us, the functioning of general IT controls is assessed separately from model validation.
Unfortunately, the functioning of IT controls is often only reported to managers other than the owners of the models. It is therefore possible that an organization has determined that the procedures for granting access rights are not working properly without the owners of the models being explicitly informed of this. This is not to mislead anyone, but simply because the model owner is not responsible for the proper functioning of the general IT controls.
Models are also sometimes used outside the regular IT infrastructure of organizations. This is referred to as end-user computing. As a result, these models also fall outside the scope of investigations into the functioning of general IT controls.
Finally, it is questionable whether investigations into the functioning of IT controls go into sufficient depth on the specific requirements that are important for the model. After all, the risk analysis that forms the basis for the controls and the investigation into their functioning is usually not carried out at the level of the model, but at the level of the institution or the business unit as a whole. This means that elements that are specifically important for a single model may be overlooked.
Practical example
Imagine that a financial institution uses a credit risk model without robust IT security. A cyberattack leads to the manipulation of customer data, causing the model to produce incorrect risk assessments. This results in incorrect credit decisions and significant financial losses. This example demonstrates how necessary it is to explicitly include IT risks in the validation processes.
Need for integration
The realization that model validation and IT risk management are closely linked is essential for ensuring reliable model outcomes. An integrated approach means that IT risks are included in the model validation process. This means that, in addition to assessing the model's assumptions and methodology, the IT infrastructure and security measures that support the use of the model are also evaluated. This requires collaboration between model validation experts and IT security professionals.
Practical Steps for Integration
- Joint risk analysis: Conduct a joint risk analysis involving both model validation experts and IT risk managers. Identify the IT risks that could affect the model outcomes and develop strategies to mitigate these risks.
- Security measures: Implement robust security measures to ensure the integrity and availability of the data used by the model. This includes the use of encryption, access controls, and regular security audits.
- Continuous monitoring: Ensure continuous monitoring of both model performance and the IT infrastructure. This helps to identify potential problems early on and take corrective measures before they affect the reliability of the model outputs.
Conclusion
When IT control and model validation remain separate worlds, various practical problems can arise. Model owners will be insufficiently involved in the proper functioning of IT controls. This can lead to data breaches and unauthorized access, and even to unreliable model outcomes. This can result in wrong strategic decisions and significant financial losses.
Model validation and IT risk management both contribute to accurate model-based predictions. By integrating both disciplines more explicitly and making clear agreements on scoping, institutions can increase the reliability of models and minimize the impact of IT risks. This not only improves the quality of risk assessments, but also strengthens the confidence of customers and regulators in the robustness of financial systems.
It is therefore essential to explicitly include IT risks in model validation in order to guarantee the integrity and reliability of models. This includes access management, data integrity, external dependencies, IT infrastructure, and model hosting. The integration of model validation and IT risk should therefore be considered best practice.