How does Model Risk Management affect buy-build choices in surveillance?

Model risk management (MRM) teams were deeply involved in validating surveillance models (deterministic and stochastic) even before the Prudential Regulation Authority (PRA) brought out SS23/1 (model risk management principles for banks) earlier this year. That supervisory statement makes clear the regulators’ view that MRM should be applied much more broadly than before, all but guaranteeing the inclusion of surveillance in it.

One of the many issues this creates is a possible divergence between MRM’s ability to evaluate software built in-house and that which has been bought in. There are also nuances around what level of solution requires MRM intervention – for example, voice surveillance relies upon transcription and perhaps translation software. Who evaluates the data quality from those systems? And if they are supplied by Big Tech firms, is there any chance that they will be prepared to engage in that discussion at all?

So, the involvement of MRM affects the choice of surveillance technology and potentially complicates the buy/build decision. But how is this working out in practice, given that the application of MRM to surveillance is far from mature?

Banks agree that the MRM approach applies to all surveillance solutions, regardless whether they're developed by in-house teams or vendors. But they differ on how to proceed from there.

Some surveillance heads take the view that vendor solutions with limited scope for the customisation of parameters may be treated as low risk and “so it may be possible that an MRM department might do a light review or leverage vendor's existing documentation, which has been tried and tested at other institutions”, in the words of one.

Others say that the bar is no lower with respect to vendor surveillance models compared to in-house builds. For in-house developments it is possible for compliance, technology and MRM quant teams to work together throughout the project, including creating documentation tailored to helping regulators get comfortable with what is being done.

For 3rd-party solutions, yes, the 1^st and 2^nd lines can leverage the existing documentation and explain tuning and calibration, why certain features were chosen, how these features correspond to the risks and how the training process was carried out, but there will be a limit to their ability to explain any underlying functionality if it is complex or in any way ‘black box’. And even ‘simple’ deterministic, rules-based models are now becoming hard to analyse because of advances in technology: our ability to process huge amounts of data through long chains of rules are blurring the lines between deterministic and stochastic models.

There is clearly an opportunity here for vendors.

As one surveillance head puts it bluntly, “Off-the-shelf products that are industry best practice and which are documented in a way that is tailored towards satisfying MRM concerns will shorten our lifecycle to delivery – and that is a significant consideration. If someone can help reduce the number of weeks it takes to go through model validation, absolutely that is a competitive advantage – because there is a time to market element.”

Another agrees: “If I have a vendor who comes to me and says, ‘I'm going to make your life a lot easier for model validation because I have fantastic documentation and I can show you that it's already gone through a model validation-like process, I’ll go for that! So, there's a competitive advantage there for vendors.”

Off the record, surveillance leaders say that certain vendors are much easier to work with than others in terms of the documentation they have available and that that is a key consideration when they are looking at solutions.

Ultimately, surveillance is about reducing risk in a proportionate and cost-effective way. Regulators and management need assurance that the technology banks are implementing does actually manage the risk that it is there to manage. Vendors need to help banks give that assurance regardless of the demands of MRM team. But those willing to work with their customers, and those model risk experts, stand a much better chance of being bought. They’re also most likely to address the risks banks want to reduce.