Getting model governance wrong?

Models set up to monitor credit risk or trade FX have a direct impact on a bank’s P&L and potentially its viability. Alteration to those models’ parameters, whether malicious or not, therefore need to some kind of external validation – hence model governance teams.

But 1LoD conversations with surveillance professionals, and also leaders in other areas such as adverse media screening, reveal that model governance teams are increasingly an obstacle to the improvement of the ‘models’ that set the parameters for trade surveillance systems, for example.

Now, for sure, there are superficial similarities between the trading systems and surveillance systems: if their parameters are adjusted by the team running them, then it is possible that changes could be for the benefits of the person changing them, rather than the bank. And it is conceivable that changes to a surveillance system could trigger fines that resemble the smaller losses banks have incurred from trading errors.

But the similarities end there. No surveillance model parameter change is going to expose the bank to immediate loss because no ‘model’ in any three lines of defence context is connected to anything except an alert. So, the output isn’t a trade, or a risk position, or a direct action in a market of any kind. All that happens if you alter a surveillance ‘model’ is that different alerts are generated. By itself this difference ought to make it clear that applying conventional model governance to non-financial risk monitoring models is absurd.

It's even more absurd because 99% of surveillance (and transaction monitoring and adverse media screening) alerts generated are noise anyway, and are not analysed in detail. So, any change to parameters is unlikely to expose the bank to immediate material risk. In fact, any change to the parameters is more likely to make things better.

In addition, since no-one has any quantitative measure for non-financial risk either in general or in specific areas such as market abuse, it’s impossible to quantify with any exactitude the amount of risk incurred through any parameter change. All anyone can say is that if an alert is removed, or its threshold changed, the type of market abuse that might get through, in the absence of any other controls, would be of a particular type and size.

Model risk teams are also trained to believe that complexity drives materiality: as one surveillance head explains, “Take [vendor name redacted]. It’s a very, very sophisticated approach. So, the model governance team is like, ‘Oh my God, this is about as sophisticated as it gets. The materiality of this model is at the top tier, so we now need to have to do our governance to the highest level.’ But that isn’t right because non-financial risk models pose a very different type of risk to the firm.”

The problem with applying inappropriate levels of pseudo-quantification to issues that cannot be quantified to the degree desired is that it gives false confidence. The most famous example of this is when then CFO of Goldman Sachs, David Viniar, announced in August 2007 that Goldman’s flagship GEO hedge fund had lost 27% of its value since the start of the year, explaining, “We were seeing things that were 25-standard deviation moves, several days in a row.”

In other words, things were happening that were only supposed to happen once in every 100,000 years. Clearly in fact what happened was that quants were ignoring the messiness of real life and attempting to put a number on the uncountable.

Model risk governance is increasingly hobbling dynamic surveillance model tuning; it is being applied across market abuse and financial crime to sets of parameters which should not be thought of as ‘models’ at all; and regulators themselves are looking at calibration and parameters in a way that seems increasingly divorced from materiality and true risk.

The problem is, as with so much about the growth of non-financial risk management teams, who is incentivised to push back?