Blind Reliance on Models: a Recipe for Trouble

 

We are delighted that Professor John Kay has allowed us to reprint this column, on the dangers of relying on models to predict what is beyond their power to predict.

 

As the global financial crisis began to break in 2007, David Viniar, then chief financial officer of Goldman Sachs, reported in astonishment that his firm had experienced “25 standard deviation events, several days in a row”. Mr Viniar’s successor, Harvey Schwartz, has been similarly surprised. When the Swiss franc was unpegged last month, he described Goldman Sachs’ experience as a “20-plus standard deviation” occurrence.

Assume these experiences were drawn from the bell-shaped normal distribution on which such claims are generally based. If I were to write down as a percentage the probability that Goldman Sachs would encounter three 25 standard deviation events followed by a 20 standard deviation event, the next 15 lines of this column would be occupied by zeros. Such things simply do not occur. So what did?

The Swiss franc was pegged to the euro from 2011 to January 2015. Shorting the Swiss currency during that period was the epitome of what I call a “tailgating strategy”, from my experience of driving on European motorways. Tailgating strategies return regular small profits with a low probability of substantial loss. While no one can predict when a tailgating motorist will crash, any perceptive observer knows that such a crash is one day likely.

Some banks were using “risk models” in which volatility was drawn from past daily movements in the Swiss franc. Some even employed data from the period during which the value of the currency was pegged. The replacement of common sense by quantitative risk models was a contributor to the global financial crisis. And nothing much, it seems, has changed.

It is true that risk managers now pay more attention to “long-tail” events. But such low-probability outcomes take many forms. The Swiss franc revaluation is at one end of the spectrum — a predictable improbability. Like the tailgater’s accident this is, on any particular day, unlikely. Like the tailgater’s accident, it has not been observed in the historical data set — but over time the cumulative probability that it will occur becomes extremely high. At the other end of the spectrum of low-probability outcomes is Nassim Taleb’s “black swan” — the event to which you cannot attach a probability because you have not imagined the event. There can be no such thing as a probability that someone will invent the wheel because to conceive of such a probability is to have invented the wheel.

But most of what is contingent in the world falls somewhere in between. We can describe scenarios for developments in the stand-off between Greece and the eurozone, or for the resolution of the crisis in Ukraine, but rarely with such precision that we can assign numerical probabilities to these scenarios. And there is almost zero probability that any particular scenario we might imagine will actually occur.

What Mr Viniar and Mr Schwartz meant — or should have meant — is that events had occurred that fell outside the scope of their models. When the “off-model” event was the breakdown of parts of the wholesale money market in 2007, their surprise was just about forgivable: in the case of the Swiss revaluation, to have failed to visualise the possibility is rank incompetence.

Extremes among observed outcomes are much more often the product of “off-model” events than the result of vanishingly small probabilities. Sometimes the modellers left out something that plainly should have been included. On other occasions they left out something no one could have anticipated. The implication, however, is that most risk models — even if they have uses in every­day liquidity management — are unsuitable for the principal purpose for which they are devised: protecting financial institutions against severe embarrassment or catastrophic failure.

First published in the Financial Times.