We are delighted that Professor John Kay has allowed us to reprint this column, on the dangers of relying on models to predict what is beyond their power to predict.
As the global financial crisis began to break in 2007, David
Viniar, then chief financial officer of Goldman Sachs, reported in astonishment
that his firm had experienced “25 standard deviation events, several days in a
row”. Mr Viniar’s successor, Harvey Schwartz, has been similarly surprised.
When the Swiss franc was unpegged last month, he described Goldman Sachs’
experience as a “20-plus standard deviation” occurrence.
Assume these experiences were drawn from the bell-shaped
normal distribution on which such claims are generally based. If I were to
write down as a percentage the probability that Goldman Sachs would encounter
three 25 standard deviation events followed by a 20 standard deviation event,
the next 15 lines of this column would be occupied by zeros. Such things simply
do not occur. So what did?
The Swiss franc was pegged to the euro from 2011 to January
2015. Shorting the Swiss currency during that period was the epitome of what I
call a “tailgating strategy”, from my experience of driving on European
motorways. Tailgating strategies return regular small profits with a low
probability of substantial loss. While no one can predict when a tailgating
motorist will crash, any perceptive observer knows that such a crash is one day
likely.
Some banks were using “risk models” in which volatility was
drawn from past daily movements in the Swiss franc. Some even employed data
from the period during which the value of the currency was pegged. The
replacement of common sense by quantitative risk models was a contributor to
the global financial crisis. And nothing much, it seems, has changed.
It is true that risk managers now pay more attention to
“long-tail” events. But such low-probability outcomes take many forms. The
Swiss franc revaluation is at one end of the spectrum — a predictable
improbability. Like the tailgater’s accident this is, on any particular day,
unlikely. Like the tailgater’s accident, it has not been observed in the
historical data set — but over time the cumulative probability that it will
occur becomes extremely high. At the other end of the spectrum of
low-probability outcomes is Nassim Taleb’s “black swan” — the event to which
you cannot attach a probability because you have not imagined the event. There
can be no such thing as a probability that someone will invent the wheel
because to conceive of such a probability is to have invented the wheel.
But most of what is contingent in the world falls somewhere
in between. We can describe scenarios for developments in the stand-off between
Greece and the eurozone, or for the resolution of the crisis in Ukraine, but
rarely with such precision that we can assign numerical probabilities to these
scenarios. And there is almost zero probability that any particular scenario we
might imagine will actually occur.
What Mr Viniar and Mr Schwartz meant — or should have meant
— is that events had occurred that fell outside the scope of their models. When
the “off-model” event was the breakdown of parts of the wholesale money market
in 2007, their surprise was just about forgivable: in the case of the Swiss
revaluation, to have failed to visualise the possibility is rank incompetence.
Extremes among observed outcomes are much more often the
product of “off-model” events than the result of vanishingly small
probabilities. Sometimes the modellers left out something that plainly should
have been included. On other occasions they left out something no one could
have anticipated. The implication, however, is that most risk models — even if
they have uses in everyday liquidity management — are unsuitable for the
principal purpose for which they are devised: protecting financial institutions
against severe embarrassment or catastrophic failure.
First published in the Financial Times.
