Something New

 We feature each week Nicholas Reid's reviews and comments on new and recent books

“THE IMPROBABILITY PRINCIPLE” by David Hand  (Bantam Press / Random House, $NZ37:99)

            A man is an expert in his field. The man writes a book about his field of expertise. In his own terms, the man cannot be argued with by the likes of you and me. We are not experts in his field. But we have the uneasy sense that the man has got something badly wrong. And to make matters worse, we also decide that the whole of the man’s book-length argument could have been expressed in four or five pages. In effect, the man’s book is a padded magazine article.

There now. I have short-circuited my own review of David Hand’s The Improbability Principle (subtitled Why Incredibly Unlikely Things Keep Happening) by giving my conclusions first rather than delaying them until the last paragraph or going all evasive and elusive, which is the way with many reviewers when they do not wish to speak clearly or make enemies.

Having passed judgment and called for execution, however, the infallible court of my mind thinks it would be fair to examine the evidence.

David Hand is certainly an expert in the field of Mathematics, especially as it pertains to probability and statistics. He is Emeritus Professor of Mathematics and Senior Research Investigator at Imperial College London, and has been president of the Royal Statistical Society. He has also published a number of popular books on mathematical topics.

The basic question he asks here is “Why do incredibly unlikely events happen? ”

He opens with true anecdotes of the most outrageous and improbable events to show us that such things do indeed happen. He then turns to pre-scientific explanations for such improbable events and huge coincidences, and has fun debunking them one by one. Old superstitions? They all depended entirely on the “confirmation bias” of selecting only that evidence which confirmed beliefs already held, and ignoring all other evidence that contradicted such beliefs. The power of prophets in predicting events accurately? It was always a matter of prophecies being ambiguous enough to be interpreted in different ways, so that the prophet would be validated regardless of what happened. Miracles? Why, David Hume’s dismissal of them is good enough for David Hand – don’t believe in ‘em unless the reason to believe in ‘em is more compelling than the reason not to. And as for psychic phenomena and Jung’s theory of “synchronicity”, please don’t make Professor Emeritus David Hand laugh. He will speak at length on the nature of chance, and on fallacies about luck with which gamblers delude themselves, but he will not take pre-scientific explanations seriously.

Why, then, do incredibly unlikely events happen?

Why (as Hand records it) does a woman buy two lottery tickets in two separate lotteries, and get two winning numbers - except that the winning number for Lottery A is the number of the ticket she has for Lottery B and vice versa? Why does a woman have four daughters born in four different years, but on exactly the same date – 3 August? Why does one man have the ill fortune to be struck by lightning on seven different occasions? Why does the actor Anthony Hopkins happen to find, discarded in a railway carriage, a copy of the very book he is looking for to research a character he is about to play; and then later discover that this discarded and randomly-found copy is the very copy the book’s author lost some years before?

The answer, according to Hand, is very simple. Given a sufficiently large number of events, extraordinary things will inevitably happen. Gaussian distribution means that most numbers will tend to averages, but a small minority of numbers will not. Thus with human events. We have shifted from the deterministic universe of Newtonian “clockwork”, where events in nature always had to happen in a completely predictable fashion; to a probabilistic universe with Heisenberg’s “uncertainty principle” and chaos theory telling us that it is quite in the nature of things for the unexpected to sometimes happen. The mere fact that multiple millions of golfers play rounds of golf every year means that inevitably, by the laws of statistical probability, there were be one or two who score three or four successive holes-in-one. There is nothing extraordinary about this at all. It is simply the fact of very large numbers at work.

And that, repeated and spun out with the use of many anecdotes over about 240 pages, is the whole argument of Hand’s book, which is why I call it an expanded magazine article. Really huge numbers of events happen everyday. Some of those events will therefore be unexpected or coincidental. No mystery and end of story. I note that Hand has created a piece of jargon to give his book a jazzy title (“The Improbability Principle”), presumably in the hope that the phrase will catch on. This technique is often used by authors of pop-science treatises, when they hope to create a bestseller.

            Now it would be extremely ungrateful of me not to admit that I enjoyed many of the anecdotes, even if I knew at once what point each of them was going to make. It is fun reading about Jean Dixon (pp.23 ff.). She was an American horoscope-deviser and fortune-teller, much trusted by President Ronald Reagan and his wife, whose reputation was built on “predicting”, in the 1950s, that a Democrat would be elected in 1960 and that he would be assassinated while in office. Wow! She “predicted” JFK’s assassination. Except that, as Hand points out, Dixon made many thousands of predictions in her life and nearly all of them proved dead wrong. Thus Hand speaks of the “Jean Dixon effect” - one lucky hit is inevitable given so many predictions. It’s just big numbers at work again.

Hand’s account of the famous “stock tipster” scam is also very entertaining (pp.180-182), showing simply that if you begin with a large enough pool of suckers, you will be able to convince some of them that your predictions are always right. Taking a mild slap at his own scientific community, Hand also amused me when he considered the “publication bias” (pp.135 ff.) in scientific journals, this being a variation of “selection bias”, where those articles about experiments showing the “success” of a certain drug or procedure are the ones likely to be published – i.e. in the main, scientific journals do not favour articles about long series of experiments that have proved nothing or have shown that a certain procedure is worthless. And I suppose it was also fun to see Hand using the “law of near enough” (i.e. people validating what they think is miraculous by means of stretching and selecting their evidence) to defuse Jung’s “synchronicity” or Arthur Koestler’s arguments in his Roots of Coincidence, although in both cases Hand is in no position to debunk those aspects of the two thinkers’ ideas that are not directly related to statistics.

But having noted these incidental pleasures, this is still a book that bashes away at a remarkably small stock of ideas.

And there is another problem. So determined is Hand to prove that everything extraordinary is just the ordinary power of big numbers at work, that he succeeds in warning against any sense of wonder or delight we might have in the unexpected happening. In the last sentences of his Epilogue, before two appendices and after having given us yet more true tales of wild coincidences, Hand concludes thus: “None of this is surprising at all. It’s the Improbability Principle.” (p.237)

But surely unexpected and extraordinary events and coincidences ARE surprising by their very nature. We would cease to be human if we were not surprised by them, even though we already know that many of them are just the work of probability. Hence the colloquial phrase “a one-in-a-million chance”. We understand that a million other, un-extraordinary, events have happened before this one event stands out to us as extraordinary, but that does not take away our delight and wonder in the one event, regardless of Hand’s dour injunction.

And, in a bigger sense, where is Hand’s single-minded and repetitious argument going?

Ah me! I think I smell the folly of the narrowly-focused specialist, in this case a statistician who, by talking in terms of the inevitability of big coincidences and “miracles” because of big numbers, can discount everything that is miraculous and beyond human computation. So he applies his theory to evolution and says randomness of events and huge numbers of life forms and mutations over millions of years mean that therefore there is no need to posit a purpose or a direction of even a particular development in life, let alone a creator. It is just a matter of huge numbers and chance.

And so, dear reader, to his own satisfaction the mathematician solves the riddle of the universe, but fails to take the final step of recognising that own context, the context of big numbers working in a particular way, has an origin. Why should huge numbers behave as they do? Only because the universe is made in a certain way.

The thin argument of The Improbability Principle, amusing in snatches but repetitious overall, is purely functional-operational, but the author presumes to stray into ontological territory, which is way beyond his expertise.