Book Review – The Signal and the Noise

If you’re like me, you’re probably more than a little glad that the election is over. You might not be happy with the results (and with Obama being re-elected, the Democrats retaining the Senate and Republicans keeping control of the House, just about everyone can find something to hate about the results), but it’s nice that it’s over. In particular, I’m glad that we no longer have to hear weekly, daily, and sometimes hourly updates to the predictions about who will win. With how many predictions there were out there, how did so many manage to be so, so wrong?

The Signal and the Noise attempts to answer that question, by looking at predictions, how they are made, why they fail so frequently, and how they can be improved. Nate Silver, whose FiveThirtyEight blog stood out as one of the most accurate predictors of this past election, looks at the process of predicting and, to quote the subtitle, ‘why so many predictions fail – but some don’t’. What insight into the process of prediction does it provide? As always, let’s read on to find out!

Summary

The Signal and the Noise opens with an Introduction that looks at the rise of information availability over the past several centuries. It notes that though the increasing levels of information has lead to advantages in many areas (such as boosting the economy), it has also increased the sheer amount of incorrect or misleading information (the ‘noise’) that exists in the world. It finishes up with a short overview of what the book will cover, first diagnosing the problems with predictions and then looking at how to change our predictive methods to improve our results.

Chapter 1 – A Catastrophic Failure of Prediction

The first chapter looks at the financial crash and how different methods of prediction failed to point it out. From the difficulty of rating agencies to correctly judge the risk in credit default swaps to the overuse of leverage by financial institutions thinking such swaps were perfectly safe to the failure of government policies to improve the economy after the crash (at least, as quickly and effectively as assumed), the failure of predictive ability leading to and following the financial crash is reviewed. The common thread in all these predictions was noted as a false sense of confidence on the part of the predictors.

Chapter 2 – Are You Smarter Than a Television Pundit?

You probably immediately said yes upon seeing this chapter’s title, but it turns out there’s a reason that you are probably correct. There is a tendency for the pundits who appear on such shows to be ‘hedgehogs’, who believe in Big Ideas politically and take sharply defined, unchanging positions, usually at one end of the political spectrum. A better predictive technique (if one that is less entertaining to watch) is to be a ‘fox’, considering a multitude of approaches to a situation and adapting to new information as it is uncovered. Some of the techniques of a good fox are then covered, such as thinking probabilistically and looking for consensus among other predictions.

Chapter 3 – All I Care About is W’s and L’s

The third chapter looks at baseball, where Silver used to work, and which has a huge data base for future predictions. It looks at how there has been disagreements about whether relying on statistics or in-person scouting is the better method to determine the quality of a player. As it turns out, a combination approach seems to be the best, if the scouts take advantage of the stats that are available but also use the in-person knowledge that they are able to gain by watching and speaking with baseball players in person (such as the attitude that the players take toward the game).

Chapter 4 – For Years You’ve Been Telling Us that Rain is Green

In a perfect world, with perfect knowledge of all the current conditions, it should be possible to perfectly predict the weather. However, as you likely know, that’s not the case, and the fourth chapter looks at why, noting that chaos theory implies that systems that are dynamic and nonlinear (like the weather) tend to be difficult, if not impossible, to predict with any level of accuracy. It also notes that putting goals other than accuracy first in your prediction can lead to less accurate forecasts (as when weather forecasters call for rain to draw a larger audience).

Chapter 5 – Desperately Seeking Signal

The fifth chapter goes from weather to earthquakes, noting some of the difficulty in drawing conclusions about (more powerful) earthquakes in large part due to the relatively small number that occur, and the resulting lack of reasonable data points. It leads to a temptation to overfit, to make overly specific predictions that fit the data you have, but don’t allow for the actual nature of the situation. Trying to find a pattern when there isn’t enough data can lead to incorrect predictions of future probabilities of earthquakes, at times causing their frequency to be under-predicted.

Chapter 6 – How to Drown in Three Feet of Water

There is a tendency when predictions are presented to not give the level of uncertainty associated with them. Particularly with economic predictions, the numbers are presented as highly precise, and a slight difference from the prediction (like 9.2% unemployment rather than 9.1%) can have a large effect on the market that depends on those numbers. Given the large amount of economic data out there, it’s easy to find causation where it doesn’t exist, assuming that things like GDP and the level of debt will have a substantial effect on the market, for example. (Side note: The title of the chapter comes from the fact that it’s possible to drown in a river that is on average three feet deep if it happens to be sixteen feet deep when you are in it.)

Chapter 7 – Role Models

The swine flu ‘epidemics’ of the late seventies and of 2009 serve as an example of how extrapolation can lead to improper predictions, particularly if you assume that things will keep proceeding as they have in the recent past. It notes that self-fulfilling and self-canceling prophecies complicate the process of determining the future, by altering which directions the given traits proceed and altering their progress. The efforts to change the progress of certain events, helping the good and thwarting the bad, mean that many traits change their course from their initial progress (as when the swine flu outbreaks were stopped shortly after starting).

Chapter 8 – Less and Less and Less Wrong

With seven chapters about how predictions can failure, it’s now time for some thoughts on how to improve the predicting process. This chapter introduces Bayesian theory, which seeks to calculate the probability of events by taking new events into account and altering our expected probabilities to fit the new data. It also discusses how we’ve gone from Bayesian methods to ‘frequentist’ statistical methods, and how this method leads to increasing levels of false positives and statistical bias. Most of these concepts are illustrated throughout the chapter with examples from gambling, particularly the results of one highly successful professional gambler who applies Bayesian methodology.

Chapter 9 – Rage Against the Machines

This chapter shows the potential for machines to dominate us…in chess. It looks at the chess matches between Garry Kasparov and the Deep Blue computer, noting how computers are think much faster than people but are unable to work better than the programs on which they run. It also notes that relying on computers to make predictions is only as good as the data and calculations that are put into the computer, and that as a result, predictions that state ‘the computer says…’ shouldn’t be viewed as any more accurate than predictions made solely via human effort.

Chapter 10 – The Poker Bubble

This chapter covers the recent boom in poker, online and off. It discusses how successful poker players take advantage of probabilities to determine how likely it is that another player has a legitimately good hand, and how likely it is that they are bluffing. It looks at how the rise in poker interest around 2004-2996 allowed people (including Silver) to produce a sizable income for a while due to the large number of unskilled people trying their hand at the game, by using probabilistic knowledge to beat the less skilled.

Chapter 11 – If You Can’t Beat’em…

This chapter looks at how Bayes’ theories can be applied to financial markets. It notes that the ‘free hand’ of the market can be seen as a Bayesian method, adjusting the cost of stocks to meet the market consensus as new data is discovered and people adjust to that knowledge. It also examines the efficiency of the market, and notes there are several reasons why it doesn’t always behave completely efficiently, from overconfidence among traders to the herding behavior of many people, particularly mutual fund managers who don’t want to do substantially worse than their competitors.

Chapter 12 – A Climate of Healthy Skepticism

The twelfth chapter looks at climate change (previously, and perhaps more convincingly, known as the ‘greenhouse effect’). It notes that while most of the book shows how people can mistakenly find a signal in all the noise of the many sets of data out there, in this situation the signal (the broad evidence of increasing global temperature) is getting lost in the noise (the seasonal temperature shifting and regional highs and lows that accompany weather related data). It looks at the importance of estimating uncertainty, to determine how likely it is that the observed temperature changes are just noise rather than a true trend and to think in terms of probability when looking at things like a decade of cooling (like 2001-2011) when considering a broader warming trend.

Chapter 13 – What You Don’t Know Can Hurt You

The final chapter of the book looks at one of the toughest areas to predict successfully: terrorism and terrorist attacks. It discusses whether the 9/11 attacks were truly an ‘unknown unknown’, as Donald Rumsfeld famously put it, or whether they should have been considered (although not necessarily known about in advance). How to calculate the expected number of terrorist attacks and the amount of fatalities caused by those attacks is discussed, as well as how to use that knowledge to be more prepared for the possibility of future attacks.

The book ends with a conclusion stressing the importance of following the suggestions within the book when it comes to making predictions. It notes the need to think in terms of probabilities, not absolutes, to know your initial view of the probability of an event, and to be ready to have your predictions turn out to be incorrect (especially at first, until you gather more data). The book then concludes with a huge notes section; most chapters have at least fifty references, with several have more than one hundred.

Pros

• Thought-Provoking: The book definitely will change how you think about probabilities and approach the task of making predictions.
• Demonstrative of the Techniques: You get a chance to see how the Bayesian calculations are put into practice, allowing you the opportunity to follow along with Silver’s reasoning most of the time (and adapt the technique as your own, if you wish).
• Well-Researched: As noted, the resource section is huge; Silver does a great job of backing up his claims and calculations with solid sources.

Cons

• Somewhat Random Examples: While the broad range of topics discussed in the chapters, ranging from poker and chess to global warming and terrorism, the chapters seem to jump around, sometimes making it difficult to follow the main points from one chapter to another.
• Lack of Background to Some Chapters: Silver occasionally dives into a subject without providing much background to allow readers to follow along (particularly in the chess and poker chapters), making it harder to follow as he makes his points.

Overall

The Signal and the Noise looks at an interesting and useful way to view the process of making predictions and determining how the future will proceed. Its look at Bayesian probability methods and how to apply them makes an interesting approach to viewing the future. The book itself makes a very interesting and thought-provoking read.