Superforecasting tells the story of a group of ordinary people who became so good at making predictions they consistently outperformed so-called experts, professionals and entire government departments. Who are these people and why are they so special? According to author Philip Tetlock, superforecasting can be learnt and in this book he explains how.
Summary of main ideas
In 2011, IARPA, a US intelligence agency launched a forecasting tournament with the aim of discovering methods of prediction that would help them to increase the accuracy of their own forecasts.
The agency had learnt the hard way that the consequences of making bad predictions could be catastrophic, the decision to go to war in Iraq based on flimsy evidence being a case in point.
The Good Judgement Project, led by Philip Tetlock (the co-author of this book) and Barbara Mellers, was one of the organisations invited to compete. "Harnessing the wisdom of the crowd to forecast world events," the Good Judgment Project recruited volunteers and set them a series of questions.
Four years later, the volunteers had collectively worked on 500 questions (typically questions like "Will the president of Tunisia go into exile in the next month?" or "Will the Euro fall below $1.20 in the next twelve months?") and over a million forecasts. As well as emerging as the clear tournament winner, Tetlock noticed an interesting trend among these seemingly ordinary volunteers.
A subset of them were consistently achieving remarkable results, producing forecasts whose accuracy outperformed intelligence analysts with access to classified data.
What was it about these ‘superforecasters’ that made them different?
At this point, it’s worth taking a step back and mentioning Tetlock’s background. He previously authored Expert Political Judgment, which demonstrated that so-called experts were often wrong with their predictions and then not held accountable. He famously concluded that dart throwing chimpanzees would have achieved a higher success rate at predicting future events.
Superforecasting therefore builds on the foundations of Tetlock’s previous work to show that, while forecasting is an extremely difficult activity, there is an art and a science to doing it well.
Superforecasters, although not out of the ordinary, share certain traits and approaches that lower performers do not. In other words, they are intelligent people but don’t need to be geniuses and ‘super’ forecasting skills can be learned.
So what are the main characteristics of superforecasters?
They are in ‘Perpetual Beta’
Superforecasters work in constant feedback loops. They are hungry for new information and use this information to regularly update their forecasts. They understand that practice is essential to improving their forecasts.
They are foxes rather than hedgehogs
Related to the Perpetual Beta point, Tetlock categorises forecasters into two groups (although like everything else in Superforecasting, this is not binary but a continuum):
Hedgehogs are people who have a solid world-view which they reflect in their forecasts.
Foxes are people who do not base their forecasts on one big central idea. Instead, they aggregate many different perspectives and question their assumptions. Superforecasters mainly come from this group
Superforecasters understand that practice is essential to improving their forecasts.
They are extremely specific
Whereas most people would answer a question about a future event with either a binary “yes” or “no” response, or perhaps with a “maybe” if uncertain, superforecasters typically make their forecasts to percentage point levels of granularity.
For example, I currently believe there is a 57% chance that Biden will beat Trump in the 2020 election because I have seen predictions of 60% from sources I respect and I am naturally paranoid (and I am in no way a superforecaster!).
*Note from the future: with hindsight I feel good about that prediction....
They are humble
Superforecasters are acutely aware that the human brain is prone to making errors and they take measures to combat it. Tetlock discusses the concept of Systems One and Two, the two modes of thought popularised in Daniel Kahneman’s Thinking Fast and Slow.
System One represents the unconscious part of the brain which makes decisions quickly and intuitively.
System Two represents conscious and deliberate thought. Both are important but System Two needs to be engaged when forecasting. However, System Two requires more effort and the brain has a cunning habit of performing a “bait-and-switch”, swapping a hard question for an easier one.
Tetlock uses the example of an experiment in which his colleague asked one group of ‘regular’ forecasters “How likely is it that the Assad regime will fall in the next three months?” and another group “How likely is it that the Assad regime will fall in the next six months?”
As Tetlock explains: “Unconsciously, they would do a bait and switch, ducking the hard question that requires calibrating the probability to the time frame and tackling the easier question about the relative weight of the arguments for and against the regime’s downfall.”
They take a base value using an outside view and then adjust with the inside view
This means using external information as a starting point, such as considering how long it will take to complete a project based on the time taken to finish similar projects in the past, rather than calculating based on how long you think it will take to complete each individual task.
Only once the superforecaster has used the outside view to find a rough anchor value will they dive into the inside view to adjust their prediction upwards or downwards.
They systematically unpack questions into smaller components
Tetlock refers to this as “Fermi-izing” which is named after the Italian engineer Enrico Fermi. He developed a method of estimating that is often described as a ‘back-of-the-envelope’ calculation that can get surprisingly close to the real answer.
Fermi-izing is designed for the types of questions that cannot be solved by common mathematical or scientific information. Tetlock uses an example: “How many piano tuners are there in Chicago?” At first it seems ridiculous, but Tetlock walks through how this can be broken down into smaller questions.
To know how many piano tuners there are in Chicago, we need to have an idea of how many pianos. To get to a rough estimate of that number we can take the population of Chicago and then consider what percentage of the population owns a piano.
From there, how often does a piano likely need tuning and how long does it take? For a detailed breakdown of this example, read this (this type of puzzle is often used in interviews for roles at tech companies).
Superforecasters think this way, enabling them to reach a comprehensive understanding of what they know and what they do not know.
How this book can help you
There’s much more that I haven’t even attempted to go into in the section above, such as the use of Brier Scores to measure performance and Tetlock’s discoveries that combining the forecasts of superforecasters led to even better prediction accuracy.
At its core, Superforecasting will lead you to perform an audit of your own decision making skills. Decisions are based on predictions about the future so the better you are at framing predictions more specifically and then forecasting accurately, the better your decisions will be.
Some good further reading/watching/listening
Watch Predicting the future: A lecture by Philip Tetlock
Where to get it
Get Superforecasting on Amazon.
Get Superforecasting as an audiobook.
About the author
Philip E. Tetlock is a Canadian-American political science writer. He is currently the Annenberg University Professor at the University of Pennsylvania. He co-founded the Good Judgement Project and has published multiple books including Expert Political Judgment and Superforecasting.
Dan Gardner is a New York Times best-selling author, speaker, consultant, and freelance writer/editor. Gardner’s books have been published in 25 countries and 19 languages. He holds a law degree and a masters in modern history. He is currently a senior fellow with the University of Ottawa’s Graduate School of Public Policy and International Affairs.
Published by
Crown Publishers
Publication Date
September 24, 2015
Want to Build an
Effective Reading Habit?
Subscribe to our sporadic newsletter
and we will send you a pdf guide:
How to Build an Effective
Reading Habit in 2021.