Superforecasting - 2: What Makes for Great Forecasting?

In the competition, the randomly enrolled, volunteer forecasters crushed the other groups. Year after year. So what made for good forecasters?

In his terrific book, Superforecasting, Philip Tetlock says there are many aspects to being a great forecaster. For one, good forecasters don’t subscribe to one Big Idea. Rather, they hold multiple perspectives and don’t get carried away by ideological beliefs.

“There is no divinely mandated link between morality and competence.”

They also do not believe anything is destined to happen. As Daniel Kahneman famously said, there was a 50% chance that each of Stalin and Hitler could have been born as girls, not boys. It was only a 25% chance that both were boys. And who can really say that history wouldn’t have been very different then? Who’s to say that the Soviet Union would have imploded peacefully even without a Gorbachev at the helm?

The best forecasters can break down seemingly impossible questions (“Will they find plutonium during Yasser Arafat’s autopsy?”) into simpler questions: (1) What are the different ways in which plutonium could enter a body? (2) Could it enter his body after his death? (3) How long after death can it be detected? Etc.

They also do not change the question to an easier one, even unconsciously. In the Arafat question, they would not change the question to “Would Israel have poisoned Arafat?”, a mistake many others tend to make.

They know and compensate for human bias: an easily visualizable event seems more probable, regardless of its true probability. When in doubt, they start with historical averages as the baseline. Thus, for “After the Charlie Hebdo attacks, how many more terrorist attacks would happen in Europe that year?”, they start with previous year’s count of terrorist attacks in Europe, not the one, highly publicized attack that happened recently.

They tend to follow simple tricks like rewording the question to its opposite. Thus “Will South Africa issue a visa for the Dalai Lama within 6 months?” is answered both ways: as the original question as well as its opposite: Will South Africa not issue a visa for the Dalai Lama within that year? This ensures they truly evaluate both possibilities.

They are not just open-minded but actively open-minded, i.e., they deliberately seek out opposing views, not just process such views when they encounter them:

“Beliefs are hypotheses to be tested, not treasures to be guarded.”

They can make their peace with being wrong despite the right reasons e.g. a freak accident might make a forecast wrong. Thus, they don’t change their methods based on the outcome; they change their methods only when they see a flaw. And they don’t rationalize with “I was right but for this” kind of thinking.

And as you might have guessed, superforecasters are comfortable with probability and numbers. They also tend to update their forecasts far more often than others as time passes and new information comes to light (Updates were allowed in the project Tetlock ran). But here’s the key point: though they made frequent updates, they usually didn’t change the predicted probability by much. In other words, they could think and absorb a lot of new information but not let each new item sway them disproportionately.

Not being experts in any field may actually have helped: they had less ego invested! And they were voracious learners. If they didn’t know anything about the background to a question (“Who will win the next election in Angola?”), they’d start finding out everything they could about the country, its politics etc.

Being humble, strangely, was not a must-have quality. After all, if you think you don’t know much, how would you have confidence in your prediction? As Tetlock said, they probably struck the right combo:

“It’s quite possible to think highly of yourself and be intellectually humble.”

So it’s not one quality. It’s a lot of them, and yes, a lot of hard work. And even then, they could be wrong. Remember that the next time you watch those election polls. Unless, of course, you’re just looking for entertainment.