Superforecasting - 1: Evaluating Forecasts


When we think of forecasters, sadly, we rarely know anyone beyond TV. And as Philip Tetlock writes in his terrific book, Superforecasting:
“The one undeniable talent that talking heads have is their skill at telling a compelling story with conviction…”
The aim doesn’t seem to be accuracy; rather, it’s just entertainment… and certainty.

Given that all forecasting can only be probabilistic, the recipient of the forecast needs to be open to uncertainty. Not an easy thing, since not everyone can adopt the Richard Feynman attitude:
“Doubt is not a fearful thing, but a thing of very great value.”
This opens a new question: if a weather forecaster says 70% chance of rain and it didn’t rain, was he wrong? Or was he right since he also meant 30% chance of no rain? Then again, does this just make his prediction right, rain or no rain?

Tetlock and team came up with a system to answer exactly such questions to evaluate the accuracy of forecasters. First, forecasts must have clearly defined outcomes (rain or no rain), clear timelines (within a day/month), and yes, assigned probability of being right (65%). And, secondly, a huge number of forecasts should be made to be able to gauge a forecaster meaningfully.

Ok, but what’s the way to evaluate the “70% chance of rain” kind of forecasts? The answer is called “calibration”. Perfect calibration would mean outcomes predicted with 70% probability happen 70% of the time; outcomes predicted with 60% probability happen 60% of the time and so on. Deviations from perfect calibration would indicate under-confidence or over-confidence: neither is a good thing for a forecaster, as you might have guessed.

That’s a start, but it’s not perfect. A forecaster who sticks to the safe zone of, say, 60% probability, is likely to be right more often than one who calls out 80% odds. But he’s not saying anything with too much confidence, so what good is that?

Tetlock added another criteria called “resolution”. If two forecasters predicted the same event and got it right, but one said 70% probability and other said 90% probability, the latter gets more points. Conversely, if they’re wrong, the latter is penalized more.

Combine the two (“calibration” and “resolution”) and you get the Brier score. Tetlock and team now had a solid criteria to evaluate forecasters. They enrolled volunteers for the Good Judgment Project (GJP), a project funded by the US government. A huge number of predictions were sought and tracked over the course of a year. The comparison was against university-affiliated teams and even professional intelligence analysts. What would be the findings?

Comments

Post a Comment

Popular posts from this blog

Europe #3 - Innsbruck

The last city we visited in Europe was Innsbruck in the Austrian Alps. After the cold and windy experience (at times) of Paris and Netherlands, Innsbruck was a wonderful change – it didn’t feel that cold, it was not windy all of which made it easy to walk around. Something absolutely necessary given how scenic the place is.   Since the hotel check-in time was much later in the day, we left the baggage and went to the Swarovski HQ-cum-museum about an hour by bus from Innsbruck. The crystal exhibits were brilliant – colorful, bright, shiny. Jewellery aside, it was a display of what one can do with crystals. While my wife and daughter loved the jewellery, I liked the superhero themed exhibits of Batman, Spiderman, the Hulk, and Star Wars. We didn’t buy anything there, though we did buy stuff in the Swarovski store in Innsbruck – jewellery, not the superhero stuff.   Innsbruck is a major tourist attraction during the skiing season but now in summer, we saw many groups of ...
Read more

Why we Deceive Ourselves

In their book, The Elephant in the Brain , the authors describe this weird aspect of deception that we practice: deceiving ourselves! That is so weird: “If our minds contain maps of our worlds, what good comes from having an inaccurate version of these maps?” The Sigmund Freud school of thought treats self-deception as a defense mechanism, “a way for the ego to protect itself”. The mind seeks to protect itself from anxiety and other negative emotions. Many object to this reasoning: since accurate information is so critical to our survival, surely distorting information would be dangerous. Wouldn’t the goal of protecting our self-esteem have been better achieved by making the brain’s self-esteem mechanism stronger instead? The new school of thought explains self-deception as “primarily outward-facing, manipulative, and ultimately self-serving”. This is based on Thomas Schelling’s work on game theory: “In a variety of scenarios, limiting or sabotaging yourself is the winn...
Read more

Chess is too Boring

“Chess has problems at the grandmaster level...They're not getting a great game”, wrote Christian Donlan . Amen to that: Anand and his challenger, Magnus Carlsen drew their first game in just 16 moves. It gets worse: games between the top 20 end in draws 60% of the time. And Donlan adds that “games often end in resignation when an actual victory starts to loom on the distant horizon.” That often means that the average player (or fan) often has no clue why one guy won (since he can’t see that far ahead)! Then there’s all that memorization (of openings, of end games and everything in between). All that memorization is sort of against the spirit of the game, says Zachary Burns, “We want it to be a game of creativity and positional play.” Many have tried to change the rules of the game in the past for these exact reasons, including Bobby Fischer with his Chess960 variant. More recently, David Sirlin tried to make things interesting via his variant, Chess 2. So what are his c...
Read more