Superforecasting - 1: Evaluating Forecasts


When we think of forecasters, sadly, we rarely know anyone beyond TV. And as Philip Tetlock writes in his terrific book, Superforecasting:
“The one undeniable talent that talking heads have is their skill at telling a compelling story with conviction…”
The aim doesn’t seem to be accuracy; rather, it’s just entertainment… and certainty.

Given that all forecasting can only be probabilistic, the recipient of the forecast needs to be open to uncertainty. Not an easy thing, since not everyone can adopt the Richard Feynman attitude:
“Doubt is not a fearful thing, but a thing of very great value.”
This opens a new question: if a weather forecaster says 70% chance of rain and it didn’t rain, was he wrong? Or was he right since he also meant 30% chance of no rain? Then again, does this just make his prediction right, rain or no rain?

Tetlock and team came up with a system to answer exactly such questions to evaluate the accuracy of forecasters. First, forecasts must have clearly defined outcomes (rain or no rain), clear timelines (within a day/month), and yes, assigned probability of being right (65%). And, secondly, a huge number of forecasts should be made to be able to gauge a forecaster meaningfully.

Ok, but what’s the way to evaluate the “70% chance of rain” kind of forecasts? The answer is called “calibration”. Perfect calibration would mean outcomes predicted with 70% probability happen 70% of the time; outcomes predicted with 60% probability happen 60% of the time and so on. Deviations from perfect calibration would indicate under-confidence or over-confidence: neither is a good thing for a forecaster, as you might have guessed.

That’s a start, but it’s not perfect. A forecaster who sticks to the safe zone of, say, 60% probability, is likely to be right more often than one who calls out 80% odds. But he’s not saying anything with too much confidence, so what good is that?

Tetlock added another criteria called “resolution”. If two forecasters predicted the same event and got it right, but one said 70% probability and other said 90% probability, the latter gets more points. Conversely, if they’re wrong, the latter is penalized more.

Combine the two (“calibration” and “resolution”) and you get the Brier score. Tetlock and team now had a solid criteria to evaluate forecasters. They enrolled volunteers for the Good Judgment Project (GJP), a project funded by the US government. A huge number of predictions were sought and tracked over the course of a year. The comparison was against university-affiliated teams and even professional intelligence analysts. What would be the findings?

Comments

Post a Comment

Popular posts from this blog

Student of the Year

Image
As a toddler, my daughter loved Alia Bhatt’s songs, esp. the ones from Student of the Year . (That movie did have many good songs and well-choreographed dances). Fast forward to present day, when she is 12 yo. She rarely watches Hindi movies now, but one of the few ones she likes of late, Raja aur Rani ki Prem Kahani has, yes, Alia Bhatt.   If I had any hopes that there was anything to be read in the name of her toddler-time favorite ( Student of the Year ), they have been dashed. Our conversations come exam time are perfectly captured by this photo and its caption: In fact, the pic below captures how she and I look at her answer sheets – me intently, she with an expression of “It’s over, so why are you even looking at it?”   Of late, we pull her leg and tell her that she isn’t showing any signs of academic prowess, she’d better find a rich guy to marry. Some time later, she showed us this meme on the Internet: “Me No Study, Me No Care Me Go Marry Millionaire If He Die
Read more

The Retort of the "Luxury Person"

Image
Coming up with biting and witty retorts. It’s something we dream of, but rarely manage to pull off. Worse, the more we stew over the offending remark, the more likely we are to come up with that dream retort, as Calvin ruefully points out: During our last holiday at the Taj Resort, seeing my wife carrying towels to our poolside chairs, one of the kids there mistook her for a member of the staff and asked for a towel. When my wife narrated this to us, I did not imagine that my 8 yo daughter would come up with her own retort. But a day later, my daughter recapped the incident, an incident that didn’t even involve her in the first place , and said, “You know what I’d have told that kid who asked for the towel?”. And without waiting for an answer, she told us anyway: “I don’t work here. I too am a luxury person who is living at this resort. Go get your own towel.” We were highly amused by the phrase she had coined: “luxury person”. It certainly captures the kind of holid
Read more

Animal Senses #7: Touch and Remote Touch

Instinctively, we think touch is based on contact. Not entirely true, explains Ed Yong in Immense World – there’s a more important aspect than contact: “These incredible feats are possible through movement. If you rest a fingertip upon a surface, you can get only a limited idea of its features. But as soon you’re allowed to move, everything changes. Hardness becomes apparent with a press. Textures resolve at a stroke.” As you run your fingers over a surface, they hit the tiny peaks and troughs and set off vibrations in the mechanoreceptors at their tips.   At the tip of its snout, the star-nosed mole has many pairs of hairless appendages. It uses them to touch things around itself. Alive or dead? Food or ignore? It does these movements at an insane speed. In the time it takes to blink our eyes, the star-nosed mole can probe, identify and swallow its food. “Light may be the fastest thing in the universe, but light sensors have their limits, and the star-nosed mole’s sense of
Read more