Predictions and Confidence Values

When thinking of the increasing frequency with which pollsters fail to predict election outcomes (Modi in 2014, Kejriwal sweep of Delhi, Brexit, Trump), I’ve wondered whether the issue might be that no pollster wants to announce a radically-different-from-all-the-rest prediction. Does he feel it’s better to fail with the pack than to fail by being the outlier? If he fails with the pack, at least he can say, “Hey, nobody else got it either”.

But is that what is really happening? Or is the problem that pollsters don’t bring out a fairly obvious point made by Elroy Dimson as part of their predictions:
“Risk means more things can happen than will happen.”
Put differently, it means that you can only assign probabilities to your predictions; almost nothing is a certainty. Since that’s obviously true, shouldn’t the pollster not just say that he believes that X will win, but also what his confidence in that prediction is?

Nate Silver is a guy who has been assigning probabilities (confidence values) to his predictions of elections and sports results. But, you wonder, who wants to hear a predictor say X is more likely to win than Y? Wouldn’t that just sound like a fence-sitter? Don’t we seek certainties? Who’d pay for such an opinion?

And yet, Silver is vastly respected because he calls out probabilities! When he says X is 55% likely to win, people can see what the odds are (55-45), it gives an idea of how close he feels the call is, and allows people to gauge how likely a swing of that magnitude is in the time between prediction and election day.

It gets even better when Silver makes many predictions and cites probabilities for each of them. Like making state by state predictions in a national election. The count of predictions with lower probabilities (confidence) on his list gives you a feel of how often he himself expects to be wrong in his predictions.

The key benefit with this approach is that if Silver is wrong about a state, you can see how confident he was to begin with. Being wrong when he was 53% confident will happen now and then. But if he’s wrong when he was 65% confident, that would (rightly) make you wonder about his approach.

Contrast that with how our media houses show poll predictions. With no probabilities (confidence values) called out upfront, we have no way to know whether their being wrong was always on the cards (a la the 53% scenario) or something unexpected (a la the 65% scenario).

Maybe it’s time the mainstream media switched to the Silver approach of calling out confidence values (probabilities): it might lend some credibility to a field that is genuinely tough but ends up projecting itself to be an oracle with its current style of presentation.

Comments

Post a Comment

Popular posts from this blog

Student of the Year

Image
As a toddler, my daughter loved Alia Bhatt’s songs, esp. the ones from Student of the Year . (That movie did have many good songs and well-choreographed dances). Fast forward to present day, when she is 12 yo. She rarely watches Hindi movies now, but one of the few ones she likes of late, Raja aur Rani ki Prem Kahani has, yes, Alia Bhatt.   If I had any hopes that there was anything to be read in the name of her toddler-time favorite ( Student of the Year ), they have been dashed. Our conversations come exam time are perfectly captured by this photo and its caption: In fact, the pic below captures how she and I look at her answer sheets – me intently, she with an expression of “It’s over, so why are you even looking at it?”   Of late, we pull her leg and tell her that she isn’t showing any signs of academic prowess, she’d better find a rich guy to marry. Some time later, she showed us this meme on the Internet: “Me No Study, Me No Care Me Go Marry Millionaire If He Die
Read more

The Retort of the "Luxury Person"

Image
Coming up with biting and witty retorts. It’s something we dream of, but rarely manage to pull off. Worse, the more we stew over the offending remark, the more likely we are to come up with that dream retort, as Calvin ruefully points out: During our last holiday at the Taj Resort, seeing my wife carrying towels to our poolside chairs, one of the kids there mistook her for a member of the staff and asked for a towel. When my wife narrated this to us, I did not imagine that my 8 yo daughter would come up with her own retort. But a day later, my daughter recapped the incident, an incident that didn’t even involve her in the first place , and said, “You know what I’d have told that kid who asked for the towel?”. And without waiting for an answer, she told us anyway: “I don’t work here. I too am a luxury person who is living at this resort. Go get your own towel.” We were highly amused by the phrase she had coined: “luxury person”. It certainly captures the kind of holid
Read more

Animal Senses #7: Touch and Remote Touch

Instinctively, we think touch is based on contact. Not entirely true, explains Ed Yong in Immense World – there’s a more important aspect than contact: “These incredible feats are possible through movement. If you rest a fingertip upon a surface, you can get only a limited idea of its features. But as soon you’re allowed to move, everything changes. Hardness becomes apparent with a press. Textures resolve at a stroke.” As you run your fingers over a surface, they hit the tiny peaks and troughs and set off vibrations in the mechanoreceptors at their tips.   At the tip of its snout, the star-nosed mole has many pairs of hairless appendages. It uses them to touch things around itself. Alive or dead? Food or ignore? It does these movements at an insane speed. In the time it takes to blink our eyes, the star-nosed mole can probe, identify and swallow its food. “Light may be the fastest thing in the universe, but light sensors have their limits, and the star-nosed mole’s sense of
Read more