Predictions and Confidence Values

When thinking of the increasing frequency with which pollsters fail to predict election outcomes (Modi in 2014, Kejriwal sweep of Delhi, Brexit, Trump), I’ve wondered whether the issue might be that no pollster wants to announce a radically-different-from-all-the-rest prediction. Does he feel it’s better to fail with the pack than to fail by being the outlier? If he fails with the pack, at least he can say, “Hey, nobody else got it either”.

But is that what is really happening? Or is the problem that pollsters don’t bring out a fairly obvious point made by Elroy Dimson as part of their predictions:

“Risk means more things can happen than will happen.”

Put differently, it means that you can only assign probabilities to your predictions; almost nothing is a certainty. Since that’s obviously true, shouldn’t the pollster not just say that he believes that X will win, but also what his confidence in that prediction is?

Nate Silver is a guy who has been assigning probabilities (confidence values) to his predictions of elections and sports results. But, you wonder, who wants to hear a predictor say X is more likely to win than Y? Wouldn’t that just sound like a fence-sitter? Don’t we seek certainties? Who’d pay for such an opinion?

And yet, Silver is vastly respected because he calls out probabilities! When he says X is 55% likely to win, people can see what the odds are (55-45), it gives an idea of how close he feels the call is, and allows people to gauge how likely a swing of that magnitude is in the time between prediction and election day.

It gets even better when Silver makes many predictions and cites probabilities for each of them. Like making state by state predictions in a national election. The count of predictions with lower probabilities (confidence) on his list gives you a feel of how often he himself expects to be wrong in his predictions.

The key benefit with this approach is that if Silver is wrong about a state, you can see how confident he was to begin with. Being wrong when he was 53% confident will happen now and then. But if he’s wrong when he was 65% confident, that would (rightly) make you wonder about his approach.

Contrast that with how our media houses show poll predictions. With no probabilities (confidence values) called out upfront, we have no way to know whether their being wrong was always on the cards (a la the 53% scenario) or something unexpected (a la the 65% scenario).

Maybe it’s time the mainstream media switched to the Silver approach of calling out confidence values (probabilities): it might lend some credibility to a field that is genuinely tough but ends up projecting itself to be an oracle with its current style of presentation.