Data: What Gets Reported, How the Process Works

Tim Harford, in How to Make the World Add Up, his book on how to make sense of data and statistics, highlights the problem with what comes to our attention:

“When media outlets want to grab our attention, they look for stories that are novel and unexpected over a short time horizon – and these stories are more likely to be bad than good.”

 

As an example, he cites all those headlines about the growth in global inequality. As always, the devil is in the details: yes, the gap between the richest of the richest and the poorest of the poor is indeed growing by leaps and bounds. But it’s also true that about 150,000 people are coming out of poverty every day. Yes, every day. Over the last couple of decades, China and India have pulled out almost a billion people out of poverty. See how this example aligns with Harford’s point on the headlines? Short term v/s long term; bad news v/s good news.

 

Plus, as Harford reminds us, both “Another terrible crime has occurred!” and “Overall, crime is way down” can be true at the same time. No surprises for which headline would make the news...

 

Another area to understand is the backstory to the data/facts. Take academia, for example. In people related topics, the first study that finds something new or unexpected gets published (after some checks, peer reviews etc). But here’s the kicker: if subsequent attempts to reproduce the result don’t succeed, or even contradict the first study, they do not get published. (This is only in people related sciences, not in physical sciences). This has major consequences: if a study showed something surprising, it will get published and make headlines. But any followups that disagree will never see light of day. And so, in the general population, the first study and its result sticks and becomes the accepted reality. There’s even a term for this: “publication bias”…

 

In addition, many scientific studies are HARK’ed. HARK stands for “Hypothesising After the Results are Known”. See the danger with this? You do a study in full sincerity, get some results, then come up with a theory based on the results. And cite the results as support for the theory. The last step is the problem. Sure, one always gathers facts and then comes up with a theory. But one should do new tests after the theory has been framed, and only those should be cited in support. Can you be sure that study you just saw didn’t HARK it?

 

These examples may be from academia, but the points apply across fields. Once you know how things work behind the scenes, you can understand the possible side-effects, the blind spots, the likelihood of conclusions based on incomplete data. Note that Harford isn’t talk of deliberate deceit here (we know that problem all too well); he is talking of structural risks that exist no matter how data is gathered and interpreted.

 

Tough lessons to internalize, no doubt, but worth trying anyway.

Comments

Post a Comment

Popular posts from this blog

Student of the Year

Image
As a toddler, my daughter loved Alia Bhatt’s songs, esp. the ones from Student of the Year . (That movie did have many good songs and well-choreographed dances). Fast forward to present day, when she is 12 yo. She rarely watches Hindi movies now, but one of the few ones she likes of late, Raja aur Rani ki Prem Kahani has, yes, Alia Bhatt.   If I had any hopes that there was anything to be read in the name of her toddler-time favorite ( Student of the Year ), they have been dashed. Our conversations come exam time are perfectly captured by this photo and its caption: In fact, the pic below captures how she and I look at her answer sheets – me intently, she with an expression of “It’s over, so why are you even looking at it?”   Of late, we pull her leg and tell her that she isn’t showing any signs of academic prowess, she’d better find a rich guy to marry. Some time later, she showed us this meme on the Internet: “Me No Study, Me No Care Me Go Marry Millionaire If He Die
Read more

Animal Senses #7: Touch and Remote Touch

Instinctively, we think touch is based on contact. Not entirely true, explains Ed Yong in Immense World – there’s a more important aspect than contact: “These incredible feats are possible through movement. If you rest a fingertip upon a surface, you can get only a limited idea of its features. But as soon you’re allowed to move, everything changes. Hardness becomes apparent with a press. Textures resolve at a stroke.” As you run your fingers over a surface, they hit the tiny peaks and troughs and set off vibrations in the mechanoreceptors at their tips.   At the tip of its snout, the star-nosed mole has many pairs of hairless appendages. It uses them to touch things around itself. Alive or dead? Food or ignore? It does these movements at an insane speed. In the time it takes to blink our eyes, the star-nosed mole can probe, identify and swallow its food. “Light may be the fastest thing in the universe, but light sensors have their limits, and the star-nosed mole’s sense of
Read more

The Retort of the "Luxury Person"

Image
Coming up with biting and witty retorts. It’s something we dream of, but rarely manage to pull off. Worse, the more we stew over the offending remark, the more likely we are to come up with that dream retort, as Calvin ruefully points out: During our last holiday at the Taj Resort, seeing my wife carrying towels to our poolside chairs, one of the kids there mistook her for a member of the staff and asked for a towel. When my wife narrated this to us, I did not imagine that my 8 yo daughter would come up with her own retort. But a day later, my daughter recapped the incident, an incident that didn’t even involve her in the first place , and said, “You know what I’d have told that kid who asked for the towel?”. And without waiting for an answer, she told us anyway: “I don’t work here. I too am a luxury person who is living at this resort. Go get your own towel.” We were highly amused by the phrase she had coined: “luxury person”. It certainly captures the kind of holid
Read more