Data and Definitions

In his book on how to make sense of data and statistics titled How to Make the World Add Up, Tim Harford points out that even when we ask questions, we tend to start by asking about the sample size, the margins of error etc. Whereas there’s an even more basic question we should be asking first.

 

Let’s go over a few examples he cites to get to that question. In the US, for example, a foetus born at 24 weeks (and then dies) is counted as “infant mortality”. Whereas the same thing in Finland would be counted as a “miscarriage”. As you can see, such definitions affect the infant mortality rate numbers in the two countries. See the danger? If you didn’t know this, you’d likely have concluded that there is a difference in the two countries, tried to find a cause for it, and depending on your leanings, even concluded that it’s another sign that Scandinavia rocks in public welfare spending!

 

How do you measure poverty? A proxy that is used often is net worth (what you have minus what you owe). Now take a farmer who has a loan of a few lakhs, with little prospect of ever being able to repay it. If he owns practically nothing, his net worth would be a few lakhs (negative). Next look at a doctor who just graduated from college, but with a college loan to repay. She too has a negative net worth. In fact, the college grad is probably even more negative than that farmer. But surely she shouldn’t be counted as poor! Are you sure the system used to measure poverty would have noticed, let alone differentiated such things?

 

How about a policy paper in the UK that was based on the number of unskilled workers. Fine, but who defines what’s “skilled”? In this case, “unskilled” was based on income. Based on the number they used as the cutoff, nurses and primary school teachers ended up falling under the unskilled category!

 

We hear of the number of gun deaths in the US all the time. We tend to think most of that must be murder, with some mass murder at times. After all, those are the cases the media highlights. In fact, 60% of those deaths are not even accidents, but (hold your breath) suicide…

 

How about suicide rates among kids? How do you know it wasn’t an accident? In the UK, their counting methodology doesn’t even try to find out. Instead, it has a flat policy: if the kid’s under 15, it’s an accident. Above 15, and it’s suicide.

 

Hardford’s takeaway for us is that the next step (after asking yourself how the data makes you feel) is to ask:

“What is being measured, or counted? What definition is being used?”