When N = All

Statistics is all about analyzing data for patterns. In the past, the size and quality of the sample set was critical. Sometimes, even a problem.

Enter Big Data. Or as Kenneth Neil Cukier and Viktor Mayer-Schoenberger wrote in their article, The Rise of Big Data wrote:
“But if we collect all the data -- “n = all,” to use the terminology of statistics -- the problem disappears.”
Sure, “n = all” is an exaggeration. But it is true that the size of data samples has gone through the roof over the last decade or so.

And no, Big Data doesn’t just refer to the size of the data:
“Big data is also characterized by the ability to render into data many aspects of the world that have never been quantified before; call it “datafication.””
A few examples would help understand “datafication” better: take location and friendship. They got datafied due to GPS and Facebook respectively!

And if datafication is here, can algorithms be far behind? Google Translate is based on statistical analysis of all the content on the Net. UPS, the courier service, places sensors on vehicle parts to identify signs that it may break down soon, thereby allowing them to take pre-emptive action. A Japanese institute has datafied how you sit (posture, weight distribution etc) and is considering making car seats that will use the data to “recognize” you, thereby preventing car theft!

The critical thing about Big Data is that it changed the emphasis from causation to correlation. Knowing why is no longer necessary in a lot of scenarios. Of course, correlation is never a guarantee of anything. Like that time Google’s mining of its search queries suggested there was a flu epidemic…except there wasn’t.

Another amusing consequence of datafication in turn leading to commercial applications is the following:
“…in the future, when China censors Internet searches, it might face complaints not only about unjustly muzzling speech but also about unfairly restraining commerce.”

But one thing Big Data cannot do is predict innovation or disruptive technologies. As Henry Ford famously said:
“If I had asked people what they wanted, they would have said a faster horse.”

So as long as we know its potential and its limitations, I think we will benefit a lot from Big Data, privacy fears notwithstanding.

Comments

Post a Comment

Popular posts from this blog

Student of the Year

Image
As a toddler, my daughter loved Alia Bhatt’s songs, esp. the ones from Student of the Year . (That movie did have many good songs and well-choreographed dances). Fast forward to present day, when she is 12 yo. She rarely watches Hindi movies now, but one of the few ones she likes of late, Raja aur Rani ki Prem Kahani has, yes, Alia Bhatt.   If I had any hopes that there was anything to be read in the name of her toddler-time favorite ( Student of the Year ), they have been dashed. Our conversations come exam time are perfectly captured by this photo and its caption: In fact, the pic below captures how she and I look at her answer sheets – me intently, she with an expression of “It’s over, so why are you even looking at it?”   Of late, we pull her leg and tell her that she isn’t showing any signs of academic prowess, she’d better find a rich guy to marry. Some time later, she showed us this meme on the Internet: “Me No Study, Me No Care Me Go Marry Millionaire If He Die
Read more

Why we Deceive Ourselves

In their book, The Elephant in the Brain , the authors describe this weird aspect of deception that we practice: deceiving ourselves! That is so weird: “If our minds contain maps of our worlds, what good comes from having an inaccurate version of these maps?” The Sigmund Freud school of thought treats self-deception as a defense mechanism, “a way for the ego to protect itself”. The mind seeks to protect itself from anxiety and other negative emotions. Many object to this reasoning: since accurate information is so critical to our survival, surely distorting information would be dangerous. Wouldn’t the goal of protecting our self-esteem have been better achieved by making the brain’s self-esteem mechanism stronger instead? The new school of thought explains self-deception as “primarily outward-facing, manipulative, and ultimately self-serving”. This is based on Thomas Schelling’s work on game theory: “In a variety of scenarios, limiting or sabotaging yourself is the winn
Read more

Handling of the Satyam Scam

The Satyam scam. How does that fit into Montek Singh Ahluwalia’s book on reforms in India, Backstage ? The reason is that, as both NASSCOM and the government of India felt: “(Satyam’s) collapse would create a great deal of disruption for clients worldwide and seriously damage India’s reputation as a reliable source of software services.”   Therefore, the government decided to work with the private sector to “organize a quick takeover by a reputable new buyer, who would quickly restart the company”. But that could not be done overnight. In the meantime, the new board would have to reach out to Satyam’s clients and “reassure them that the company would survive”.   For the new board to have credibility when it made that statement, it had to be staffed with people who understood the private sector, the IT sector in particular, and commanded respect and trust. Deepak Parikh from HDFC was made chairman, and other members of the board included a former NASSCOM chairman, CII member,
Read more