Viswanathan K's Thoughts

Back in 2008, Chris Anderson wrote an oft-quoted article on what he called the “End of Theory” (exaggerated for effect) in science. Here’s the summary of his article: -          The scientific way has been to come up with models that describe reality; then test those models for any errors or mismatches with what is observed. -          Conversely, he said: “Scientists are trained to recognize that correlation is not causation, that no conclusions should be drawn simply on the basis of correlation between X and Y (it could just be a coincidence). Instead, you must understand the underlying mechanisms that connect the two.,,Data without a model is just noise.” -          Then came Anderson’s kicker: in an age where we were getting enormous amounts of data about just about everything (aka Big Data), he said that “this approach to science — hypothesize, model, t...

In response to my blog on the three generations of the Internet , my dad had commented: “Are we all stuffing ourselves with data and information, with very little time and inclination left for sharpening our innate, marvelous tool that evolution has led us to - the ability for digging meaning out of abundant data?...Will we have humankind reduced its ability to mind's ability for keen insights, failing to appropriately sharpening our grand mind potential?” Such questions have been asked and debated for years (ironically) on the Internet! Is correlation good enough? For example, Big Data would allow algorithms to tell you where the planet would be without ever discovering Kepler’s laws of planetary motion; but is that the same as knowledge? On the other hand, can laws only be found for the inanimate universe? And is Big Data the (only) way to go when it comes to predicting humans? Is George Box’s statement (“All models are wrong, but some are useful.” ) so true for humans th...

Statistics is all about analyzing data for patterns. In the past, the size and quality of the sample set was critical. Sometimes, even a problem. Enter Big Data. Or as Kenneth Neil Cukier and Viktor Mayer-Schoenberger wrote in their article, The Rise of Big Data wrote: “But if we collect all the data -- “n = all,” to use the terminology of statistics -- the problem disappears.” Sure, “n = all” is an exaggeration. But it is true that the size of data samples has gone through the roof over the last decade or so. And no, Big Data doesn’t just refer to the size of the data: “Big data is also characterized by the ability to render into data many aspects of the world that have never been quantified before; call it “datafication.”” A few examples would help understand “datafication” better: take location and friendship. They got datafied due to GPS and Facebook respectively! And if datafication is here, can algorithms be far behind? Google Translate is based on statisti...