Is Correlation Enough?

Back in 2008, Chris Anderson wrote an oft-quoted article on what he called the “End of Theory” (exaggerated for effect) in science. Here’s the summary of his article:
-         The scientific way has been to come up with models that describe reality; then test those models for any errors or mismatches with what is observed.
-         Conversely, he said:
“Scientists are trained to recognize that correlation is not causation, that no conclusions should be drawn simply on the basis of correlation between X and Y (it could just be a coincidence). Instead, you must understand the underlying mechanisms that connect the two.,,Data without a model is just noise.”
-         Then came Anderson’s kicker: in an age where we were getting enormous amounts of data about just about everything (aka Big Data), he said that “this approach to science — hypothesize, model, test — is becoming obsolete”. Why? Because:
“Petabytes (of data) allow us to say: "Correlation is enough."…We can throw the numbers into the biggest computing clusters the world has ever seen and let statistical algorithms find patterns where science cannot.”

My instinctive feeling was that it didn’t make sense to rely on correlations in science. As Viktor Mayer-Schönberger and Kenneth Cukier wrote in their book titled Big Data, correlations are never perfect and if we rely on correlations, we will be fooled by randomness at times.

And yet, if only we had trusted data and correlations, how many mothers’ lives might have been saved? Here’s that story: In 1847, Ignaz Semmelweis, a Viennese physician noticed that:
“Doctors were killing women by not washing their hands in between dissecting corpses and delivering their babies. The death rate from childbed fever was twice as high for doctor-assisted births as it was for midwife-assisted births.”
How did the medical community react to Semmelweis’ call for doctors to wash their hands and use antiseptic measures? They ignored him altogether. Why? Mainly because he “had no theory” to back him up: he only had data (The germ theory only came about after Semmelweis’ death).

But is that just a one-off case? Do we want our doctors and scientists to say, as Anderson proposed, that correlation is good enough? Or do we want them to understand things, to form their models and theories and only then proceed on them? Or with so much data, is it now the case that nobody can even analyze all of it before coming up with a theory to explain/fit it all? There are no easy answers…

Comments

  1. Ramiah KumarOctober 4, 2017 at 9:19 PM

    This is a crucial debate. Nobody knows the answer yet.

    When a discoverer of a law proposes something, that is derived from limited data, it extends to not only more to fit more observed data of the same phenomenon but often can show some other correlating law linked with or an extension emerging. That supports the argument mentioned, "you must understand the underlying mechanisms that connect the two.,,Data without a model is just noise".

    Evan with such cases in not only abundance but also being good at compelling our admiration to the scientist of such insight, there is a persisting opinion that all theories are only empirical: i.e. just curve fitting of observed data, nothing more.

    A little confusing for me, because I still find that by addressing something as underlying cause, we are dealing with things that are universal and also pin-pointing further direction. That doesn't seem to have takers these days.

    Nowadays I am tiring of intellectual debates. I am perhaps not sharp enough to grasp things well.

    ReplyDelete

Post a Comment

Popular posts from this blog

Student of the Year

Image
As a toddler, my daughter loved Alia Bhatt’s songs, esp. the ones from Student of the Year . (That movie did have many good songs and well-choreographed dances). Fast forward to present day, when she is 12 yo. She rarely watches Hindi movies now, but one of the few ones she likes of late, Raja aur Rani ki Prem Kahani has, yes, Alia Bhatt.   If I had any hopes that there was anything to be read in the name of her toddler-time favorite ( Student of the Year ), they have been dashed. Our conversations come exam time are perfectly captured by this photo and its caption: In fact, the pic below captures how she and I look at her answer sheets – me intently, she with an expression of “It’s over, so why are you even looking at it?”   Of late, we pull her leg and tell her that she isn’t showing any signs of academic prowess, she’d better find a rich guy to marry. Some time later, she showed us this meme on the Internet: “Me No Study, Me No Care Me Go Marry Millionaire If He Die
Read more

The Retort of the "Luxury Person"

Image
Coming up with biting and witty retorts. It’s something we dream of, but rarely manage to pull off. Worse, the more we stew over the offending remark, the more likely we are to come up with that dream retort, as Calvin ruefully points out: During our last holiday at the Taj Resort, seeing my wife carrying towels to our poolside chairs, one of the kids there mistook her for a member of the staff and asked for a towel. When my wife narrated this to us, I did not imagine that my 8 yo daughter would come up with her own retort. But a day later, my daughter recapped the incident, an incident that didn’t even involve her in the first place , and said, “You know what I’d have told that kid who asked for the towel?”. And without waiting for an answer, she told us anyway: “I don’t work here. I too am a luxury person who is living at this resort. Go get your own towel.” We were highly amused by the phrase she had coined: “luxury person”. It certainly captures the kind of holid
Read more

Animal Senses #7: Touch and Remote Touch

Instinctively, we think touch is based on contact. Not entirely true, explains Ed Yong in Immense World – there’s a more important aspect than contact: “These incredible feats are possible through movement. If you rest a fingertip upon a surface, you can get only a limited idea of its features. But as soon you’re allowed to move, everything changes. Hardness becomes apparent with a press. Textures resolve at a stroke.” As you run your fingers over a surface, they hit the tiny peaks and troughs and set off vibrations in the mechanoreceptors at their tips.   At the tip of its snout, the star-nosed mole has many pairs of hairless appendages. It uses them to touch things around itself. Alive or dead? Food or ignore? It does these movements at an insane speed. In the time it takes to blink our eyes, the star-nosed mole can probe, identify and swallow its food. “Light may be the fastest thing in the universe, but light sensors have their limits, and the star-nosed mole’s sense of
Read more