Viswanathan K's Thoughts

In the West, we see periodic clamouring that Google or Facebook or Amazon has too much data, and that they be split up e.g. Google into search, YouTube, Gmail and Android each as a separate company. Or Facebook into, er, Facebook, Instagram and WhatsApp as separate companies.   While India sees the problem, the DEPA framework acknowledges some of the benefits of one company having a lot of data. A massive Amazon or Flipkart that delivers all over the country can bring down costs for consumers, something not possible if every company in that sector was small.   What then would be a good balance? Rahul Matthan asks us to consider a bank in in The Third Way . As the entity with all the personal financial data, it has an outsized advantage. No new entity with a new idea in the lending space can ever hope to compete against a bank with so much information. DEPA’s architecture has a way, at least in theory. The data the banks holds is actually the account owner’s data. There...

Visualized data. That’s graphs, pie-charts and all kinds of creative representations of data. While they make it easier to understand many things, they can also be misleading, write Carl Bergstrom and Jevin West in Calling Bullshit . And not always because of malice or bad intentions…   As software makes visual graphics easy, people tend to come up with eye-catching ways of presenting data. Nothing wrong with that, unless… “The attempt to be cute makes it harder for the reader to understand the underlying data.” For example, a 3D graph makes it hard to compare the bars: how much taller is the 3 rd class bar compared to the 1 st class bar? It’s hard to say (visually) since the 1 st class bar is farther away and don’t far off objects look shorter than they are?   Sure, the numerical values are written, but if one has to read it to make sense, then why use a graphic? Even worse, why create an easy-to-misread graphic?   Another common error is to show informat...

In an earlier blog, we saw the American attitude towards data and how it became the philosophy of the Internet, simply because the US was the first country on the Net and also its biggest market. Over time, the EU became a big market too. A significant difference in European views is rooted in the fact that few, if any, big Internet companies are European. Thus, the lobbying against data/privacy laws in the EU was far less (though the big American ones do lobby in EU), explains Rahul Matthan in The Third Way .   The EU attitude data is far more citizen-centric. Even before the Internet, that was the case in (Western) Europe largely because of their experience with fascism, Nazism and communism over the past century.   That history culminated in the GDPR doctrine for EU, a “full blown regulation… which became the most advanced data protection framework”. It says (1) all data about an individual belongs to that individual, not the company that collected it, (2) any dat...

Many believe that computers and software knowing and monitoring everything about us is a recent phenomenon. Not so, says Rahul Matthan in The Third Way : “Since they were first created, computers have been designed to monitor, categorize and classify us. Everything that followed from there was just the natural consequence of that original objective.”   He elaborates on that. It started in 1890, when the US wanted to conduct its census. By then, it was taking too long to conduct one. By the time they could finish conducting and collating it, it was time for the next census! The task needed to be automated and Herman Hollerith gave it a shot. He first reduced the data into a standard format – age, sex, religion, occupation etc – and second called for it to be digitized by giving census takers a card in which they had to punch holes to indicate answers. He then fed each card into a machine that “read” the information by pressing a set of electrical pins on the card. “The pins ...

“Old-school bullshit” has been there for ages – it includes lies, rumours and propaganda, write Carl Bergstrom and Jevin West in Calling Bullshit . Their book, though, is about ways in which we can detect a different kind of bullshit: “ New-school bullshit uses the language of math and science and statistics to create the impression of rigour and accuracy.” Is there a difference though between the two kinds? Yes, unlike old-school benefits where one can Google or apply one’s own knowledge, one often has no idea how to question data and numbers: “New-school bullshit can be particularly effective because many of us don’t feel qualified to challenge information that is presented in quantitative form.” The book is about ways to overcome this self-perceived limitation.   The first thing to ask, they point out, is if the data is valid. This is increasingly important since we rely and trust AI and Machine Learning algorithms so much these days. We’d do well to remember that ...

There is the company/site/app called FlightAware that provides real-time and historical flight tracking data (flight paths, statuses, cancellations, delays and predictive analytics). It is popular among both aviation enthusiasts and travellers.   Ben Burwell wrote this piece on the eternal problem engineers have to deal with: “While we as engineers might hope for aviation data to be clean and well-standardized, the real world is messy.” All kinds of assumptions about standardized data types and schemas (formats) turn out to be false, making the development of FlightAware very challenging. He lists examples of invalid assumptions from multiple categories, some unsurprising, some downright weird. Click on the hyperlinks below for examples.   Flights . Flights depart from a gate. Flights leave their gate only once. Flights take off and land at airports . Flights are never longer than a few days . Flight numbers consist of an airline’s code plus some numbers, like UA...

When it comes to Big Data, privacy and security, India’s view is totally different than that of the West. The difference is due to the answer to the following question – Do the benefits and applications of analyzing large data outweigh the privacy and security risks? And it’s not just India, all countries without pre-existing well-functioning systems look at data sets, AI and machine learning favourably since they enable better systems and better governance.   It reminded me of that term WEIRD - Western, Educated, Industrialized, Rich, and Democratic. It was used a lot to refer to the Western view of how things should be, how studies and research should be done etc is unconsciously skewed since they didn’t know or understand non-Western (or non-WEIRD) countries and cultures. Today, more and more people think that the Western views on privacy and security should be added to the list of WEIRD worldview topics , i.e., matters where Western views are inapplicable to non-WEIRD coun...

In Accelerating India’s Development , Karthik Muralidharan says: “The centrality of data for governance is seen by noting that the very origin of the word ‘statistics’ comes from its crucial role in managing the affairs of the state.” But India’s data systems are outdated. They are designed to track national progress, not for supporting day to day governance. The focus is on measuring visible inputs (how many schools?) rather than harder to measure outcomes (quality of education). The actual citizen’s experience is barely measured (was the service easy or convenient?).   Even worse, any data that is gathered is by the respective departments themselves. Who have a vested interest in making themselves look good. And who may not have the right data analysis or statistical skills anyway.   The lack of investment in our measurement infrastructure thus has many consequences: (1) money is spent on the wrong policies, (2) a focus on easily measured inputs instead of ha...

During Donald Trump’s tenure, there was a big fight between the US and China about Huawei , the Chinese telecom equipment (and smartphone) manufacturer. The West was worried that if Huawei equipment grabbed the largest share of the upcoming 5G market, then China could insert malware and spyware in telecom networks all over the world…   The West would know. As Anirudh Suri writes in The Great Tech Game , in the 19 th and 20 th centuries, Britain had the monopoly over “telegraph communication network” across the world. Progressively, physical infrastructure monopoly had resulted in a monopoly over the raw materials needed for telegraph cables. As Britain became the #1 player in telegraph systems, it was cheaper and more economical for other countries to use British systems. With ever larger systems under their control, British expertise at laying cables and repairing these systems became better. It had become a circle that reinforced British dominance over telegraph systems ac...

What are “smart” and “connected” products? “Smart” means they have sensors to measure things, software to analyze that data, and memory to store the data. And “connected” means it has ports, and wireless communication capabilities.   Earlier, the software logic to interpret the data and spot patterns in it was written by humans. But increasingly, it’s done by “machine learning” algorithms – the machine “learns” on its own by looking at huge data sets. Now, with 5G, we may be on the cusp of the next wave of changes. As communication speeds increase, it will become cheap and practical to collate multiple data sources (say tractor data with other farm equipment data with weather data) to produce previously unimaginable insights. There’s a lot of money in such intelligence.   In The Great Tech Game , Anirudh Suri says that in such a world, data is king: “Will the next retail innovation likely come from those professionals who have the data to identify new opportunities,...

In the age of the Internet, it’s the easiest thing in the world to make very jazzy graphics of data. The term for it is “infographics”. Attractive while it may be, Tim Harford points out their danger in How to Make the World Add Up : “Many of us who are dazzled by infographic don’t suspect a thing (about the data or the conclusions being drawn).” Often, the aim of the infographic is even worse: “The goal of the graphics is not to convey information but to stir feelings.”   Of course, not all infographics are wrong or misleading as I’d written a while back about Florence Nightingale and her charts . The problem is knowing which ones are valid and which ones are “persuasive art pretending to be a piece of statistical analysis”.   One tends to think that the purpose of charts is always about “inviting people to draw certain conclusions”. While that is indeed the most common use, charts can also be “exploratory” at times. Huh? “If you’re handling a complex dataset,...

By definition, any statistical analysis is based on data. Setting aside genuine errors (wrong sample size, unrepresentative sample set, wrong questions asked), Tim Harford talks at length about the problem of data used by governments, in his book, How to Make the World Add Up .   As we know all too well, data can be misleading, even without malice. With governments though, the danger of such wrong data is enormous: “It’s one thing to be wrong… (but) because the state is powerful, its misperceptions of the world often take physical form, producing well-meaning but clumsy and oppressive schemes.”   But of course, with governments, the problem is worse. They often pressurize agencies to come up with data that suits them. I’ll avoid such examples from India since that just becomes a political discussion in no time.   Independent agencies are not a magic bullet that solves the problem: “(Independent) official agencies often get them wrong… (just that) they don’...

From Tim Harford’s book on data and statistics, How to Make the World Add Up , let’s look at a couple of other things to watch out for when we see conclusions drawn.   Let’s start with an example: “Consider the historical under-representation of women in clinical trials. One grim landmark was thalidomide, which was widely used by pregnant women to ease morning sickness only for it to emerge that the drug could cause severe disability and death to unborn children.” Even if we try and collect data with the right representation of men/women, rich/poor, urban/rural etc, we still run into the problem that many don’t respond. Or certain types of people can’t be found easily. All these are a form of “dark data”. (This problem cannot be solved entirely). The general danger here is that all analysis is, by definition, done with “found data” only. And that is a lesson Harford asks us to remember: “We can and should remember to ask who or what might be missing from the data we’re b...

Tim Harford, in How to Make the World Add Up , his book on how to make sense of data and statistics, highlights the problem with what comes to our attention: “When media outlets want to grab our attention, they look for stories that are novel and unexpected over a short time horizon – and these stories are more likely to be bad than good.”   As an example, he cites all those headlines about the growth in global inequality. As always, the devil is in the details: yes, the gap between the richest of the richest and the poorest of the poor is indeed growing by leaps and bounds. But it’s also true that about 150,000 people are coming out of poverty every day. Yes, every day. Over the last couple of decades, China and India have pulled out almost a billion people out of poverty. See how this example aligns with Harford’s point on the headlines? Short term v/s long term; bad news v/s good news.   Plus, as Harford reminds us, both “Another terrible crime has occurred!” and...

In his book on how to make sense of data and statistics titled How to Make the World Add Up , Tim Harford points out that even when we ask questions, we tend to start by asking about the sample size, the margins of error etc. Whereas there’s an even more basic question we should be asking first.   Let’s go over a few examples he cites to get to that question. In the US, for example, a foetus born at 24 weeks (and then dies) is counted as “infant mortality”. Whereas the same thing in Finland would be counted as a “miscarriage”. As you can see, such definitions affect the infant mortality rate numbers in the two countries. See the danger? If you didn’t know this, you’d likely have concluded that there is a difference in the two countries, tried to find a cause for it, and depending on your leanings, even concluded that it’s another sign that Scandinavia rocks in public welfare spending!   How do you measure poverty? A proxy that is used often is net worth (what you have...

In 1954, Darrell Huff published How to Lie with Statistics . In his book on how to use, question, and understand statistical data, How to Make the World Add Up , Tim Harford agrees that Huff’s “book deserves the popularity, and the praise”, but fears Huff’s influence on people’s view of statistics may have gone too far, that Huff “made statistics seem like a stage magician’s trick”.   Today, of course, we have another problem when we encounter any stats: “It’s not that we feel every statistic is a lie, but that we feel helpless to pick out the truths. So we believe whatever we want to believe…” Harford’s book is on how we should learn to read statistical data. No, not in a dry academic way, but in ways that are very relevant to real life.   The first rule he suggests is amusing until you think a bit: When you see a stat, ask yourself how it makes you feel. Happy? Angry? Vindicated? Any strong emotion should be taken as a warning sign that you should look more close...

Data is the new oil, they say. That seems obvious when you think of Google, Facebook, Baidu and Tencent, writes Yuval Noah Harari in 21 Lessons for the 21 st Century . And yet, we give our data away so easily: “People are happy to give away their most valuable asset – their personal data – in exchange for free email services and funny cat videos. It is a bit like African and Native Americans tribes who unwittingly sold entire countries to European colonialists in exchange for colorful beads and cheap trinkets.” We are heading for a future described below: “What will happen when we can ask Google, “Hi Google, based on everything you know about cars, and based on everything you know about me (including my needs, my habits, my views on global warming, and even my opinions about Middle Eastern politics) – what is the best car for me?” Or maybe we’ll be asking Facebook or Alexa/Amazon that question, but the point is the same. If we, as individuals and as a society, don’t f...

Circular reasoning may lie at the heart of why all the criticism hurled at Facebook (and to a much lesser extent at Google) doesn’t trigger any change. Fred Turner elaborates by explaining what Facebook and Google believe in… truly: “It started with, “Don't be evil.” So then the question became, “Okay, what's good?” Well, information is good. Information empowers people. So providing information is good. Okay, great. Who provides information? Oh, right: Google provides information. So you end up in this loop where what's good for people is what's good for Google, and vice versa.” Replace “information” with “connecting people” above, and you get Facebook. Turner has a pretty good description of the current “ethics of engineering”: “The ethics of engineering are an ethics of: Does it work? If you make something that works, you’ve done the ethical thing. It’s up to other people to figure out the social mission for your object.” But today we increasingly see...

During my college days, there used to magazines like Voice & Data . Today, everything is Data. Thanks to the rise of the smartphone, the purpose of the phone has changed, as Thomas Ricker wrote in this awesome article : “With the invention of smartphones, the "phone" is just another communications app. One that is quickly being demoted from the favorites bar. Hell, most modern phone reviews don’t even bother discussing call reception.” Ian Bogost wrote this great article on why this transition happened: it’s not just a social phenomenon, it’s also technological! Mobile, being wireless, meant that “signal strength, traffic, and interference can make calls difficult or impossible”. We have come to accept that to a point where “phone calls (are) synonymous with unreliability”! In that unreliable world came all the messaging apps: being “asynchronous, a slow or failed message feels like less of a failure”, says Bogost. Then there’s the historical choice that tel...

Ernest Rutherford, the famous scientist, once declared: “All science is either physics or stamp collecting.” James Watson, (co)discoverer of the famous double helix structure of the DNA molecule, felt that the knowledge of how information is stored in DNA as something that would change the way biology was done. As Siddhartha Mukherjee wrote in his awesome book, The Gene : “Watson called the old biologists “stamp collectors” – mocking their preoccupation with the collection and classification of biological specimens.” Take this instance: a recent study declared that an animal that we considered as a single species, the giraffe, was actually 4 separate species . How did they come to the conclusion? Based on the genetic differences amongst them. Plus, the point that the 4 groups don’t breed with each other. Further, as this article says : “That analysis confirmed it’s been one to two million years since giraffe species diverged—showing how distinct the animals are from one a...