Information Theory - Part 1: What is it?

In their biography of Claude Shannon (the founder of Information Theory) titled A Mind at Play, Jimmy Soni and Rob Goodman explain the theory in a layman friendly way. Of course, it needed a genius to come up with the theory:

“Turning art into science would be the hallmark of Shannon’s career.”

And what could be more art-sy and non-science-sy than an abstract concept like information?

Shannon, building on the works of Harry Nyquist and Ralph Hartley, considered an example. If a biased coin always landed heads, you have no uncertainty about how it lands, and so the information conveyed by announcing its state is zero. But if it is a perfectly unbiased coin, then the information conveyed on how it lands is the maximum. Ergo, Shannon decided that “information” is a measure of the reduction in uncertainty on the subject.

Now Shannon needed a unit for this thing called “information”. He realized the smallest piece of information is a Yes/No answer. Further, all information can be conveyed via a series of multiple Yes/No questions (think of the game, 20 Questions). Therefore, he picked the unit as the smallest piece, a choice between 0 and 1, a “binary digit”, a name he then shortened to “bit”.

This was all theoretical stuff. But Shannon worked for Bell Labs and was far more interested in the topic of the transmission of information and more specifically, the problem of noise corrupting the signal during transmission:

“He proved a precise relationship between a channel’s capacity and two of its other qualities: bandwidth (or the range of frequencies it could accommodate) and its ratio of signal to noise.”

This was groundbreaking work because it proved that there was a “speed limit in bits per second on accurate communication in any medium”, a number that would come to be known as the Shannon limit:

“Shannon gave every subsequent generation of engineers a mark to aim for, as well as a way of knowing when they were wasting their time in the pursuit of the hopeless.”

As if all that wasn’t impressive, Shannon went even further and came up something even more “miraculous or inconceivable”, depending on your perspective:

“The answer to noise is not in how loudly we speak, but in how we say things.”

Here’s an example of what “how we say things” means. If the codes were:

A = 00, B = 01, C = 10, D = 11

Then any one bit getting distorted by noise (0 to 1 or 1 to 0) would garble the message. But if you added redundancy to the codes:

            A = 00000, B = 00111, C = 11100, D = 11011

With the above code, you’d be able to detect if a single bit got inverted by noise. Even fix it at the receiving end.

All of the above is why Claude Shannon is considered the “Father of Information Theory”:

-       He defined information as something measurable;

-       He showed that the actual information itself didn’t matter! Bits were the universal unit for everything from text to pictures to anything at all;

-       He showed that there was an upper limit on transmission speeds;

-       And he showed how to deal with noise, and the tradeoffs involved.

And as Charles Seife wrote in Decoding the Universe:

“(Information theory) had much greater repercussion than merely helping Ma Bell.”

Seife goes on to add:

“(Shannon) created the third great revolution in physics in the twentieth century: as did relativity and quantum theory, information theory radically changed the way scientists looked at the universe.”

Let’s look into those aspects next…