Information theory is an important field that has made significant contribution to deep learning and AI, and yet is unknown to many. Information theory can be seen as a sophisticated amalgamation of basic building blocks of deep learning: calculus, probability and statistics. Some examples of concepts in AI that come from Information theory or related fields:In the early 20th century, scientists and engineers were struggling with the question: “How to quantify the information? Is there a analytical way or a mathematical measure that can tell us about the information content?”. For example, consider below two sentences:It is not difficult to tell that the second sentence gives us more information since it also tells that Bruno is “big” and “brown” in addition to being a “dog”. How can we quantify the difference between two sentences? Can we have a mathematical measure that tells us how much more information second sentence have as compared to the first?Scientists were struggling with these questions. Semantics, domain and form of data only added to the complexity of the problem. Then, mathematician and engineer Claude Shannon came up with the idea of “Entropy” that changed our world forever and marked the beginning of “Digital Information Age”.Shannon proposed that the “semantic aspects of data are irrelevant”, and nature and meaning of data doesn’t matter when it comes to information content. Instead he quantified information in terms of probability distribution and “uncertainty”. Shannon also introduced the term “bit”, that he humbly credited to his colleague John Tukey. This revolutionary idea not only laid the foundation of Information Theory but also opened new avenues for progress in fields like artificial intelligence.Below we discuss four popular, widely used and must known Information theoretic concepts in deep learning and data sciences:Also called Information Entropy or Shannon Entropy.Entropy gives a measure of uncertainty in an experiment. Let’s consider two experiments:If we compare the two experiments, in exp 2 it is easier to predict the outcome as compared to exp 1. So, we can say that exp 1 is inherently more uncertain/unpredictable than exp 2. This uncertainty in the experiment is measured using entropy.Therefore, if there is more inherent uncertainty in the experiment then it has higher entropy. Or lesser the experiment is predictable more is the entropy. The probability distribution of experiment is used to calculate the entropy.A deterministic experiment, which is completely predictable, say tossing a coin with P(H)=1, has entropy zero. An experiment which is completely random, say rolling fair dice, is least predictable, has maximum uncertainty, and has the highest entropy among such experiments.Another way to look at entropy is the average information gained when we observe outcomes of an random experiment. The information gained for a outcome of an experiment is defined as a function of probability of occurrence of that outcome. More the rarer is the outcome, more is the information gained from observing it.For example, in an deterministic experiment, we always know the outcome, so no new information gained is here from observing the outcome and hence entropy is zero.For a discrete random variable X, with possible outcomes (states) x_1,...,x_n the entropy, in unit of bits, is defined as:where p(x_i) is the probability of i^th outcome of X.Cross entropy is used to compare two probability distributions. It tells us how similar two distributions are.Cross entropy between two probability distributions p and q defined over same set of outcomes is given by:Mutual information is a measure of mutual dependency between two probability distributions or random variables. It tells us how much information about one variable is carried by the another variable.Mutual information captures dependency between random variables and is more generalized than vanilla correlation coefficient, which captures only the linear relationship.Mutual information of two discrete random variables X and Y is defined as:where p(x,y) is the joint probability distribution of X and Y, and p(x) and p(y) are the marginal probability distribution of X and Y respectively.Also called Relative Entropy.KL divergence is another measure to find similarities between two probability distributions. It measures how much one distribution diverges from the other.Suppose, we have some data and true distribution underlying it is ‘P’. But we don’t know this ‘P’, so we choose a new distribution ‘Q’ to approximate this data. Since ‘Q’ is just an approximation, it won’t be able to approximate the data as good as ‘P’ and some information loss will occur. This information loss is given by KL divergence.KL divergence between ‘P’ and ‘Q’ tells us how much information we lose when we try to approximate data given by ‘P’ with ‘Q’.KL divergence of a probability distribution Q from another probability distribution P is defined as:KL divergence is commonly used in unsupervised machine learning technique Variational Autoencoders.Information Theory was originally formulated by mathematician and electrical engineer Claude Shannon in his seminal paper “A Mathematical Theory of Communication” in 1948.Note: Terms experiments, random variable & AI, machine learning, deep learning, data science have been used loosely above but have technically different meanings.In case you liked the article, do follow me Abhishek Parbhakar for more articles related to AI, philosophy and economics.From a quick cheer to a standing ovation, clap to show how much you enjoyed this story.Finding equilibria among AI, philosophy, and economics.Sharing concepts, ideas, and codes.