In its early years, information theory (IT) ``has perhaps been ballooned to an importance beyond its actual accomplishments'' (Shannon 1956), being applied ``to biology, psychology, linguistics, fundamental physics, economics, the theory of organizations, and many others.'' While criticizing these superficial applications, Shannon did believe that serious applications of IT concepts in other fields were forthcoming, ``indeed, some results are already quite promising - but the establishing of such applications is not a trivial matter ... but rather the slow and tedious process of hypothesis and experimental verification.'' Shannon (loc. cit.) also emphasized that ``the hard core of IT is, essentially, a branch of mathematics'' and ``a thorough understanding of the mathematical foundation ... is surely a prerequisite to other applications.''
As ``the hard core of IT is a branch of mathematics,'' one could expect a natural two-way interaction of IT with other branches of mathematics that, in addition to enriching IT, also leads to significant applications of IT ideas within mathematics. Indeed, such applications had already been around in 1956, such as Kullback's information theoretic approach to statistics, and others were to follow soon. A celebrated example (Kolmogorov 1958) was to use the IT fact that stationary coding does not increase entropy rate to show that stationary processes of different entropy rate are never isomorphic in the sense of ergodic theory. This demonstrated that not all i.i.d. processes are mutually isomorphic, solving a long-standing problem. Kolmogorov's work initiated spectacular developments in ergodic theory, and entropy became a basic concept in that field.
The times when some scientists regarded IT as a panacea have long passed, but today's information theorists are proudly aware of well established and substantial applications of their discipline in quite a few other ones. This author, a mathematician, is particularly fascinated by the many applications of IT in various branches of pure and applied mathematics, including combinatorics, ergodic theory, algebra, operations research, systems theory, and perhaps primarily probability and statistics. The goal of this paper is to give a flavor of such applications, surveying some typical ones specifically in probability and statistics.
For the applications treated in this paper, the main IT tools are the properties of information measures, the method of types, and the concept of coding. Applications of IT in probability will be treated in Section 2, and those in statistics in Section 3.