The word entropy was invented in 1865 by the German physicist and mathematician Rudolf Clausius, one of the founding pioneers of thermodynamics. In the theory of systems in thermodynamical equilibrium, the entropy quantifies the degree of “disorder” in the system. The second law of thermodynamics states that, when an isolated system passes from an equilibrium state to another, the entropy of the final state is necessarily bigger than the entropy of the initial state. For example, when we join two containers with different gases (oxygen and nitrogen, say), the two gases mix with one another until reaching a new macroscopic equilibrium, where they are both uniformly distributed in the two containers. The entropy of the new state is larger than the entropy of the initial equilibrium, where the two gases were separate.

The notion of entropy plays a crucial role in different fields of science. An important example, which we explore in our presentation, is the field of information theory, initiated by the work of the American electrical engineer Claude Shannon in the mid 20th century. At roughly the same time, the Russian mathematicians Andrey Kolmogorov and Yakov Sinai were proposing a definition of the entropy of a system in ergodic theory. The main purpose was to provide an invariant of ergodic equivalence that, in particular, could distinguish two Bernoulli shifts. This Kolmogorov–Sinai entropy is the subject of this chapter.

In Section 9.1 we define the entropy of a transformation with respect to an invariant probability measure, by analogy with a similar notion in information theory. The theorem of Kolmogorov–Sinai, which we discuss in Section 9.2, is a fundamental tool for the actual calculation of the entropy in specific systems. In Section 9.3 we analyze the concept of entropy from a more local viewpoint, which is more closely related to Shannon's formulation of this concept. Next, in Section 9.4, we illustrate a few methods for calculating the entropy, by means of concrete examples.

In Section 9.5 we discuss the role of the entropy as an invariant of ergodic equivalence. The highlight is the theorem of Ornstein (Theorem 9.5.2), according to which any two-sided Bernoulli shifts are ergodically equivalent if and only if they have the same entropy. In that section we also introduce the class of Kolmogorov systems, which contains the Bernoulli shifts and is contained in the class of systems with Lebesgue spectrum.