In this chapter we present a study of probabilistic aspects of codes. We have already seen in Chapters 2 and 3 that probability distributions play an important role in this theory.

In Section 13.1, we present some basics on probability measures, and we state and prove Kolmogorov's extension theorem. In Section 13.2, the notion of density of a subset L of A * is introduced. It is the limit in mean, provided it exists, of the probability that a word of length n is in L. In Section 13.3, we introduce the topological entropy and we give a way to compute it for a free submonoid. We will see how it is related to the results of Chapter 2 on Bernoulli distributions.

In Section 13.4, we describe how to compute the density of a set of words by defining probabilities in abstract monoids. In Section 13.5, we use this study for the proof of a fundamental formula (Theorem 13.5.1) that relates the density of the submonoid generated by a thin complete code to that of its sets of prefixes and suffixes.

Probability

We start with a short description of probability spaces, random variables, infinite words, and a result on the average length of prefix codes. We then give a proof of Kolmogorov's extension theorem.

Let S be a set. A family F of subsets of S is a Boolean algebra of subsets of S if it contains S and is closed under finite unions and under complement.