In this Appendix we consider data compression algorithms which guarantee that the reconstructed file from the compressed bitstream will bit-by-bit coincide with the original input file. These lossless (entropy-coding) algorithms have many applications. They can be used for archiving different types of datum: texts, images, speech, and audio. Since multimedia data usually can be considered as outputs of a source with memory or, in other words, have significant redundancy, then the entropy coding can be combined with different types of preprocessing, for example, linear prediction.

However, multimedia compression standards are based mainly on lossy coding schemes which provide much larger compression ratios compared to lossless coding. It might seem that for such compression systems entropy coding plays no role but that is not the case. In fact, lossless coding is an important part of lossy compression standards also. In this case we consider quantized outputs of a preprocessing block as the outputs of a discrete-time source, estimate statistics of this source, and apply to its outputs entropy-coding techniques.

Symbol-by-symbol lossless coding

Let a random variable take on values x from the discrete set X = {0, 1, …, M − 1} and let p(x) > 0 be the probability mass function of X or, in other words, the probability distribution on the set X. If we do not take into account that different values x are not equally probable, we can only construct a fixed-length code for all possible values x ∈ X.