Book contents
- Frontmatter
- Contents
- Editor's Statement
- Section Editor's Foreword
- Preface
- Special Symbols
- Mathematical Theory of Entropy
- Chapter 1 Topics from Probability Theory
- Chapter 2 Entropy and Information
- Chapter 3 Information Theory
- Chapter 4 Ergodic Theory
- Chapter 5 Topological Dynamics
- Chapter 6 Statistical Mechanics
- Bibliography
- Index
- About the Author
Chapter 3 - Information Theory
Published online by Cambridge University Press: 05 June 2013
- Frontmatter
- Contents
- Editor's Statement
- Section Editor's Foreword
- Preface
- Special Symbols
- Mathematical Theory of Entropy
- Chapter 1 Topics from Probability Theory
- Chapter 2 Entropy and Information
- Chapter 3 Information Theory
- Chapter 4 Ergodic Theory
- Chapter 5 Topological Dynamics
- Chapter 6 Statistical Mechanics
- Bibliography
- Index
- About the Author
Summary
A Model of an Information System
Information theory is concerned with constructing a mathematical model for systems which transmit information and then analyzing this model. The object of this analysis is to develop techniques to reproduce at one point (the destination) a message, or an adequate approximation to a message, which has been chosen at another point (the source) and transmitted over a channel.
Fundamental to this problem is a measure of the information transmitted by such a system. It is only in terms of quantity or rate of information processed by a system that one can judge the effectiveness of the system. Central to the notion of quantity of information is an interpretation of the idea of entropy. This interpretation equates the removal of uncertainty with the gain in information. Since entropy is a numerical measure of uncertainty, this interpretation of information gives entropy its central position. The interpretation and the resulting theories were initiated by C. E. Shannon [138] in his fundamental paper “A Mathematical Theory of Communication” published in 1948.
In this chapter we shall describe a standard model of an information system and show how the notion of entropy provides a quantitative basis for measuring the information processed by a system. We shall focus on the construction of the various models, and the definitions and theorems necessary to understand these models.
- Type
- Chapter
- Information
- Mathematical Theory of Entropy , pp. 117 - 146Publisher: Cambridge University PressPrint publication year: 1984