Book contents
- Frontmatter
- Contents
- Preface
- Introduction
- PART ONE PROBABILITY IN ACTION
- PART TWO ESSENTIALS OF PROBABILITY
- 7 Foundations of probability theory
- 8 Conditional probability and Bayes
- 9 Basic rules for discrete random variables
- 10 Continuous random variables
- 11 Jointly distributed random variables
- 12 Multivariate normal distribution
- 13 Conditioning by random variables
- 14 Generating functions
- 15 Discrete-time Markov chains
- 16 Continuous-time Markov chains
- Appendix: Counting methods and ex
- Recommended reading
- Answers to odd-numbered problems
- Bibliography
- Index
16 - Continuous-time Markov chains
Published online by Cambridge University Press: 05 August 2012
- Frontmatter
- Contents
- Preface
- Introduction
- PART ONE PROBABILITY IN ACTION
- PART TWO ESSENTIALS OF PROBABILITY
- 7 Foundations of probability theory
- 8 Conditional probability and Bayes
- 9 Basic rules for discrete random variables
- 10 Continuous random variables
- 11 Jointly distributed random variables
- 12 Multivariate normal distribution
- 13 Conditioning by random variables
- 14 Generating functions
- 15 Discrete-time Markov chains
- 16 Continuous-time Markov chains
- Appendix: Counting methods and ex
- Recommended reading
- Answers to odd-numbered problems
- Bibliography
- Index
Summary
Many random phenomena happen in continuous time. Examples include occurrence of cell phone calls, spread of epidemic diseases, stock fluctuations, etc. A continuous-time Markov chain is a very useful stochastic process to model such phenomena. It is a process that goes from state to state according to a Markov chain, but the times between state transitions are continuous random variables having an exponential distribution.
The purpose of this chapter is to give an elementary introduction to continuous-time Markov chains. The basic concept of the continuous-time Markov chain model is the so-called transition rate function. Several examples will be given to illustrate this basic concept. Next we discuss the time-dependent behavior of the process and give Kolmogorov's differential equations to compute the time-dependent state probabilities. Finally, we present the flow-rate-equation method to compute the limiting state probabilities and illustrate this powerful method with several examples dealing with queueing systems.
Markov chain model
A continuous-time stochastic process {X(t), t ≥ 0} is a collection of random variables indexed by a continuous time parameter t ∈ [0, ∞), where the random variable X(t) is called the state of the process at time t. In an inventory problem X(t) might be the stock on hand at time t and in a queueing problem X(t) might be the number of customers present at time t. The formal definition of a continuous-time Markov chain is a natural extension of the definition of a discrete-time Markov chain.
- Type
- Chapter
- Information
- Understanding Probability , pp. 507 - 531Publisher: Cambridge University PressPrint publication year: 2012