To send content items to your account,
please confirm that you agree to abide by our usage policies.
If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account.
Find out more about sending content to .
To send content items to your Kindle, first ensure firstname.lastname@example.org
is added to your Approved Personal Document E-mail List under your Personal Document Settings
on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part
of your Kindle email address below.
Find out more about sending to your Kindle.
Note you can select to send to either the @free.kindle.com or @kindle.com variations.
‘@free.kindle.com’ emails are free but can only be sent to your device when it is connected to wi-fi.
‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.
In this chapter we turn to the study of degenerate controlled diffusions. For the nondegenerate case the theory is more or less complete. This is not the case if the uniform ellipticity hypothesis is dropped. Indeed, the differences between the nondegenerate and the degenerate cases are rather striking. In the nondegenerate case, the state process X is strong Feller under a Markov control. This, in turn, facilitates the study of the ergodic behavior of the process. In contrast, in the degenerate case, under a Markov control, the Itô stochastic differential equation (2.2.1) is not always well posed. From an analytical viewpoint, in the nondegenerate case, the HJB equation is uniformly elliptic and the associated regularity properties benefit its study. The degenerate case, on the other hand, is approached via a particular class of weak solutions known as viscosity solutions. This approach does not yield as satisfactory results as in the case of classical solutions. In fact ergodic control of degenerate diffusions should not be viewed as a single topic, but rather as a class of problems, which are studied under various hypotheses.We first formulate the problem as a special case of a controlled martingale problem and then summarize those results from Chapter 6 that are useful here. Next, in Section 7.3, we study the HJB equations in the context of viscosity solutions for a specific class of problems that bears the name of asymptotically flat diffusions.
We conclude by highlighting a string of issues that still remain open.
In the controlled martingale problem with ergodic cost, we obtained existence of an optimal ergodic process and optimal Markov process separately, but not of an optimal ergodic Markov process, as one would expect from one's experience with the nondegenerate case. This issue still remains open. In particular it is unclear whether the Krylov selection procedure of Section 6.7, which has been used to extract an optimal Markov family for the discounted cost problem under nondegeneracy, can be similarly employed for the ergodic problem. The work in Bhatt and Borkar  claims such a result under very restrictive conditions, but the proof has a serious flaw.
The HJB equation was analyzed in two special cases. The general case remains open. In particular, experience with discrete state space problems gives some pointers:
(a) In the multichain case for Markov chains with finite state space S and finite action space A, a very general dynamic programming equation is available due to Howard , viz.,
for i ∈ S. Here the unknowns are the value function V and the state dependent optimal cost ϱ. An analog of this for the degenerate diffusion case could formally be written down as
This comprehensive volume on ergodic control for diffusions highlights intuition alongside technical arguments. A concise account of Markov process theory is followed by a complete development of the fundamental issues and formalisms in control of diffusions. This then leads to a comprehensive treatment of ergodic control, a problem that straddles stochastic control and the ergodic theory of Markov processes. The interplay between the probabilistic and ergodic-theoretic aspects of the problem, notably the asymptotics of empirical measures on one hand, and the analytic aspects leading to a characterization of optimality via the associated Hamilton–Jacobi–Bellman equation on the other, is clearly revealed. The more abstract controlled martingale problem is also presented, in addition to many other related issues and models. Assuming only graduate-level probability and analysis, the authors develop the theory in a manner that makes it accessible to users in applied mathematics, engineering, finance and operations research.
Ergodic is a term appropriated from physics that derives from the Greek words έργον and οόος meaning “work” and “path”. In the context of controlled Markov processes it refers to the problem of minimizing a time averaged penalty, or cost, over an infinite time horizon. It is of interest in situations when transients are fast and therefore relatively unimportant, and one is essentially comparing various possible equilibrium behaviors. One typical situation is in communication networks, where continuous time and space models arise as scaled limits of the underlying discrete state and/or time phenomena.
Ergodic cost differs from the simpler “integral” costs such as finite horizon or infinite horizon discounted costs in several crucial ways. Most importantly, one is looking at a cost averaged over infinite time, whence any finite initial segment is irrelevant as it does not affect the cost. This counterintuitive situation is also the reason for the fundamental difficulty in handling this problem analytically – one cannot use for this problem the naive dynamic programming heuristic because it is perforce based on splitting the time horizon into an initial segment and the rest. One is thus obliged to devise altogether different techniques to handle the ergodic cost. One of them, the more familiar one, is to treat it as a limiting case of the infinite horizon discounted cost control problem as the discount factor tends to zero. This “vanishing discount” approach leads to the correct dynamic programming, or “Hamilton–Jacobi–Bellman” (HJB) equation for the problem, allowing one to characterize optimal control policies at least in the “nicer” situations when convenient technical hypotheses hold.