Book contents
- Frontmatter
- Contents
- Preface
- 1 Introduction
- Part I Stochastic Models and Bayesian Filtering
- Part II Partially Observed Markov Decision Processes: Models and Applications
- Part III Partially Observed Markov Decision Processes: Structural Results
- 9 Structural results for Markov decision processes
- 10 Structural results for optimal filters
- 11 Monotonicity of value function for POMDPs
- 12 Structural results for stopping time POMDPs
- 13 Stopping time POMDPs for quickest change detection
- 14 Myopic policy bounds for POMDPs and sensitivity to model parameters
- Part IV Stochastic Approximation and Reinforcement Learning
- Appendix A Short primer on stochastic simulation
- Appendix B Continuous-time HMM filters
- Appendix C Markov processes
- Appendix D Some limit theorems
- References
- Index
11 - Monotonicity of value function for POMDPs
from Part III - Partially Observed Markov Decision Processes: Structural Results
Published online by Cambridge University Press: 05 April 2016
- Frontmatter
- Contents
- Preface
- 1 Introduction
- Part I Stochastic Models and Bayesian Filtering
- Part II Partially Observed Markov Decision Processes: Models and Applications
- Part III Partially Observed Markov Decision Processes: Structural Results
- 9 Structural results for Markov decision processes
- 10 Structural results for optimal filters
- 11 Monotonicity of value function for POMDPs
- 12 Structural results for stopping time POMDPs
- 13 Stopping time POMDPs for quickest change detection
- 14 Myopic policy bounds for POMDPs and sensitivity to model parameters
- Part IV Stochastic Approximation and Reinforcement Learning
- Appendix A Short primer on stochastic simulation
- Appendix B Continuous-time HMM filters
- Appendix C Markov processes
- Appendix D Some limit theorems
- References
- Index
Summary
This chapter gives sufficient conditions on the POMDP model so that the value function in Bellman's dynamic programming equation is decreasing with respect to the monotone likelihood ratio (MLR) stochastic order. That is, π1 ≥;r π2 (in terms of MLR dominance) implies V(π1) ≤ V(π2). To prove this result, we will use the structural properties of the optimal filter established in Chapter 10.
Giving conditions for a POMDP to have a monotone value function is useful for several reasons: it serves as an essential step in establishing sufficient conditions for a stopping time POMDPs to have a monotone optimal policy – this is discussed in Chapter 12. For more general POMDPs (discussed in Chapter 14), it allows us to upper and lower bound the optimal policy by judiciously constructed myopic policies. Please see Figure 10.1 for the sequence of chapters on POMDP structural results.
After giving sufficient conditions for a monotone value function, this chapter also gives two examples of POMDPs to illustrate the usefulness of this result:
• Example 1: Monotone optimal policy for two-state POMDP: §11.3 gives sufficient conditions for a two-state POMDP to have a monotone optimal policy. The optimal policy is characterized by at most U − 1 threshold belief states (where U denotes the number of possible actions). One only needs to compute (estimate) these U − 1 threshold belief states in order to determine the optimal policy. This is considerably easier than solving Bellman's equation. Also real-time implementation of a controller with a monotone policy is simple; only the threshold belief states need to be stored in a lookup table. Figure 11.1 illustrates a monotone policy for a two-state POMDP with U = 3.
• Example 2: POMDP Multi-armed bandits and opportunistic scheduling: §11.4 discusses how monotone value functions can be used to solve POMDP multi-armed bandit problems efficiently. It is shown that for such problems, the optimal strategy is “opportunistic”: choose the bandit with the largest belief state in terms of MLR order.
Model and assumptions
Consider a discrete-time, infinite horizon discounted cost POMDP which was formulated in §7.6.
- Type
- Chapter
- Information
- Partially Observed Markov Decision ProcessesFrom Filtering to Controlled Sensing, pp. 241 - 254Publisher: Cambridge University PressPrint publication year: 2016