This chapter gives sufficient conditions on the POMDP model so that the value function in Bellman's dynamic programming equation is decreasing with respect to the monotone likelihood ratio (MLR) stochastic order. That is, π1 ≥;r π2 (in terms of MLR dominance) implies V(π1) ≤ V(π2). To prove this result, we will use the structural properties of the optimal filter established in Chapter 10.
Giving conditions for a POMDP to have a monotone value function is useful for several reasons: it serves as an essential step in establishing sufficient conditions for a stopping time POMDPs to have a monotone optimal policy – this is discussed in Chapter 12. For more general POMDPs (discussed in Chapter 14), it allows us to upper and lower bound the optimal policy by judiciously constructed myopic policies. Please see Figure 10.1 for the sequence of chapters on POMDP structural results.
After giving sufficient conditions for a monotone value function, this chapter also gives two examples of POMDPs to illustrate the usefulness of this result:
• Example 1: Monotone optimal policy for two-state POMDP: §11.3 gives sufficient conditions for a two-state POMDP to have a monotone optimal policy. The optimal policy is characterized by at most U − 1 threshold belief states (where U denotes the number of possible actions). One only needs to compute (estimate) these U − 1 threshold belief states in order to determine the optimal policy. This is considerably easier than solving Bellman's equation. Also real-time implementation of a controller with a monotone policy is simple; only the threshold belief states need to be stored in a lookup table. Figure 11.1 illustrates a monotone policy for a two-state POMDP with U = 3.
• Example 2: POMDP Multi-armed bandits and opportunistic scheduling: §11.4 discusses how monotone value functions can be used to solve POMDP multi-armed bandit problems efficiently. It is shown that for such problems, the optimal strategy is “opportunistic”: choose the bandit with the largest belief state in terms of MLR order.
Model and assumptions
Consider a discrete-time, infinite horizon discounted cost POMDP which was formulated in §7.6.