Book contents
- Frontmatter
- Contents
- Preface
- 1 Introduction
- Part I Stochastic Models and Bayesian Filtering
- Part II Partially Observed Markov Decision Processes: Models and Applications
- 6 Fully observed Markov decision processes
- 7 Partially observed Markov decision processes (POMDPs)
- 8 POMDPs in controlled sensing and sensor scheduling
- Part III Partially Observed Markov Decision Processes: Structural Results
- Part IV Stochastic Approximation and Reinforcement Learning
- Appendix A Short primer on stochastic simulation
- Appendix B Continuous-time HMM filters
- Appendix C Markov processes
- Appendix D Some limit theorems
- References
- Index
8 - POMDPs in controlled sensing and sensor scheduling
from Part II - Partially Observed Markov Decision Processes: Models and Applications
Published online by Cambridge University Press: 05 April 2016
- Frontmatter
- Contents
- Preface
- 1 Introduction
- Part I Stochastic Models and Bayesian Filtering
- Part II Partially Observed Markov Decision Processes: Models and Applications
- 6 Fully observed Markov decision processes
- 7 Partially observed Markov decision processes (POMDPs)
- 8 POMDPs in controlled sensing and sensor scheduling
- Part III Partially Observed Markov Decision Processes: Structural Results
- Part IV Stochastic Approximation and Reinforcement Learning
- Appendix A Short primer on stochastic simulation
- Appendix B Continuous-time HMM filters
- Appendix C Markov processes
- Appendix D Some limit theorems
- References
- Index
Summary
Introduction
Statistical signal processing deals with extracting signals from noisy measurements. Motivated by physical, communication and social constraints, this chapter addresses the deeper issue of how to dynamically control and optimize signal processing resources to extract signals from noisy measurements. In such controlled sensing problems, several types of sensors (or sensing modes) are available for measuring a given process. Associated with each sensor is a per unit-of-time measurement cost, reflecting the fact that measurements that are more costly to make typically contain more reliable information. Which sensor (or sensing mode) should the decision-maker choose at each time instant to provide the next measurement? Such problems are motivated by technological advances in the design of flexible sensors such as sophisticated multi-function radars which can be configured to operate in one of many modes for each measurement.
The controlled sensing problem considered in this chapter is also called the sensor scheduling problem, measurement control problem or active sensing problem. In the context of signal processing, the phrase sensor-adaptive signal processing is apt since the sensors adapt (reconfigure) their sensing modes in real time. Controlled sensing arises in numerous applications including adaptive radar (how much resources to devote to each target ([246]), cognitive radio (how to sense the radio spectrum for available channels [353]), and social networks (how do social sensors affect marketing and advertising strategies [185]).
This chapter discusses the formulation of controlled sensing problems with three examples. The first example considers state and measurement control of a linear Gaussian state space model. As an application, radar scheduling is discussed. The second example deals with POMDPs in controlled sensing. Unlike the formulation in Chapter 7, POMDPs arising in controlled sensing have instantaneous costs which are explicit functions of the belief. As a result, the cost in terms of the belief state is nonlinear. The third example discusses POMDPs in social learning. In social learning the observation probabilities are an explicit function of the belief. POMDPs with social learning involve the interaction of local and global decision-makers. The chapter concludes with a discussion of risk-averse MDPs and POMDPs which are of recent interest in mathematical finance.
State and sensor control for state space models
Consider state and measurement control for the state space model discussed in Chapter 2.
- Type
- Chapter
- Information
- Partially Observed Markov Decision ProcessesFrom Filtering to Controlled Sensing, pp. 179 - 200Publisher: Cambridge University PressPrint publication year: 2016