Hostname: page-component-76fb5796d-22dnz Total loading time: 0 Render date: 2024-04-26T19:19:08.925Z Has data issue: false hasContentIssue false
  • English
  • Français

The Vanishing Discount Approach for the Average Continuous Control of Piecewise Deterministic Markov Processes

Part of: Markov processes Stochastic systems and control

Published online by Cambridge University Press:  14 July 2016

O. L. V. Costa  and
F. Dufour
O. L. V. Costa*
Affiliation:
Escola Politécnica da Universidade de São Paulo
F. Dufour*
Affiliation:
Université Bordeaux I and INRIA Bordeaux Sud Ouest
*
∗Postal address: Departamento de Engenharia de Telecomunicações e Controle, Escola Politécnica da Universidade de São Paulo, CEP: 05508 900, São Paulo, Brazil. Email address: oswaldo@lac.usp.br
∗∗Postal address: Institut Mathématiques de Bordeaux, Université Bordeaux I, 351 cours de la Liberation, 33405 Talence Cedex, France. Email address: dufour@math.u-bordeaux1.fr
Rights & Permissions [Opens in a new window]

Abstract

Core share and HTML view are not available for this content. However, as you have access to this content, a full PDF is available via the ‘Save PDF’ action button.

This work is concerned with the existence of an optimal control strategy for the long-run average continuous control problem of piecewise-deterministic Markov processes (PDMPs). In Costa and Dufour (2008), sufficient conditions were derived to ensure the existence of an optimal control by using the vanishing discount approach. These conditions were mainly expressed in terms of the relative difference of the α-discount value functions. The main goal of this paper is to derive tractable conditions directly related to the primitive data of the PDMP to ensure the existence of an optimal control. The present work can be seen as a continuation of the results derived in Costa and Dufour (2008). Our main assumptions are written in terms of some integro-differential inequalities related to the so-called expected growth condition, and geometric convergence of the post-jump location kernel associated to the PDMP. An example based on the capacity expansion problem is presented, illustrating the possible applications of the results developed in the paper.

Keywords

Piecewise-deterministic Markov processcontinuous timelong-run average costoptimal controlintegro-differential optimality inequationvanishing discount approach

MSC classification

Secondary: 60J25: Continuous-time Markov processes on general state spaces 90C40: Markov and semi-Markov decision processes 93E20: Optimal stochastic control
Type
Research Article
Information
Journal of Applied Probability , Volume 46 , Issue 4 , December 2009 , pp. 1157 - 1183
Copyright
Copyright © Applied Probability Trust 2009 

References

[1] Costa, O. L. V. and Dufour, F. (2007). Relaxed long-run average continuous control of piecewise deterministic Markov processes. In Proc. European Control Conf., Kos, Greece, 5052–5059.Google Scholar
[2] Costa, O. L. V. and Dufour, F. (2008). Average continuous control of piecewise deterministic Markov processes. Submitted. Available at http://arxiv.org/abs/0809.0477.Google Scholar
[3] Costa, O. L. V. and Dufour, F. (2008). Stability and ergodicity of piecewise deterministic Markov processes. SIAM J. Control Optimization 47, 1053–1077.CrossRefGoogle Scholar
[4] Davis, M. H. A. (1984). Piecewise-deterministic Markov processes: a general class of non-diffusion stochastic models. J. R. Statist. Soc. B 46, 353–388.Google Scholar
[5] Davis, M. H. A. (1993). Markov Models and Optimization. Chapman and Hall, London.CrossRefGoogle Scholar
[6] Davis, M. H. A., Dempster, M. A. H., Sethi, S. P. and Vermes, D. (1987). Optimal capacity expansion under uncertainty. Adv. Appl. Prob. 19, 156–176.CrossRefGoogle Scholar
[7] Dufour, F. and Costa, O. L. V. (1999). Stability of piecewise-deterministic Markov processes. SIAM J. Control Optimization 37, 1483–1502.CrossRefGoogle Scholar
[8] Guo, X. and Rieder, U. (2006). Average optimality for continuous-time Markov decision processes in Polish spaces. Ann. Appl. Prob. 16, 730–756.CrossRefGoogle Scholar
[9] Guo, X. and Zhu, Q. (2006). Average optimality for Markov decision processes in Borel spaces: a new condition and approach. J. Appl. Prob. 43, 318–334.CrossRefGoogle Scholar
[10] Hernández-Lerma, O. and Lasserre, J. B. (1996). Discrete-Time Markov Control Processes (Appl. Math. 30). Springer, New York.CrossRefGoogle Scholar
[11] Hernández-Lerma, O. and Lasserre, J. B. (1999). Further Topics on Discrete-Time Markov Control Processes (Appl. Math. 42). Springer, New York.CrossRefGoogle Scholar
[12] Luss, H. (1982). Operations research and capacity expansion problems: a survey. Operat. Res. 30, 907–947.CrossRefGoogle Scholar
[13] Meyn, S. and Tweedie, R. (1992). Stability of Markovian processes. I. Criteria for discrete-time chains. Adv. Appl. Prob. 24, 542–574.CrossRefGoogle Scholar
[14] Meyn, S. P. and Tweedie, R. L. (1993). Markov Chains and Stochastic Stability. Springer, Berlin.CrossRefGoogle Scholar
[15] Widder, D. V. (1941). The Laplace Transform (Princeton Math. Ser. v. 6). Princeton University Press.Google Scholar
[16] Zhu, Q. (2008). Average optimality for continuous-time Markov decision processes with a policy iteration approach. J. Math. Anal. Appl. 339, 691–704.CrossRefGoogle Scholar