Skip to main content Accessibility help
×
Home

On the Kesten–Spitzer algorithm for minimal hitting time in controlled Markov chains

  • Manish C. Bhattacharjee (a1) and Sujit K. Basu (a1)

Abstract

For a Markov chain with optional transitions, except for those to an arbitrary fixed state accessible from all others, Kesten and Spitzer proved the existence of a control policy which minimized the expected time to reach the fixed state and for constructing an optimal policy, proposed an algorithm which works in certain cases. For the algorithm to work they gave a sufficient condition which breaks down if there are countably many states and the minimal hitting time is bounded. We propose a modified algorithm which is shown to work under a weaker sufficient condition. In the bounded case with countably many states, the proposed sufficient condition is not necessary but a similar condition is. In the unbounded case as well as when the state space is finite, the proposed condition is shown to be both necessary and sufficient for the original Kesten–Spitzer algorithm to work. A new iterative algorithm which can be used in all cases is given.

Copyright

Corresponding author

Postal address: Indian Institute of Management Calcutta, P.O. Box 16757, Calcutta 700 027, India. Research partially carried out while the first author was visiting Laurentian University, Canada.

References

Hide All
Kesten, H. and Spitzer, F. (1975) Controlled Markov chains. Ann . Prob. 3, 3240.

Keywords

On the Kesten–Spitzer algorithm for minimal hitting time in controlled Markov chains

  • Manish C. Bhattacharjee (a1) and Sujit K. Basu (a1)

Metrics

Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed