We consider an investment problem where observing and trading are only possible at random times. In addition, we introduce drawdown constraints which require that the investor's wealth does not fall under a prior fixed percentage of its running maximum. The financial market consists of a riskless bond and a stock which is driven by a Lévy process. Moreover, a general utility function is assumed. In this setting we solve the investment problem using a related limsup Markov decision process. We show that the value function can be characterized as the unique fixed point of the Bellman equation and verify the existence of an optimal stationary policy. Under some mild assumptions the value function can be approximated by the value function of a contracting Markov decision process. We are able to use Howard's policy improvement algorithm for computing the value function as well as an optimal policy. These results are illustrated in a numerical example.

A stochastic scheduling model with linear waiting costs and unknown routing probabilities is considered. Using a Bayesian approach and methods of Bayesian dynamic programming, we investigate the finite-horizon stochastic scheduling problem with incomplete information. In particular, we study an equivalent nonstationary bandit model and show the monotonicity of the total expected reward and of the Gittins index. We derive the monotonicity and well-known structural properties of the (greatest) maximizers, the so-called stay-on-a-winnerproperty and the stopping-property. The monotonicity results are based on a special partial ordering on .

We consider a general discrete-time stochastic recursive model that is influenced by an external Markov chain. Our aim is to investigate the effect that the transition matrix of the external process has on the system states of the model. To answer this question, we use new stochastic ordering concepts. Especially interesting are the results for infinite-stage Markov-modulated models. We illustrate our main results by three applications: an inventory model, a consumption model, and a queueing model for a time division multiplexing system.

We study portfolio optimization problems in which the drift rate of the stock is Markov modulated and the driving factors cannot be observed by the investor. Using results from filter theory, we reduce this problem to one with complete observation. In the cases of logarithmic and power utility, we solve the problem explicitly with the help of stochastic control methods. It turns out that the value function is a classical solution of the corresponding Hamilton-Jacobi-Bellman equation. As a special case, we investigate the so-called Bayesian case, i.e. where the drift rate is unknown but does not change over time. In this case, we prove a number of interesting properties of the optimal portfolio strategy. In particular, using the likelihood-ratio ordering, we can compare the optimal investment in the case of observable drift rate to that in the case of unobservable drift rate. Thus, we also obtain the sign of the drift risk.

A martingale approach to a dynamic program with general state and action spaces is taken. Several necessary and sufficient conditions are given for a policy to be optimal. The results comprehend and modify different criteria of optimality given for dynamic programming problems. Finally, two applications are stated.

We consider a non-stationary Bayesian dynamic decision model with general state, action and parameter spaces. It is shown that this model can be reduced to a non-Markovian (resp. Markovian) decision model with completely known transition probabilities. Under rather weak convergence assumptions on the expected total rewards some general results are presented concerning the restriction on deterministic generalized Markov policies, the criteria of optimality and the existence of Bayes policies. These facts are based on the above transformations and on results of Hindererand Schäl.