A Markov process in discrete time with a finite state space is controlled by choosing the transition probabilities from a prescribed set depending on the state occupied at any time. Given the immediate cost for each choice, it is required to minimise the expected cost over an infinite future, without discounting. Various techniques are reviewed for the case when there is a finite set of possible transition matrices and an example is given to illustrate the unpredictable behaviour of policy sequences derived by backward induction. Further examples show that the existing methods may break down when there is an infinite family of transition matrices. A new approach is suggested, based on the idea of classifying the states according to their accessibility from one another.

We shall be concerned here with limit theorems arising in the fluctuation theory of random walks, processes with stationary independent increments, recurrent events and regenerative phenomena. In Section 1 on discrete time, we consider limit theorems for ladder-points (Theorem 1) and for maxima of partial sums of random variables (Theorem 2), and discuss some related questions. In Section 2 (Theorems 3 to 6) we consider the analogues of these results in continuous time.

In the last ten years the theory of weak convergence of probability measures has been used extensively in studying the models of applied probability. By far the greatest consumer of weak convergence has been the area of queueing theory. This survey paper represents an attempt to summarize the experience in queueing theory with the hope that it will prove helpful in other areas of applied probability. The paper is organized into the following sections: queues in light traffic, queues in heavy traffic, queues with a large number of servers, continuity of queues, rates of convergence, and special queueing models.

A batch service queue is considered where each batch size and its time of service is subject to control. Costs are incurred for serving the customers and for holding them in the system. Viewing the system as a Markov decision process (i.e., dynamic program) with unbounded costs, we show that policies which minimize the expected continuously discounted cost and the expected cost per unit time over an infinite time horizon are of the form: at a review point when x customers are waiting, serve min {x, Q} customers (Q being the, possibly infinite, service capacity) if and only if x exceeds a certain optimal level M. Methods of computing M for both the discounted and average cost contexts are presented.

In this paper a storage model is described in which fluctuations in the content are governed by a sequence of independent identically distributed (i.i.d.) random inputs and i.i.d. random releases. This sequence proceeds according to an underlying semi-Markov process. Laplace transforms of the exact distribution of the content are given for the case of negative exponential distributions for both inputs and releases. Exact expressions for limiting (in time) content distributions are found. In the general case, the asymptotic behavior of the content is described for critical and supercritical limiting conditions.

The model considered here consists of n operating units which are subject to stochastic failure according to an exponential failure time distribution. Failures can be of two types. With probability p(q) a failure is of type 1(2) and is sent to repair facility 1(2) for repair. Repair facility 1(2) operates as a -server queue with exponential repair times having parameter μ1 (μ2). The number of units waiting for or undergoing repair at each of the two facilities is a continuous-parameter Markov chain with finite state space. This paper derives limit theorems for the stationary distribution of this Markov chain as n becomes large under the assumption that both and grow linearly with n. These limit theorems give very useful approximations, in terms of the six parameters characterizing the model, to a distribution that would be difficult to use in practice.

Customers arrive in groups to a single server queue with finite waiting room. Two-dimensional distributions for times and numbers of served customers between occurrences of states in the embedded Markov chain are obtained by linear algebra giving systems of equations for joint Laplace-Stieltjes transforms. For M/M/1 a simple recursion relation for the joint transform of the two variables in the title is derived and used to obtained the first and second moments.

In a multi-server queueing system in which the customers are of several different types, it is useful to define states which specify the types of customers being served as well as the total number present. Analogies with some problems in statistical mechanics are found fruitful. Certain generating functions are defined in such a way that they satisfy a system of linear equations. Solution of the associated eigenvector problem shows that the steady-state probabilities for states in which all the servers are busy can be represented by a weighted sum of geometric probabilities.