We derive formulas for the first- and higher-order derivatives of the steady state performance measures for changes in transition matrices of irreducible and aperiodic Markov chains. Using these formulas, we obtain a Maclaurin series for the performance measures of such Markov chains. The convergence range of the Maclaurin series can be determined. We show that the derivatives and the coefficients of the Maclaurin series can be easily estimated by analysing a single sample path of the Markov chain. Algorithms for estimating these quantities are provided. Markov chains consisting of transient states and multiple chains are also studied. The results can be easily extended to Markov processes. The derivation of the results is closely related to some fundamental concepts, such as group inverse, potentials, and realization factors in perturbation analysis. Simulation results are provided to illustrate the accuracy of the single sample path based estimation. Possible applications to engineering problems are discussed.

In this paper we study the so-called random coeffiecient autoregressive models (RCA models) and (generalized) autoregressive models with conditional heteroscedasticity (ARCH/GARCH models). Both models can be represented as random systems with complete connections. Within this framework we are led (under certain conditions) to CL-regular Markov processes and we will give conditions under which (i) asymptotic stationarity, (ii) a law of large numbers and (iii) a central limit theorem can be shown for the corresponding models.

The paper considers stability and instability properties of the Markov chain generated by the composition of an i.i.d. sequence of random transformations. The transformations are assumed to be either linear mappings or else mappings which can be well approximated near 0 by linear mappings. The main results concern the risk probabilities that the Markov chain enters or exits certain balls centered at 0. An application is given to the probability of extinction in a model from population dynamics.

A number of stationary stochastic processes are presented with properties pertinent to modelling time series from turbulence and finance. Specifically, the one-dimensional marginal distributions have log-linear tails and the autocorrelation may have two or more time scales. Discrete time models with a given marginal distribution are constructed as sums of independent autoregressions. A similar construction is made in continuous time by considering sums of Ornstein-Uhlenbeck-type processes. To prepare for this, a new property of self-decomposable distributions is presented. Also another, rather different, construction of stationary processes with generalized logistic marginal distributions as an infinite sum of Gaussian processes is proposed. In this way processes with continuous sample paths can be constructed. Multivariate versions of the various constructions are also given.

In this paper we consider a Galton-Watson process whose particles move according to a Markov chain with discrete state space. The Markov chain is assumed to be positive recurrent. We prove a law of large numbers for the empirical position distribution and also discuss the large deviation aspects of this convergence.

We consider a problem similar in many respects to a finite horizon Markov decision process, except that the reward to the individual is a strictly concave functional of the distribution of the state of the individual at final time T. Reward structures such as these are of interest to biologists studying the fitness of different strategies in a fluctuating environment. The problem fails to satisfy the usual optimality equation and cannot be solved directly by dynamic programming. We establish equations characterising the optimal final distribution and an optimal policy π*. We show that in general π* will be a Markov randomised policy (or equivalently a mixture of Markov deterministic policies) and we develop an iterative, policy improvement based algorithm which converges to π*. We also consider an infinite population version of the problem, and show that the population cannot do better using a coordinated policy than by each individual independently following the individual optimal policy π*.

In this paper we consider a Galton-Watson process in which particles move according to a positive recurrent Markov chain on a general state space. We prove a law of large numbers for the empirical position distribution and also discuss the rate of this convergence.

Let {Yn | n = 1, 2,…} be a stochastic process and M a positive real number. Define the time of ruin by T = inf{n | Yn > M} (T = +∞ if Yn ≤ M for n = 1, 2,…). Using the techniques of large deviations theory we obtain rough exponential estimates for ruin probabilities for a general class of processes. Special attention is given to the probability that ruin occurs up to a certain time point. We also generalize the concept of the safety loading and consider its importance to ruin probabilities.

We transpose the ordinary differential equation method (used for decreasing stepsize stochastic algorithms) to a dynamical system method to study dynamical systems disturbed by a noise decreasing to zero. We prove that such an algorithm does not fall into a regular trap if the noise is exciting in an unstable direction.

We study a point process model with stochastic intensities for a particular branching population of individuals of two types. Type-I individuals immigrate into the population at the times of a Poisson process. During their lives they generate type-II individuals according to a random age dependent birth rate, which themselves may multiply and die. Living type-II descendants increase the death intensity of their type-I ancestor, and conversely, the multiplication and dying intensities of type-II individuals may depend on the life situation of their type-I ancestor. We show that the probability generating function of the marginal distribution of a type-I individual's life process, conditioned on its individual infection and death risk, satisfies an initial value problem of a partial differential equation, and derive its solution. This allows for the determination of additional distributions of observable random variables as well as for describing the complete population process.

The input of a multiserver loss system is assumed to be a periodic random marked point process which has, with probability one, infinitely many construction points. It is shown that, independently of the initial distribution, there exists a unique periodic process modeling the periodic steady-state behaviour of the loss system. In addition, practical sufficient conditions for the existence of enough construction points are derived.

Modelling malaria with consistency necessitates the introduction of at least two families of interconnected processes. Even in a Markovian context the simplest fully stochastic model is intractable and is usually transformed into a hybrid model, by supposing that these two families are stochastically independent and linked only through two deterministic connections. A model closer to the fully stochastic model is presented here, where one of the two families is subordinated to the other and just a unique deterministic connection is required. For this model a threshold theorem can be proved but the threshold level is not the one obtained in a hybrid model. The difference disappears only when the human population size approaches infinity.

Let Mn denote the size of the largest amongst the first n generations of a simple branching process. It is shown for near critical processes with a finite offspring variance that the law of Mn/n, conditioned on no extinction at or before n, has a non-defective weak limit. The proof uses a conditioned functional limit theorem deriving from the Feller-Lindvall (CB) diffusion limit for branching processes descended from increasing numbers of ancestors. Subsidiary results are given about hitting time laws for CB diffusions and Bessel processes.

We generalise the work of Cramér and Leadbetter, Ylvisaker and Ito on the level crossings of a stationary Gaussian process to multivariate processes. Necessary and sufficient conditions for the existence of the expected number of crossings E(C) of the boundary of a region of ℝp by a stationary vector stochastic process are obtained, along with an explicit formula for E(C) in the Gaussian case. A rigorous proof of Belyaev's integral formula for the factorial moments of the number of exits from a region of ℝp is given for a class of processes which includes Gaussian processes having a finite second order spectral moment matrix. Applications to χ2 processes are briefly considered.

Much of the literature in reliability and survival analysis considers failure models indexed by a single scale. There are situations which require that failure be described by several scales. An example from reliability is items under warranty whose failure is recorded by time and amount of use. An example from survival analysis is the death of a mine worker which is noted by age and the duration of exposure to dust.

This paper proposes an approach for developing probabilistic models indexed by two scales: time, and usage, a quantity that is related to time. The relationship between the scales is described by an additive hazards model. The evolution of usage is described by stochastic processes like the Poisson, the gamma and the Markov additive. The paper concludes with an application involving the setting of warranties. Two features differentiate this work from related efforts: a use of specific processes for describing usage, and a use of Monte Carlo techniques for generating the models.