Introduction

The intuitive idea of a model is like a smooth, low-order polynomial curve fitted to a cloud of points on a plane. It, then, represents the proverbial ‘law’ which the data are supposed to follow except for the inevitable ‘noise’. We see already in this simple example the basic difficulty in modelling problems: it is not at all clear how to formalize the idea of a ‘smooth’ curve, or what is considered to be a ‘low’-order polynomial. Needless to add that by assuming some probability distribution for the noise and then calculating the conditional mean contributes nothing towards the solution of the problem. And the same goes for any Bayesian approach, where not only the data but even the parameters must be assumed to be samples from a distribution.

One can argue that a conceptually ideal model of an observed data string is the shortest program that generates the string in a universal programming language; for a thorough account of the algorithmic theory of information or complexity, see Li and Vitányi (1993). Indeed, if the data string has any regular features, i.e. if there is a ‘law’ that holds even approximately, the shortest program must take advantage of it. In this spirit, then, the length of the shortest program, the Kolmogorov complexity, which really is due to Solomonoff, serves as a measure of the string's complexity.