Introduction

This part of the book examines supervised learning problems in which we require a learning system to model the relationship between a pattern and a real-valued quantity. For example, in using a neural network to predict the future price of shares on the stock exchange, or to estimate the probability that a particular patient will experience problems during a surgical procedure, the predictions are represented by the real-valued output of the network.

In the pattern classification problems studied in Parts 1 and 2, the (x, y) pairs are generated by a probability distribution on the product space X × {0, 1}. In a similar way, we assume in this part of the book that the data is generated by a probability distribution P on X × ℝ. This is a generalization of the pattern classification model, and includes a number of other data-generating processes as special cases. For example, it can model a deterministic relationship between patterns and their labels, where each (x, y) pair satisfies y = f(x) for some function f. It can model a deterministic relationship with additive independent observation noise, where yi = f(xi) + ηi, and the ηi are independent and identically distributed random variables. It can also model a noisy relationship in which the observation noise variables ηi are mutually independent, but the distribution of ηi depends on the pattern xi.