Introduction

Visual classification presents many challenges, but in this chapter we will focus on the problem of generalization. This is the ability to learn about a new class of object from some images, and then to recognize instances of that class in very different images. To do this we must somehow account for variations in appearance due to changes in viewpoint, alterations in lighting, variations in the shape of different objects within the same class, and a myriad of other causes.

At the core of this generalization is the question: how best can we determine the similarity between a small set of 2-D images? Most of our visual input comes in the form of 2-D images. Our ability to classify is based on experience with a large number of images, but if we want to know how to get the most from many images, it makes sense to begin by asking how we can get the most from just a few images.

One of the most important constraints that we can use when comparing 2-D images is our knowledge that these are images of the 3-D world. We do not need to treat images as generic 2-D signals, and to compare them as such. Rather, we can base our comparisons on knowledge derived from the 3-D world. Geometry provides strong constraints on how the 2-D appearance of a 3-D object can change with viewpoint. The physics of light and the material properties of objects that reflect this light constrain how much variation there can be in the appearance of an object as the lighting varies.