Introduction

Allow me to pose a few questions on the nature of visual recognition. Why do we bother studying it? What are the “things” we recognize? How many things is it useful to recognize? What is the nature of different recognition tasks? What have we learned so far? What are the open problems that face us? I am assuming that the reader has some familiarity with the technical aspects of vision and visual recognition. This is not a survey, and the references are meant to exemplify an idea or an approach; they are not meant to give proper credit to the many excellent people who work in the field. Also, some of the interesting technical issues are not visible from the mile-high perspective I take here and are therefore not mentioned.

What?

What is it that we recognize in images? We recognize both the component elements and the overall scene: materials and surface properties (“leather,” “wet”), objects (“frog,” “corkscrew”) and the gist of the ensemble (“kitchen,” “prairie”). We recognize things both as individuals (“Gandhi,” “my bedroom”) and as members of categories (“people,” “mountainscape”). These distinctions are important because the visual statistics of materials are different from those of objects and scenes. Also, the visual variability of individual objects is different from that of categories; different approaches may be needed to model and recognize each. In the following text, for brevity, when referring to the thing to be recognized, I will often call it “object,” although it could be a material or a scene, an individual or a category.