In this chapter, we describe techniques that model both shape and appearance. Techniques in this category have their origins in shape-from-shading. The basic idea of shape-from-shading is to use a generative imaging model (e.g., Lambertian reflectance) to fit an image and solve for the shape and illumination parameters. It is an ill-conditioned problem due to large number of unknowns. To make the problem more constrained, people usually make certain assumptions such as a constant albedo, a single directional light source, or surface smoothness. In 1999, Blanz and Vetter [19] introduced a linear space representation called face morphable model. Even though their work may not be directly motivated from shape-from-shading problem, it is interesting to look at the face morphable model from the perspective of shape-from-shading. We can think of the linear space representations of the face geometry and albedo as a regularization constraints on the shape-from-shading framework. With this representation, the number of unknowns is significantly reduced. As a result, it is possible to relax the other assumptions. For example, one does not have to assume a constant albedo anymore.

We will first give an introduction to shape-from-shading and then describe the face morphable model framework. In Sections 7.3.1 and 7.5, we describe the integration of face morphable model with spherical harmonics representation to handle general and even harsh lighting conditions.