Given an image of an object, if we can recover the geometry, reflectance, and lighting, we will be able to make modifications on the recovered parameters and generate a new image of the object with a different appearance. One example is to modify the lighting condition of a face image as described in Chapters 6 and 7. Such a model-based image-editing approach is powerful, but it has limitations. One important requirement for image-editing applications is that the appearance of the background needs to be consistent with that of the foreground. Unfortunately, modeling background is much more challenging than modeling the foreground object because the background is typically cluttered and has many different types of objects that occlude each other. Without background models, it is difficult to modify its appearance in a physically correct manner. An alterative is to remove the background, but foreground–background separation remains a challenging problem.

Another limitation of model-based image editing approach is that it is difficult to model and recover geometric details. For example, we would need extremely dense polygons to represent face wrinkles. How to recover such detailed geometry from images is an open problem. Thus, it is impractical to rely on recovered geometric models to edit such geometric details.