Image inpainting is a mathematically highly ill-posed process. Once something is completely lost, we can never know for sure what has been there – except if we knew the image before it was damaged or we can travel back in time and have a look. This means that image inpainting does not have a unique solution in general. Since we are intelligent human beings, we can narrow done our search by using roughly two main strategies: our natural visual perception and experience and our historical knowledge (almost like travelling back in time) about the image and its hole. Both these strategies are crucial for modelling of the desired inpainting process and will determine the inpainting methods that we will consider in subsequent chapters.

Historical knowledge of the image could be many things and depend very much on the kind of application. In art restoration, known characteristics and techniques used by the painter, intact illustrations of a similar scene by different artists (such is the case in many religious paintings from the eighteenth century), and knowledge about general painting techniques, colour mixtures and materials used at the time the painting was created have been used. In video inpainting, knowledge from earlier frames could be used to extrapolate in time into later frames where damage appears. In medical imaging applications, anatomical knowledge of a template human body is used, and similarly, there are many other application in which geometrical knowledge about the shapes of objects one is looking for can be used (compare the application on the inpainting of satellite images of roads in Section 9.2). This part of the interpolation process depends on the context of the interpolation.

Visual perception, however, gives a context-free continuation of an incomplete scene. It constitutes our – either natural or learned – ability to automatically interpolate broken or occluded structures. This automatic continuation follows certain laws.