Video signals represent sequences of images or frames which can be transmitted with a rate from 10 up to 60 frames/s (fps) providing the illusion of motion in the displayed signal. They can be represented in different formats which differ in frame size and number of fps. For example, the video format QCIF intended for video conferencing and mobile applications uses frames of size 176 × 144 pixels transmitted at rate 10 fps. The High Definition Digital Television (HDTV) standard uses frames of much larger size, 1280 × 720 pixels, transmitted at rate 60 fps. The frames can be represented in RGB or YUV formats with 24 or fewer (due to decimation of U and V components) number of bits per pixel. If the frames are in RGB format, then the first processing step in any video coder is the RGB to YUV transform.

Unlike images, video sequences contain the so-called temporal redundancy which arises from repeating objects in consecutive frames of a video sequence. The simplest method of temporal prediction can use the previous frame as a prediction of the current frame. However, the residual formed by subtracting the prediction from the current frame typically has large energy, the reason being the object movements between the current and the previous frames. Better predictions can be performed by compensating the motion between two frames.