Although our interest in the human visual system is mainly in the role it plays as the observer in a color imaging system, we cannot completely treat it as a big black box, because the system is nonlinear and far too complicated to be characterized as such. There have been many attempts to apply linear system theory to the human visual system and characterize it with a system transfer function. That approach may serve some purposes in a few applications, but it is inadequate for most color imaging applications. Another possible approach is to treat it as many medium-sized black boxes, one for each special aspect in image perception. This is a more practical approach and it has been used very well for many applications. For example, we can measure the human contrast sensitivity as a function of luminance, field size, viewing distance, noise level, and chromatic contents, and the results can be used to design the DCT quantization tables for color image compression. However, the medium-sized black box approach does not give us much insight or guidance when our problem becomes more complicated or when we have a new problem. The size of the black boxes has to be reduced. An extreme limit is when each box corresponds to a single neuron in the visual pathway. Even then, some details inside the neuron may still be important to know. In general, how much detail we need to know is highly application-dependent, but the more we know the better we are equipped to deal with image perception related questions.