Parallel processing takes several flavors, depending on the unit of parallelism and the number of processing units. Early on in the development of computer systems, we saw the emergence of multiprogramming (Section 2.3), whereby several (portions of) programs share main memory. When the program currently executing requires some I/O processing, it relinquishes the use of the CPU via a context switch, and another program takes ownership of the CPU. Parallel processing occurs between the program using the CPU and the programs (there may be more than one) executing some I/O-related task. Here, the unit of parallel processing is a program, or process, and the parallelism is at the program level. An efficient implementation of multiprogramming requires the use of an operating system with a virtual memory management component. At the other extreme of the spectrum of granularity, we have the exploitation of instruction-level parallelism. Several instructions of the same program are executing simultaneously. Pipelining (Section 2.1) is the simplest form of the concurrent execution of instructions. Superscalar and EPIC processors (Chapter 3) extend this notion by having several instructions occupying the same stages of the pipeline at the same time. Of course, extra resources such as multiple functional units must be present for this concurrency to happen.
In the previous chapter, we gave a strict definition of multiprocessing, namely, the processing by several processors of portions of the same program.