This chapter introduces the concepts of vector spaces and linear mappings between such spaces. Vector spaces are akin to geometry and consist of vectors that may be added together and multiplied by scalars. We present the necessary foundations for understanding these abstract concepts and also for further study in numerous applications of signal and image processing. The remainder of this chapter is organized as follows. Section 3.1 provides a formal introduction to vector spaces and their important properties, along with many illustrative examples. In Section 3.2, we study linear operators that map the vectors in one vector space to those in another, while preserving the operations that give structure to these vector spaces. We discuss when two vector spaces are essentially the same or isomorphic, and explore the properties of two special subspaces, the kernel and range, associated with a linear operator. We show that the effect of a linear operator is equivalent to multiplication by the associated matrix. Then we discuss the eigenanalysis of linear operators and their associated matrices. Matrices are used extensively in almost all numerical mathematical computations, and can help solve complicated problems involving linear operators by simply performing matrix multiplications. We also introduce linear functionals that map a vector space to a field of scalars. Section 3.3 introduces inner product spaces, orthonormal sets and bases, and normed vector spaces. We present several types of linear operators that are especially important in signal and image processing, and then we examine some elementary properties of these operators and their associated matrices. In Section 3.4, we briefly define the concept of a topological vector space. The generalized eigenvalue problem is discussed in Section 3.5. In Section 3.6, the singular value decomposition of a matrix is described, followed by an application to image compression. Section 3.7 examines in detail the principal component analysis technique, along with an application to outlier detection in multivariate data.

Vector space theory

Generally speaking, a vector may geometrically be defined as an object that has both a magnitude and direction.