The goal of lattice basis reduction is to transform a given lattice basis into a “nice” lattice basis consisting of vectors that are short and close to orthogonal. To achieve this, one needs both a suitable mathematical definition of “nice basis” and an efficient algorithm to compute a basis satisfying this definition.

Reduction of lattice bases of rank 2 in ℝ2 was given by Lagrange and Gauss. The algorithm is closely related to Euclid's algorithm and we briefly present it in Section 17.1. The main goal of this section is to present the lattice basis reduction algorithm of Lenstra, Lenstra and Lovász, known as the LLL or L3 algorithm. This is a very important algorithm for practical applications. Some basic references for the LLL algorithm are Section 14.3 of Smart [513], Section 2.6 of Cohen [127] and Chapter 17 of Trappe and Washington [547]. More detailed treatments are given in von zur Gathen and Gerhard [220], Grötschel, Lovász and Schrijver [245], Section 1.2 of Lovász [356], and Nguyen and Vallée [416]. I also highly recommend the original paper [335].

The LLL algorithm generalises the Lagrange–Gauss algorithm and exploits the Gram–Schmidt orthogonalisation. Note that the Gram–Schmidt process is not useful, in general, for lattices since the coefficients μi,j do not usually lie in ℤ and so the resulting vectors are not usually elements of the lattice. The LLL algorithm uses the Gram–Schmidt vectors to determine the quality of the lattice basis, but ensures that the linear combinations used to update the lattice vectors are all over ℤ.