A three-dimensional linear stability analysis has been carried out to understand the origin of vortices and related density patterns in bounded uniform-shear flow of granular materials, using a kinetic-theory constitutive model. This flow is found to be unstable to pure spanwise stationary perturbations ($k_z\,{\neq}\, 0$, $k_x\,{=}\,0$ and $\partial/\partial y(.)\,{=}\,0$, where $k_i$ is the wavenumber for the $i$th direction) if the solid fraction is below some critical value $\nu\,{<}\, \nu_{3D}$. The growth rates of these spanwise instabilities are an order of magnitude larger than those of the two-dimensional ($k_z\,{=}\,0$) streamwise-independent ($k_x\,{=}\,0$) instabilities that occur if the solid fraction is above some critical value $\nu\,{>}\,\nu_{2D}$ (${>}\nu_{3D}$). The spanwise instabilities give birth to new three-dimensional travelling wave instabilities at non-zero values of the streamwise wavenumber ($k_x\,{\neq}\, 0$) in dilute flows ($\nu \,{<}\, \nu_{3D}$). For moderate-to-large densities with $k_x\,{\neq}\, 0$, there are additional three-dimensional instability modes in the form of both stationary and travelling waves, whose origin is tied to the corresponding two-dimensional instabilities.

While the two-dimensional streamwise-independent modes lead to the formation of stationary streamwise vortices for moderately dense flows ($\nu\,{>}\,\nu_{2D}$), the pure spanwise modes are responsible for the origin of such vortices in the dilute limit ($\nu\,{<}\,\nu_{3D}$). For more general kinds of perturbations ($k_x\,{\neq}\, 0$ and $k_z\,{\neq}\, 0$), ‘modulated’ streamwise vortices are born which could be either stationary or travelling depending on control parameters. The rolling motion of vortices will lead to a major redistribution of the streamwise velocity and hence such vortices can act as potential progenitors for the mixing of particles. The effect of non-zero wall slip has been investigated, and it is shown that some dilute-flow instabilities can disappear with the inclusion of the wall slip. Even though the streamwise granular vortices have similarities to the well-known stationary Taylor–Couette vortices (which are ‘hydrodynamic’ in origin), their origin is, however, tied to ‘constitutive’ instabilities, and hence they belong to a different class.