When a suspension of the bacterium Bacillus subtilis is placed in a chamber with its upper surface open to the atmosphere, complex bioconvection patterns form. These arise because the cells (a) are denser than water, and (b) swim upwards on average so that the density of an initially uniform suspension becomes greater at the top than at the bottom. When the vertical density gradient becomes large enough an overturning instability occurs which evolves ultimately into the observed patterns. The cells swim upwards because they are oxytactic, i.e. they swim up gradients of oxygen, and they consume oxygen. These properties are incorporated in conservation equations for the cell and oxygen concentrations, which, for the pre-instability stage of the pattern formation process, have been solved in a previous paper (Hillesdon, Pedley & Kessler 1995). In this paper we carry out a linear instability analysis of the steady-state cell and oxygen concentration distributions. There are intrinsic differences between the shallow-and deep-chamber cell concentration distributions, with the consequence that the instability is non-oscillatory in shallow chambers, but must be oscillatory in deep chambers whenever the critical wavenumber is non-zero. We investigate how the critical Rayleigh number for the suspension varies with the three independent parameters of the problem and discuss the most appropriate definition of the Rayleigh number. Several qualitative aspects of the solution of the linear instability problem agree with experimental observation.