The well-known Rayleigh criterion is a necessary and sufficient condition for inviscid centrifugal instability of axisymmetric perturbations. We have generalized this criterion to disturbances of any azimuthal wavenumber m by means of large-axial-wavenumber WKB asymptotics. A sufficient condition for a free axisymmetric vortex with angular velocity $\Omega(r)$ to be unstable to a three-dimensional perturbation of azimuthal wavenumber m is that the real part of the growth rate \[\sigma (r) =-{\rm i}m\Omega(r)+\sqrt{-\phi(r)}\] is positive at the complex radius $r{=}r_0$ where $\partial \sigma (r)/\partial r{=}0$, i.e. \[\phi'(r_0) =-2{\rm i}m\Omega'(r_0)\sqrt{-\phi(r_0)},\] where $\phi{=}(1/r^3)\partial{r^4\Omega^2}/\partial {r}$ is the Rayleigh discriminant, provided that some a posteriori checks are satisfied. The application of this new criterion to various classes of vortex profiles shows that the growth rate of non-axisymmetric disturbances decreases as m increases until a cutoff is reached. The criterion is in excellent agreement with numerical stability analyses of the Carton & McWilliams (1989) vortices and allows one to analyse the competition between the centrifugal instability and the shear instability. The generalized criterion is also valid for a vertical vortex in a stably stratified and rotating fluid, except that ϕ becomes $\phi{=}(1/r^3)\partial{r^4(\Omega+\Omega_b)^2/\partial r$, where $\Omega_b$ is the background rotation about the vertical axis. The stratification is found to have no effect. For the Taylor–Couette flow between two coaxial cylinders, the same criterion applies except that $r_0$ is real and equal to the inner cylinder radius. In sharp contrast, the maximum growth rate of non-axisymmetric disturbances is then independent of m.