The instability of an anticyclonic solid-body rotating eddy embedded on a quiescent environment is studied, for all possible values of the parameters of the unperturbed state, i.e. the vortex's relative thickness and rotation rate. The Coriolis force is fundamental for the existence of the eddy (because the pressure force has a centrifugal direction) and therefore this analysis pertains to the study of mesoscale vortices in the ocean or the atmosphere, as well as those in other planets.

These eddies are known to be stable when the ‘second’ layer is assumed imperturbable (infinitely deep); however, here these vortices are found to be unstable in the more realistic case of an active environment layer, which may be arbitrarily thick.

Three basic types of instability are found, classified according to the dynamic structure of the growing perturbation field, in both layers: baroclinic instability (Rossby-like in both layers), Sakai instability (Poincaré-like in the vortex layer and Rossby-like in the environment), and Kelvin–Helmholtz instability (Poincaré-like in both layers). In addition, there is a hybrid instability, which goes continuously from the baroclinic to the Sakai types, as the rotation rate is increased.

The problem is constrained by the conservation of pseudoenergy and angular pseudomomentum, which are quadratic (to lowest order) in the perturbation. The requirement that both integrals of motion vanish for a growing disturbance, determines the structure of the latter in both layers. Furthermore, that constraint restricts the region, in parameter space, where each type of instability is present.