Oceanic large- and meso-scale flows are nearly balanced in forces between Earth’s rotation and density stratification effects (i.e. geostrophic, hydrostatic balance associated with small Rossby and Froude numbers). In this regime advective cross-scale interactions mostly drive energy toward larger scales (i.e. inverse cascade). However, viscous energy dissipation occurs at small scales. So how does the energy reservoir at larger scales leak toward small-scale dissipation to arrive at climate equilibrium? Here we solve the linear instability problem of a balanced flow in a rotating and continuously stratified fluid far away from any boundaries (i.e. an interior jet). The basic flow is unstable not only to geostrophic baroclinic and barotropic instabilities, but also to ageostrophic instabilities, leading to the growth of small-scale motions that we hypothesize are less constrained by geostrophic cascade behaviours in a nonlinear regime and thus could escape from the inverse energy cascade. This instability is investigated in the parameter regime of moderate Rossby and Froude numbers, below the well-known regimes of gravitational, centrifugal, and Kelvin–Helmholtz instability. The ageostrophic instability modes arise with increasing Rossby number through a near-degeneracy of two unstable modes with coincident phase speeds. The near-degeneracy occurs in the neighbourhood of an identified criterion for the non-integrability of the ‘isentropic balance equations’ (namely with the absolute vertical vorticity and the horizontal strain rate associated with the basic flow), beyond which development of an unbalanced component of the flow is expected. These modes extract energy from the basic state with large vertical Reynolds stress work (unlike geostrophic instabilities) and act locally to modify the basic flow by reducing the isopycnal Ertel potential vorticity gradient near both its zero surface and its critical surface (phase speed equal to basic flow speed).