Two free waves propagating in a parallel shear flow generate a critical layer when their nonlinear interaction induces a perturbation whose phase velocity matches the basic-state velocity somewhere in the flow domain. The condition necessary for this to occur may be interpreted as a resonance condition for a triad formed by the two waves and a (singular) mode of the continuous spectrum associated with the shear. The formation of the critical layer is investigated in the case of freely propagating Rossby waves in a two-dimensional inviscid flow in a β-channel.

A weakly nonlinear analysis based on a normal-mode expansion in terms of Rossby waves and modes of the continuous spectrum is developed; it leads to a system of amplitude equations describing the evolution of the two Rossby waves and of the modes of the continuous spectrum excited during the interaction. The assumption of weak nonlinearity is not however self-consistent: it breaks down because nonlinearity always becomes strong within the critical layer, however small the initial amplitudes of the Rossby waves. This demonstrates the relevance of nonlinear critical layers to monotonic, stable, unforced shear flows which sustain wave propagation.

A nonlinear critical-layer theory is developed that is analogous to the well-known theory for forced critical layers. Differences arise because of the presence of the Rossby waves: the vorticity in the critical layer is advected in the cross-stream direction by the oscillatory velocity field due to the Rossby waves. An equation is derived which governs the modification of the Rossby waves that results from their interaction; it indicates that the two Rossby waves are undisturbed at leading order. An analogue of the Stewartson–Warn–Warn analytical solution is also considered.