A novel data-driven modal analysis method, reduced-order variational mode decomposition (RVMD), is proposed, inspired by the Hilbert–Huang transform and variational mode decomposition (VMD), to resolve transient or statistically non-stationary flow dynamics. First, the form of RVMD modes (referred to as an ‘elementary low-order dynamic process’, ELD) is constructed by combining low-order representation and the idea of intrinsic mode function, which enables the computed modes to characterize the non-stationary properties of space–time fluid flows. Then, the RVMD algorithm is designed based on VMD to achieve a low-redundant adaptive extraction of ELDs in flow data, with the modes computed by solving an elaborate optimization problem. Further, a combination of RVMD and Hilbert spectral analysis leads to a modal-based time-frequency analysis framework in the Hilbert view, providing a potentially powerful tool to discover, quantify and analyse the transient and non-stationary dynamics in complex flow problems. To provide a comprehensive evaluation, the computational cost and parameter dependence of RVMD are discussed, as well as the relations between RVMD and some classic modal decomposition methods. Finally, the virtues and utility of RVMD and the modal-based time-frequency analysis framework are well demonstrated via two canonical problems: the transient cylinder wake and the planar supersonic screeching jet.