Following the stability analysis method in classic fluid dynamics, a linear stability equation (LSE) suitable for rarefied flows is derived based on the Bhatnagar–Gross–Krook (BGK) equation. The global method and singular value decomposition method are used for modal and non-modal analysis, respectively. This approach is validated by results obtained from Navier–Stokes (NS) equations. The modal analysis shows that LSEs based on NS equations (NS-LSEs) begin to fail when the Knudsen number ($Kn$) increases past $\sim$0.01, regardless of whether a slip model is used. When $Kn\geq 0.01$, the growth rate of the least stable mode is generally underestimated by the NS-LSEs. Under a fixed wavenumber, the pattern (travelling or standing wave) of the least stable mode changes with $Kn$; when the mode presents the same pattern, the growth rate decreases almost linearly with increasing $Kn$; otherwise, rarefaction effects may not stabilize the flow. The characteristic lengths of the different modes are different, and the single-scale classic stability analysis method cannot predict multiple modes accurately, even when combined with a slip model and even for continuum flow. However, non-modal analysis shows that this error does not affect the transient growth because modes with small growth rates offer little contribution to the transient growth. In rarefied flow, as long as the Mach number ($Ma$) is large enough, transient growth will occur in some wavenumber ranges. The rarefaction effect plays a stabilizing role in transient growth. The NS-LSEs-based method always overestimates the maximum transient growth.