The influence of finite Larmor radius correction, tensor viscosity and uniform rotation on self-gravitational and firehose instabilities is discussed in the framework of the quantum magnetohydrodynamic and Chew–Goldberger–Low (CGL) fluid models. The general dispersion relation is obtained for transverse and longitudinal modes of propagation. In both the modes of propagation the dispersion relation is further analysed with respect to the direction of the rotational axis. In the analytical discussion the axis of rotation is considered in parallel and in the perpendicular direction to the magnetic field. (i) In the transverse mode of propagation, when rotation is parallel to the direction of the magnetic field, the Jeans instability criterion is affected by the rotation, finite Larmor radius (FLR) and quantum parameter but remains unaffected due to the presence of tensor viscosity. The calculated critical Jeans masses for rotating and non-rotating dense degenerate plasma systems are
respectively. It is clear that the presence of rotation enhances the threshold mass of the considered system. (ii) In the case of longitudinal mode of propagation when rotation is parallel to the direction of the magnetic field, Alfvén and viscous self-gravitating modes are obtained. The Alfvén mode is modified by FLR corrections and rotation. The analytical as well as graphical results show that the presence of FLR and rotation play significant roles in stabilizing the growth rate of the firehose instability by suppressing the parallel anisotropic pressure. The viscous self-gravitating mode is significantly affected by tensor viscosity, anisotropic pressure and the quantum parameter while it remains free from rotation and FLR corrections. When the direction of rotation is perpendicular to the magnetic field, the rotation of the considered system coupled the Alfvén and viscous self-gravitating modes to each other. The finding of the present work is applicable to strongly magnetized dense degenerate plasma.