An axisymmetric particle sedimenting in an otherwise quiescent Newtonian fluid, in the Stokes regime, retains its initial orientation. For the special case of a spheroidal geometry, we examine analytically the effects of weak inertia and viscoelasticity in driving the particle towards an eventual steady orientation independent of initial conditions. The generalized reciprocal theorem, together with a novel vector spheroidal harmonics formalism, is used to find closed-form analytical expressions for the $O(\mathit{Re})$ inertial torque and the $O(\mathit{De})$ viscoelastic torque acting on a sedimenting spheroid of an arbitrary aspect ratio. Here, $\mathit{Re}=UL/{\it\nu}$ is the Reynolds number, with $U$ being the sedimentation velocity, $L$ the semi-major axis and ${\it\nu}$ the fluid kinematic viscosity, and is a measure of the inertial forces acting at the particle scale. The Deborah number, $\mathit{De}=({\it\lambda}U)/L$, is a dimensionless measure of the fluid viscoelasticity, with ${\it\lambda}$ being the intrinsic relaxation time of the underlying microstructure. The analysis is valid in the limit $\mathit{Re},\mathit{De}\ll 1$, and the effects of viscoelasticity are therefore modelled using the constitutive equation of a second-order fluid. The inertial torque always acts to turn the spheroid broadside-on, while the final orientation due to the viscoelastic torque depends on the ratio of the magnitude of the first ($N_{1}$) to the second normal stress difference ($N_{2}$), and the sign (tensile or compressive) of $N_{1}$. For the usual case of near-equilibrium complex fluids – a positive and dominant $N_{1}$ ($N_{1}>0$, $N_{2}<0$ and $|N_{1}/N_{2}|>1$) – both prolate and oblate spheroids adopt a longside-on orientation. The viscoelastic torque is found to be remarkably sensitive to variations in ${\it\kappa}$ in the slender-fibre limit (${\it\kappa}\gg 1$), where ${\it\kappa}=L/b$ is the aspect ratio, $b$ being the radius of the spheroid (semi-minor axis). The angular dependence of the inertial and viscoelastic torques turn out to be identical, and one may then characterize the long-time orientation of the sedimenting spheroid based solely on a critical value ($\mathit{El}_{c}$) of the elasticity number, $\mathit{El}=\mathit{De}/\mathit{Re}$. For $\mathit{El}<\mathit{El}_{c}~({>}\mathit{El}_{c})$, inertia (viscoelasticity) prevails with the spheroid settling broadside-on (longside-on). The analysis shows that $\mathit{El}_{c}\sim O[(1/\text{ln}\,{\it\kappa})]$ for ${\it\kappa}\gg 1$, and the viscoelastic torque thus dominates for a slender rigid fibre. For a slender fibre alone, we also briefly analyse the effects of elasticity on fibre orientation outside the second-order fluid regime.