We use interface-resolved simulations to study near-wall turbulence modulation by small inertial particles, much denser than the fluid, in dilute/semi-dilute conditions. We considered three bulk solid mass fractions, $\varPsi =0.34\,\%$, $3.37\,\%$ and $33.7\,\%$, with only the latter two showing turbulence modulation. The increase of the drag is strong at $\varPsi =3.37\,\%$, but mild in the densest case. Two distinct regimes of turbulence modulation emerge: for smaller mass fractions, the turbulence statistics are weakly affected and the near-wall particle accumulation increases the drag so the flow appears as a single-phase flow at slightly higher Reynolds number. Conversely, at higher mass fractions, the particles modulate the turbulent dynamics over the entire flow, and the interphase coupling becomes more complex. In this case, fluid Reynolds stresses are attenuated, but the inertial particle dynamics near the wall increases the drag via correlated velocity fluctuations, leading to an overall drag increase. Hence, we conclude that, although particles at high mass fractions reduce the fluid turbulent drag, the solid phase inertial dynamics still increases the overall drag. However, inspection of the streamwise momentum budget in the two-way coupling limit of vanishing volume fraction, but finite mass fraction, indicates that this trend could reverse at even higher particle load.