A new type of polyelectrolyte–Al2O3/SiO2 composite nanoparticle with excellent dispersibility and superior polishing performance was successfully fabricated using a facile method. Silica acted as a bifunctional molecule by attaching to alumina via covalent bond and adsorbing polyelectrolytes by electrostatic interaction. The material removal rate of the polyelectrolyte–Al2O3/SiO2 abrasive was 30% higher than that of the pure Al2O3 abrasive. In addition, the sapphire surface was much smoother. The material removal mechanism was investigated during CMP using the microcontact and wear model. The enhanced removal rate was mainly attributed to the well-dispersed particles, which can accelerate mechanical removal process. The remarkably smooth surface was due to the decrease in penetration depth of the abrasive into the wafer. The results of this study provided a feasible strategy to satisfy the high efficiency and damage-free polishing requirements for sapphire planarization.