We have analysed the effect of symmetry breaking on the optical properties of semiconductor nanocrystals due to doping by charged impurities. Using doped CdSe nanocrystals as an example, we show the effects of a Coulomb center on the exciton fine-structure and optical selection rules using symmetry theory and then quantify the effect of symmetry breaking on the exciton fine structure, modelling the charged center using a multipole expansion. The model shows that the presence of a Coulomb center breaks the nanocrystal symmetry and affects its optical properties through mixing and shifting of the hole spin and parity sublevels. This symmetry breaking, particularly for positively charged centers, shortens the radiative lifetime of CdSe nanocrystals even at room temperature, in qualitative agreement with the increase in PL efficiency observed in CdSe nanocrystals doped with positive Ag charge centers [A. Sahu et.al., Nano Lett. 12, 2587, (2012)]. The effect of the charged center on the photoluminescence and the absorption spectra is shown, with and without the presence of compensating charges on the nanocrystal surface. While spectra of individual nanocrystals are expected to shift and broaden with the introduction of a charged center, configuration averaging and inhomogeneous broadening are shown to wash out these effects. The presence of compensating charges at the NC surface also serves to stabilize the band edge transition energies relative to NCs with no charge centers.