Lithium Niobate doped with 4f-ions is of great interest for both fundamental science and advanced applications including high efficiency lasers with frequency conversion, elements for an all-optical telecommunication network and quantum cryptography. Our study has shown that 4f-ions create an unexpected variety of completely different non-equivalent centers in both stoichiometric and lithium deficient congruent crystals. Dominant Nd1 and Yb1 centers have C3 point symmetry (axial center), whereas all Er and most other Nd and Yb centers have the lowest C1 symmetry. Distant defects create small distortions of the crystal field at the impurity site, which cause line broadening, but do not change the C3 symmetry of observed EPR spectra. Defects in the near neighborhood can lower center symmetry from C3 to C1. We concluded that Nd1 has distant charge compensation, whereas the charge excess in low-symmetry Nd(Li) centers is compensated by near lithium or niobium vacancies. Since no axial centers were found for Er, models with cation vacancies can not describe our experimental data. The dominant axial Yb1 center has no defects in its surrounding. One axial and one low-symmetry Yb centers are self compensating Yb(Li)-Yb(Nb) pairs. Six other centers are different complexes of Yb3+ and intrinsic defects. Obtained data can be used for defect engineering for tailoring properties of photonic materials.