We report an experimental (infrared reflectance spectroscopy) and theoretical study of the polar optical phonons in hexagonal ternary nitride compounds: AlNm/GaNn (n=2-8, m=4, 8) superlattices (SL) and spontaneously ordered AlxGa1−xN (x=0.08–0.55) alloys. In infrared (IR) reflectivity spectra we revealed two modes having strong LO-TO splitting (20–150 cm−1), and several modes, having a small (1–3 cm−1) LO-TO splitting. All modes have a very high damping parameter ≥20 cm−1. The unusual observation is the negative value of the oscillator strength for the weak IR mode at ∼690 cm−1, suggesting possible lattice instability, consistent with high damping observed. We found from lattice dynamical calculations that weak IR active modes correspond to modes localized at GaN-AlN interfaces. Our analysis has shown that an anomalous mode is induced by the disorder effects and arises due to strong overlapping of the LO-TO phonon branches of the bulk GaN and AlN. In SL samples the anomalous mode corresponds to phonons localized on interface inhomogenities.