To study the interface characteristics between substrates and homoepitaxially grown single crystalline diamond layers, the high-pressure/high-temperature Ib diamond seeds with homoepitaxial diamond layers were annealed by low-pressure/high-temperature treatment in a hydrogen environment. The stress evolution and related impurity transformation near the interface were characterized by Raman spectroscopy, photoluminescence, and micro-infrared spectroscopy before and after annealing. It is found that the stress is the smallest in a 100 μm wide zone near the interface, accompanying with the similar change in substitutional nitrogen (Ns) concentration. After annealing at 1050 °C, 1250 °C, and 1450 °C, the local compressive stress is released gradually with temperature change. It is decreased by 1.03 GPa in maximum after annealing at 1450 °C. The concentration of nitrogen–vacancy (NV) complexes in the chemical vapor deposition (CVD) layer is dramatically reduced at 1450 °C. The value of ${{I_{{\rm{NV}}^ \hbox- } } \mathord{\left/ {\vphantom {{I_{{\rm{NV}}^ - } } {I_{{\rm{diamond}}} }}} \right. \kern-\nulldelimiterspace} {I_{{\rm{diamond}}} }}$ decreases much more than ${{I_{{\rm{NV}}^0 } } \mathord{\left/ {\vphantom {{I_{{\rm{NV}}^0 } } {I_{{\rm{diamond}}} }}} \right. \kern-\nulldelimiterspace} {I_{{\rm{diamond}}} }}$ in the CVD layer, which is due to the lower stability of NV− compared with NV0 at high temperature.