Undoped and 10B doped diamond films were neutron irradiated at a moderately high fluence level (thermal neutron fluence of 1.3 × 1020 n/cm2 and a fast neutron (E> 0.1 MeV) fluence of 1.6 × 1020 n/cm2). The unirradiated, irradiated, irradiated and annealed samples were studied using Fourier Transform Infrared (FTIR) and Raman spectroscopies. A dependence of radiation induced stress on the initial boron concentration was observed. The radiation induced stress was lower for the undoped samples. Correlations between FTIR and Raman data were found. The radiation damage was removed after annealing, as measured by Raman and FTIR spectroscopy.