A method to remove the effects of elastic and thermal diffuse scattering (TDS) of the incident electron probe from electron energy-loss and energy-dispersive X-ray spectroscopy data for atomically resolved spectrum images of single crystals of known thickness is presented. By calculating the distribution of the probe within a specimen of known structure, it is possible to deconvolve the channeling of the probe and TDS from experimental data by reformulating the inelastic cross-section as an inverse problem. In electron energy-loss spectroscopy this allows valid comparisons with first principles fine-structure calculations to be made. In energy-dispersive X-ray spectroscopy, direct compositional analyses such as ζ-factor and Cliff–Lorimer k-factor analysis can be performed without the complications of channeling and TDS. We explore in detail how this method can be incorporated into existing multislice programs, and demonstrate practical considerations in implementing this method using a simulated test specimen. We show the importance of taking into account the scattering of the probe in k-factor analysis in a zone axis orientation. The applicability and limitations of the method are discussed.