X-ray topography data are compared with photodiode responsivity maps to identify potential candidates for electron trapping in high purity, single crystal diamond. X-ray topography data reveal the defects that exist in the diamond material, which are dominated by non-electrically active linear dislocations. However, many diamonds also contain defects configurations (groups of threading dislocations originating from a secondary phase region or inclusion) in the bulk of the wafer which map well to regions of photoconductive gain, indicating that these inclusions are a source of electron trapping which affect the performance of diamond X-ray detectors. It was determined that photoconductive gain is only possible with the combination of an injecting contact and charge trapping in the near surface region. Typical photoconductive gain regions are 0.2 mm across; away from these near-surface inclusions the device yields the expected diode responsivity.