In the conventional X-ray reflectivity (XRR) analysis, the reflectivity is calculated based on the Parratt formalism, incorporating the effect of the interface roughness according to Nevot and Croce. However, the results of calculations of the XRR have shown strange outcomes, where interference effects increase at a rough surface because of a lack of consideration of diffuse scattering within the Parratt formalism. Therefore, we have developed a new improved formalism in which the effects of the surface and interface roughness are included correctly. In this study, for deriving a more accurate formalism of XRR, we tried to compare the measurements of surface roughness of the same sample by atomic force microscopy (AFM) and XRR. It is found that the AFM result could not be completely reproduced even with the improved XRR formalism. By careful study of the AFM results, we determined the need for an additional effective roughness term within the XRR simulation that depends on the angle of incidence of the beam.