Concrete specimens of constant water-to-cement ratio and varying amounts of gravel aggregate were tested under 3-point bend. The fracture toughness of the composite and surface roughness of the paste are determined. Fracture parameters obtained from the Two ParameterFracture Model (TPFM) such as the critical stress intensity factor (KIc) and critical effective crack extension (Δao) are found to be related to the average surface roughness of the paste. The Interfacial Transition Zone (ITZ) and the bulk paste are distinguished by different values of roughness. The surface roughness of the area adjacent to the aggregate particle is evaluated as a function of the distance from the aggregate surface. In particular, the surface roughness of the paste near the aggregate is greater than that of the paste far from the aggregate, and it decreases with distance from the aggregate. The higher roughness of the paste near the aggregate indicates that aggregate particles, and more specifically the ITZ associated with them, act to toughen the paste in concrete.