The dynamic interplay between surface and subsurface flow in the presence of a permeable boundary was investigated using low and high frame-rate particle-image velocimetry measurements in a refractive-index-matching flow environment. Two idealized permeable wall models were considered. Both models contained five layers of cubically packed spheres, but one exhibited a smooth interface with the flow, while the other embodied a hemispherical surface topography. The relationship between the large-scale turbulent motions overlying the permeable walls and the small-scale turbulence just above, and within, the walls was explored using instantaneous and statistical analyses. Although previous studies have indirectly identified the potential existence of amplitude modulation in permeable-wall turbulence (a phenomenon identified in impermeable-wall turbulence whereby the outer large scales modulate the intensity of the near-wall, small-scale turbulence), the present effort provides direct evidence of its existence in flow over both permeable walls considered. The spatio-temporal signatures of amplitude modulation were also characterized using conditional averaging based on zero-crossing events. This analysis highlights the connection between large-scale regions of high/low streamwise momentum in the surface flow, downwelling/upwelling across the permeable interface and enhancement/suppression of small-scale turbulence, respectively, just above and within the permeable walls. The presence of bed roughness is found to intensify the strength and penetration of flow into the permeable bed modulated by large-scale structures in the surface flow, and linked to possible roughness-formed channelling effects and shedding of smaller-scale flow structures from the roughness elements.