This study examines the structure of shear stress and turbulent kinetic energy (TKE) flux across the roughness layer of a uniform, fully rough gravel-bed channel flow (ks+ ≫ 100, δ/k = 20) using high-resolution acoustic Doppler velocity profiler measurements. The studied gravel-bed roughness layer exhibits a complex random multi-scale roughness structure in strong contrast with conceptualized k- or d-type roughness in standard rough-wall flows. Within the roughness layer, strong spatial variability of all time-averaged flow quantities are observed affecting up to 40% of the boundary layer height. This variability is attributed to the presence of bed zones with emanating bed protuberances (or gravel clusters) acting as local flow obstacles and bed zones of more homogenous roughness of densely packed gravel elements. Considering the strong spatial mean flow variability across the roughness layer, a spatio-temporal averaging procedure, called double averaging (DA), has been applied to the analysed flow quantities. Three aspects have been addressed: (a) the DA shear stress and DA TKE flux in specific bed zones associated with three classes of velocity profiles as previously proposed in Mignot, Barthélemy & Hurther (J. Fluid Mech., vol. 618, 2009, p. 279), (b) the global and per class DA conditional statistics of shear stress and associated TKE flux and (c) the contribution of large-scale coherent shear stress structures (LC3S) to the TKE flux across the roughness layer. The mean Reynolds and dispersive shear structure show good agreement between the protuberance bed zones associated with the S-shape/accelerated classes and recent results obtained in standard k-type rough-wall flows (Djenidi et al., Exp. Fluids, vol. 44, 2008, p. 37; Pokrajac, McEwan & Nikora, Exp. Fluids, vol. 45, 2008, p. 73). These gravel-bed protuberances act as local flow obstacles inducing a strong turbulent activity in their wake regions. The conditional statistics show that the Reynolds stress contribution is fairly well distributed between sweep and ejection events, with threshold values ranging from H = 0 to H = 8. However, the TKE flux across the roughness layer primarily results from the residual shear stress between ejection and sweep of very high magnitude (H = 10–20) and of small turbulent scale. Although LC3S are seen to penetrated the interfacial roughness layer, their TKE flux contribution is found to be negligible compared to the very energetic small-scale sweep events. These sweeps are dominantly produced in the bed zones of local gravel protuberances where the velocity profiles are inflexional of S-shape type and the mean flow properties are of mixing-layer flow type as previously shown in Mignot et al. (2009).