Large-eddy simulations are conducted to contrast momentum and passive scalar transport over large, three-dimensional roughness elements in a turbulent channel flow. Special attention is given to the dispersive fluxes, which are shown to be a significant fraction of the total flux within the roughness sublayer. Based on pointwise quadrant analysis, the turbulent components of the transport of momentum and scalars are found to be similar in general, albeit with increasing dissimilarity for roughnesses with low frontal blockage. However, strong dissimilarity is noted between the dispersive momentum and scalar fluxes, especially below the top of the roughness elements. In general, turbulence is found to transport momentum more efficiently than scalars, while the reverse applies to the dispersive contributions. The effects of varying surface geometries, measured by the frontal density, are pronounced on turbulent fluxes and even more so on dispersive fluxes. Increasing frontal density induces a general transition in the flow from a wall bounded type to a mixing layer type. This transition results in an increase in the efficiency of turbulent momentum transport, but the reverse occurs for scalars due to reduced contributions from large-scale motions in the roughness sublayer. This study highlights the need for distinct parameterizations of the turbulent and dispersive fluxes, as well as the importance of considering the contrasts between momentum and scalar transport for flows over very rough surfaces.