In this study, the effects of permeable porous walls on momentum and heat transfer in a rectangular duct were studied by means of direct numerical simulation of the turbulent conjugate heat transfer. For this purpose, airflow through a rectangular duct, partially filled with a porous medium consisting of aluminium square bars, was simulated at a bulk mean Reynolds number of 3500, where the geometry of the duct used was identical to that employed in the experimental study of Suga et al. (J. Fluid Mech., vol. 884, 2020, A7). It was found that the large-scale perturbations arising from the Kelvin–Helmholtz type of instability developed over the porous medium wall, and the turbulence intensity, particularly in the porous wall-normal component, was enhanced significantly. The secondary flow was enhanced by a factor of three compared to that in a smooth-walled square duct flow, and could be characterized by a strong upward flow along the lateral walls and downward flow in the symmetry plane. The convection by the secondary flow considerably contributed to the momentum and heat transfer in the top half of the clear flow region, whereas the enhanced turbulence over the porous wall largely affected the momentum and heat transfer just above the porous medium wall, as seen in the case of a porous-walled channel flow. It should be noted that in the porous medium region, the mean temperature at the surface of the porous medium is non-uniform, with the solid- and fluid-phase temperatures reaching the equilibrium state. This could be correctly reproduced only with the conjugate heat transfer. It was found that the mean velocity dispersion as well as the turbulent velocity fluctuation contributed significantly to the energy transfer below the porous wall, which demonstrated the importance of the dispersion heat flux for heat transfer modelling of porous medium flows. Furthermore, it was observed that the secondary flow penetrated the porous medium region resulting in large-scale mean flow currents, which enhanced heat transfer inside the porous medium region.