Budgets of turbulent kinetic energy (TKE) and turbulent potential energy (TPE) at different scales $\ell$ in sheared, stably stratified turbulence are analysed using a filtering approach. Competing effects in the flow are considered, along with the physical mechanisms governing the energy fluxes between scales, and the budgets are used to analyse data from direct numerical simulation at buoyancy Reynolds number $Re_b=O(100)$. The mean TKE exceeds the TPE by an order of magnitude at the large scales, with the difference reducing as $\ell$ is decreased. At larger scales, buoyancy is never observed to be positive, with buoyancy always converting TKE to TPE. As $\ell$ is decreased, the probability of locally convecting regions increases, though it remains small at scales down to the Ozmidov scale. The TKE and TPE fluxes between scales are both downscale on average, and their instantaneous values are correlated positively, but not strongly so, and this occurs due to the different physical mechanisms that govern these fluxes. Moreover, the contributions to these fluxes arising from the sub-grid fields are shown to be significant, in addition to the filtered scale contributions associated with the processes of strain self-amplification, vortex stretching and density gradient amplification. Probability density functions (PDFs) of the $Q,R$ invariants of the filtered velocity gradient are considered and show that as $\ell$ increases, the sheared-drop shape of the PDF becomes less pronounced and the PDF becomes more symmetric about $R=0$.