The subgrid-scale (SGS) stress in the atmospheric surface layer is studied using measurement data. Field measurements employing a novel array technique were conducted to provide data for obtaining resolvable- and subgrid-scale variables. We analyse the conditional SGS stress and the conditional stress production rate conditional on the resolvable-scale velocity, which must be reproduced by the SGS model for large-eddy simulation (LES) to predict correctly the one-point resolvable-scale velocity statistics. The results show that both buoyancy and shear play important roles in the physics of the SGS stress. Strong buoyancy and vertical shear associated with updrafts and positive streamwise velocity fluctuations cause conditional forward energy transfer and strong anisotropy in the conditional SGS stress. Downward returning flows associated with large convective eddies result in conditional energy backscatter and much less anisotropic SGS stress. Predictions of the conditional SGS stress and the conditional stress production rate predicted using several SGS models are compared with measurements. None of those models are able to predict correctly the trends of both statistics. The Smagorinsky and one nonlinear model under-predict the anisotropy and the variations of the anisotropy, whereas the other nonlinear model and the mixed model over-predict both. The deficiencies of the SGS models that cause inaccurate LES statistics, such as the over-prediction of the mean shear and under-prediction of the vertical velocity skewness, are identified. The present study shows that analyses of conditional SGS stress and conditional SGS stress production provide a systematic approach for studying SGS physics and evaluating SGS models and can potentially be used to target specific aspects of LES that are important for a given application.