When an acoustic wave travels in a lossy medium such as a liquid, it progressively transfers its pseudo-momentum to the fluid, which results in a steady flow called acoustic streaming. This phenomenon involves a balance between sound attenuation and shear, such that the streaming flow does not vanish in the limit of vanishing viscosity. Hence, the effect of viscosity has long been ignored in acoustic streaming experiments. Here, we investigate the acoustic streaming in sessile droplets exposed to surface acoustic waves. According to experimental data, the flow structure and velocity magnitude are both strongly influenced by the fluid viscosity. We compute the sound wave propagation and hydrodynamic flow motion using a numerical method that reduces memory requirements via a spatial filtering of the acoustic streaming momentum source terms. These calculations agree qualitatively well with experiments and reveal how the acoustic field in the droplet, which is dominated by a few caustics, controls the flow pattern. We evidence that chaotic acoustic fields in droplets are dominated by a few caustics. It appears that the caustics drive the flow, which allows for qualitative prediction of the flow structure. Finally, we apply our numerical method to a broader span of fluids and frequencies. We show that the canonical case of the acoustic streaming in a hemispherical sessile droplet resting on a lithium niobate substrate only depends on two dimensionless numbers related to the surface and bulk wave attenuation. Even in such a baseline configuration, we observe and characterize four distinct flow regimes.