Graphene is a promising material for electronic and spintronic applications due to its high carrier mobility and low intrinsic spin-orbit interaction. However, extrinsic effects may easily dominate intrinsic scattering mechanisms. The scattering mechanisms investigated here are associated non-magnetic, charged impurities in the substrate (e.g. SiO2) beneath the graphene layer. Such impurities cause an electric field that extends through the graphene and has a non-vanishing perpendicular component. Consequently, the impurity, in addition to the conventional elastic, spin-conserving scattering can give rise to spin-flip processes. The latter is a consequence of a spatially varying Rashba spin-orbit interaction caused by the electric field of the impurity in the substrate. Scattering cross-sections are calculated and, for assumed impurity distributions, relaxation times are estimated.