In sub-Saharan Africa, there are limited data on burden of non-alcohol substance abuse (NAS) and depressive symptoms (DS), yet potential risk factors such as alcohol and intimate partner violence (IPV) are common and NAS abuse may be the rise. The aim of this study was to measure the burden of DS and NAS abuse, and determine whether alcohol use and IPV are associated with DS and/or NAS abuse. We conducted a cross-sectional study at five sites in four countries: Nigeria (nurses), South Africa (teachers), Tanzania (teachers) and two sites in Uganda (rural and peri-urban residents). Participants were selected by simple random sampling from a sampling frame at each of the study sites. We used a standardized tool to collect data on demographics, alcohol use and NAS use, IPV and DS and calculated prevalence ratios (PR). We enrolled 1415 respondents and of these 34.6% were male. DS occurred among 383 (32.3%) and NAS use among 52 (4.3%). In the multivariable analysis, being female (PR = 1.49, p = 0.008), NAS abuse (PR = 2.06, p = 0.02) and IPV (PR = 2.93, p < 0.001) were significantly associated with DS. Older age [odds ratio (OR) = 0.31, p < 0.001)], female (OR = 0.48, p = 0.036) were protective of NAS but current smokers (OR = 2.98, p < 0.001) and those reporting IPV (OR = 2.16, p = 0.024) were more likely to use NAS. Longitudinal studies should be done to establish temporal relationships with these risk factors to provide basis for interventions.