Smoking is reported to be associated with depression and anxiety. The present study (a) examines these associations taking comorbidity into account, (b) investigates possible confounders, (c) examines how former smokers compared to current and never-smokers in terms of anxiety and depression, and if anxiety and depression decline by time since cessation. Participants (66%) aged 20–89 years in a population-based health survey (N = 60,814) were screened employing the HADS. (a) The association with smoking was strongest in comorbid anxiety depression, followed by anxiety, and only marginal in depression. Associations were stronger in females and younger participants. (b) Variables partly accounting for the association comprised somatic symptoms, socio-demographics, alcohol problems, and low physical activity. (c) Anxiety and depression were most common in current smokers, followed by quitters, and then never-smokers. No decline in anxiety or depression was found with time since cessation. Previous studies of associations between depression and smoking might have overestimated the association when ignoring comorbid anxiety.