Evidence suggests that affective problems, such as depression and anxiety, increase risk for late-life dementia. However, the extent to which affective problems influence cognitive decline, even many years prior to clinical diagnosis of dementia, is not clear. The present study systematically reviews and synthesises the evidence for the association between affective problems and decline in cognitive state (i.e., decline in non-specific cognitive function) in older adults. An electronic search of PubMed, PsycInfo, Cochrane, and ScienceDirect was conducted to identify studies of the association between depression and anxiety separately and decline in cognitive state. Key inclusion criteria were prospective, longitudinal designs with a minimum follow-up period of 1 year. Data extraction and methodological quality assessment using the STROBE checklist were conducted independently by two raters. A total of 34 studies (n = 71 244) met eligibility criteria, with 32 studies measuring depression (n = 68 793), and five measuring anxiety (n = 4698). A multi-level meta-analysis revealed that depression assessed as a binary predictor (OR 1.36, 95% CI 1.05–1.76, p = 0.02) or a continuous predictor (B = −0.008, 95% CI −0.015 to −0.002, p = 0.012; OR 0.992, 95% CI 0.985–0.998) was significantly associated with decline in cognitive state. The number of anxiety studies was insufficient for meta-analysis, and they are described in a narrative review. Results of the present study improve current understanding of the temporal nature of the association between affective problems and decline in cognitive state. They also suggest that cognitive function may need to be monitored closely in individuals with affective disorders, as these individuals may be at particular risk of greater cognitive decline.