Early-life stress (ELS) has previously been identified as a risk factor for cognitive decline, but this work has predominantly focused on clinical groups and indexed traditional cognitive domains. It, therefore, remains unclear whether ELS is related to cognitive function in healthy community-dwelling older adults, as well as whether any effects of ELS also extend to social cognition. To test each of these questions, the Childhood Trauma Questionnaire (CTQ) was administered to 484 older adults along with a comprehensive neuropsychological test battery and a well-validated test of social cognitive function. The results revealed no differences in global cognition according to overall experiences of ELS. However, a closer examination into the different ELS subscales showed that global cognition was poorer in those who had experienced physical neglect (relative to those who had not). Social cognitive function did not differ according to experiences to ELS. These results indicate that the relationship between ELS and cognition in older age may be dependent on the nature of the trauma experienced.