Objectives: Low educational attainment is a risk factor for more rapid cognitive aging, but there is substantial variability in cognitive trajectories within educational groups. The aim of this study was to determine the factors that confer resilience to memory decline within educational strata. Methods: We selected 2573 initially nondemented White, African American, and Hispanic participants from the longitudinal community-based Washington Heights/Inwood Columbia Aging Project who had at least two visits. We estimated initial memory (intercept) and the rate of memory decline (slope) using up to five occasions of measurement. We classified groups according to the educational attainment groups as low (≤5 years), medium (6–11 years), and high (≥12 years). We used a multiple-group latent growth model to identify the baseline predictors of initial memory performance and rate of memory decline across groups. The model specification considered the influence of demographic, socioeconomic, biomedical, and cognitive variables on the intercept and the slope of memory trajectory. Results: Our results indicated that the three educational groups do not benefit from the same factors. When allowed to differ across groups, the predictors were related to cognitive outcomes in the highly educated group, but we found no unique predictor of cognition for the low educated older adults. Conclusions: These findings highlight that memory-protective factors may differ across older adults with distinct educational backgrounds, and the need to evaluate a broader range of potential resilience factors for older adults with few years of school.