Background: The ɛ4-allele of apolipoprotein E (APOE-ɛ4) increases the risk not only for Alzheimer’s disease (AD), but also for Parkinson’s disease dementia and dementia with Lewy bodies (collectively, Lewy body dementia [LBD]). Hippocampal volume is an important neuroimaging biomarker for AD and LBD, although its association with APOE-ɛ4 is inconsistently reported. We investigated the association of APOE-ε4 with hippocampal atrophy quantified using magnetic resonance imaging in AD and LBD. Methods: Electronic databases (PubMed, Embase, PsycINFO, Scopus, Web of Science) were systematically searched for studies published up until December 31st, 2020. Results: Thirty-nine studies (25 cross-sectional, 14 longitudinal) were included. We observed that: (1) APOE-ε4 was associated with greater rate of hippocampal atrophy in AD and those who progressed from mild cognitive impairment to AD, (2) APOE-ε4 carriers showed greater involvement of cornu ammonis-1 hippocampal subfield versus non-carriers in AD, (3) APOE-ɛ4 may influence hippocampal atrophy in dementia with Lewy bodies, although longitudinal investigations are required, and (4) APOE-ε4 associated with earlier rather than very late expression of mediotemporal degeneration and memory-related neurocognitive impairment. Conclusions: The role of APOE-ɛ4 in modulating hippocampal phenotypes may be further clarified through more homogenous, well-powered, pathology-proven studies. Understanding the underlying mechanisms will facilitate development of prevention strategies targeting APOE-ɛ4.