A prominent feature of anxiety in late life is concerns regarding physical health. Anxiety symptoms among older adults have been connected with various psychological outcomes, including social isolation and loneliness. During the coronavirus disease 2019 (COVID-19) pandemic, many societies have demonstrated increased ageist attitudes, encouraging older adults to distance themselves from society. Accordingly, the current study examined the moderating role of COVID-19-related ageism in the connection between COVID-19 health worries and anxiety symptoms among older adults. Data were collected from 243 older adults (age range 60–92; M = 69.75, SD = 6.69), who completed scales assessing COVID-19-related health worries and ageism, as well as anxiety symptoms. The results demonstrated that both health worries and ageism were positively associated with anxiety symptoms. Moreover, the connection between health worries and anxiety symptoms was more pronounced among older adults with high ageism levels. The study highlights the vulnerability of older adults in general, and ageist older adults in particular, to the negative consequences of COVID-19-related health worries, and emphasizes the role of the increased ageist stance of society during the pandemic in this regard.