Attention biases to threat are considered part of the etiology of anxiety disorders. Attention bias variability (ABV) quantifies intraindividual fluctuations in attention biases and may better capture the relation between attention biases and psychopathology risk versus mean levels of attention bias. ABV to threat has been associated with attentional control and emotion regulation, which may impact how caregivers interact with their child. In a relatively diverse sample of infants (50% White, 50.7% female), we asked how caregiver ABV to threat related to trajectories of infant negative affect across the first 2 years of life. Families were part of a multi-site longitudinal study, and data were collected from 4 to 24 months of age. Multilevel modeling examined the effect of average caregiver attention biases on changes in negative affect. We found a significant interaction between infant age and caregiver ABV to threat. Probing this interaction revealed that infants of caregivers with high ABV showed decreases in negative affect over time, while infants of caregivers with low-to-average ABV showed potentiated increases in negative affect. We discuss how both high and extreme patterns of ABV may relate to deviations in developmental trajectories.