While environmental adversity has been shown to increase risk for psychopathology, individuals differ in their sensitivity to these effects. Both genes and childhood experiences are thought to influence sensitivity to the environment, and these factors may operate synergistically such that the effects of childhood experiences on later sensitivity are greater in individuals who are more genetically sensitive. In line with this hypothesis, several recent studies have reported a significant three-way interaction (Gene × Environment × Environment) between two candidate genes and childhood and adult environment on adult psychopathology. We aimed to replicate and extend these findings in a large, prospective multiwave longitudinal study using a polygenic score of environmental sensitivity and objectively measured childhood and adult material environmental quality. We found evidence for both Environment × Environment and Gene × Environment × Environment effects on psychological distress. Children with a poor-quality material environment were more sensitive to the negative effects of a poor environment as adults, reporting significantly higher psychological distress scores. These effects were further moderated by a polygenic score of environmental sensitivity. Genetically sensitive children were more vulnerable to adversity as adults, if they had experienced a poor childhood environment but were significantly less vulnerable if their childhood environment was positive. These findings are in line with the differential susceptibility hypothesis and suggest that a life course approach is necessary to elucidate the role of Gene × Environment in the development of mental illnesses.