Neuroticism is associated with increased stress reactivity. In autism spectrum disorders (ASD), emotional stress reactivity is increased and there is some evidence for an increased negative affect (NA) when with less familiar people. The aim of this study was to compare adults with ASD and controls on levels of neuroticism and on interactions between neuroticism and appraised stress or social context in models of NA. This is a cross-sectional observational study comprising a group of 50 adults with ASD and 51 controls. Experience sampling method (ESM) reports were collected for 10 days to measure daily life stress, mood, and social context. Multilevel regression analyses revealed significantly higher neuroticism levels in ASD than in controls. Adults with ASD who scored high on neuroticism showed a significantly stronger association between activity/social stress and NA (i.e., higher stress reactivity) than those with low scores. Furthermore, the association between neuroticism and NA was stronger when adults with ASD were with less familiar people compared with being alone or with familiar people. No consistent corresponding significant interactions were found in the control group. In conclusion, in ASD, neuroticism moderates the association between appraised stress and NA as well as the association between social context and NA.