Recognition of facial affect has been studied extensively in adults with and without traumatic brain injury (TBI), mostly by asking examinees to match basic emotion words to isolated faces. This method may not capture affect labelling in everyday life when faces are in context and choices are open-ended. To examine effects of context and response format, we asked 148 undergraduate students to label emotions shown on faces either in isolation or in natural visual scenes. Responses were categorised as representing basic emotions, social emotions, cognitive state terms, or appraisals. We used students’ responses to create a scoring system that was applied prospectively to five men with TBI. In both groups, over 50% of responses were neither basic emotion words nor synonyms, and there was no significant difference in response types between faces alone vs. in scenes. Adults with TBI used labels not seen in students’ responses, talked more overall, and often gave multiple labels for one photo. Results suggest benefits of moving beyond forced-choice tests of faces in isolation to fully characterise affect recognition in adults with and without TBI.