Hostname: page-component-5db6c4db9b-k8dct Total loading time: 0 Render date: 2023-03-23T13:07:13.682Z Has data issue: true Feature Flags: { "useRatesEcommerce": false } hasContentIssue true

Prediction of depression symptoms in individual subjects with face and eye movement tracking

Published online by Cambridge University Press:  09 November 2020

Aleks Stolicyn*
Division of Psychiatry, Centre for Clinical Brain Sciences, University of Edinburgh, Kennedy Tower, Royal Edinburgh Hospital, Morningside Park, Edinburgh EH10 5HF, UK Institute for Adaptive and Neural Computation, School of Informatics, University of Edinburgh, 10 Crichton Street, Edinburgh EH8 9AB, UK
J. Douglas Steele
Division of Imaging Science and Technology, School of Medicine, Dundee University, Ninewells Hospital & Medical School, Dundee DD1 9SY, UK
Peggy Seriès
Institute for Adaptive and Neural Computation, School of Informatics, University of Edinburgh, 10 Crichton Street, Edinburgh EH8 9AB, UK
Author for correspondence: Aleks Stolicyn E-mail:



Depression is a challenge to diagnose reliably and the current gold standard for trials of DSM-5 has been in agreement between two or more medical specialists. Research studies aiming to objectively predict depression have typically used brain scanning. Less expensive methods from cognitive neuroscience may allow quicker and more reliable diagnoses, and contribute to reducing the costs of managing the condition. In the current study we aimed to develop a novel inexpensive system for detecting elevated symptoms of depression based on tracking face and eye movements during the performance of cognitive tasks.


In total, 75 participants performed two novel cognitive tasks with verbal affective distraction elements while their face and eye movements were recorded using inexpensive cameras. Data from 48 participants (mean age 25.5 years, standard deviation of 6.1 years, 25 with elevated symptoms of depression) passed quality control and were included in a case-control classification analysis with machine learning.


Classification accuracy using cross-validation (within-study replication) reached 79% (sensitivity 76%, specificity 82%), when face and eye movement measures were combined. Symptomatic participants were characterised by less intense mouth and eyelid movements during different stages of the two tasks, and by differences in frequencies and durations of fixations on affectively salient distraction words.


Elevated symptoms of depression can be detected with face and eye movement tracking during the cognitive performance, with a close to clinically-relevant accuracy (~80%). Future studies should validate these results in larger samples and in clinical populations.

Original Article
Copyright © The Author(s), 2020. Published by Cambridge University Press

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)


Alghowinem, S., Goecke, R., Wagner, M., Parker, G., & Breakspear, M. (2013). Eye movement analysis for depression detection. IEEE International Conference on Image Processing, Melbourne, VIC, 2013, pp. 42204224. Scholar
American Psychiatric Association. (2013). Diagnostic and statistical manual of mental disorders (5th ed.). Arlington, VA: Author.Google Scholar
Armstrong, T., & Olatunji, B. O. (2012). Eye tracking of attention in the affective disorders: A meta-analytic review and synthesis. Clinical Psychology Review, 32(8), 704723. ScholarPubMed
Baltrusaitis, T., Mahmoud, M., & Robinson, P. (2015). Cross-dataset learning and person-specific normalisation for automatic Action Unit detection. 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), Ljubljana, 2015, pp. 16. Scholar
Baltrusaitis, T., Robinson, P., & Morency, L.-P. (2016). OpenFace: An open source facial behavior analysis toolkit. IEEE Winter Conference on Applications of Computer Vision (WACV), Lake Placid, NY, 2016, pp. 110. Scholar
Bright, P., Jaldow, E., & Kopelman, M. D. (2002). The National Adult Reading Test as a measure of premorbid intelligence: A comparison with estimates derived from demographic variables. Journal of the International Neuropsychological Society: JINS, 8(6), 847854.CrossRefGoogle ScholarPubMed
Carvalho, N., Laurent, E., Noiret, N., Chopard, G., Haffen, E., Bennabi, D., & Vandel, P. (2015). Eye movement in unipolar and bipolar depression: A systematic review of the literature. Frontiers in Psychology, 6, 1809. ScholarPubMed
Cohn, J. F., Kruez, T. S., Matthews, I., Yang, Y., Nguyen, M. H., Padilla, M. T., … De la Torre, F. (2009). Detecting depression from facial actions and vocal prosody. 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops, Amsterdam, 2009, pp. 17. Scholar
Cortes, C., & Vapnik, V. (1995). Support-vector networks. Machine Learning, 20(3), 273297. Scholar
Cuthbert, B. N. (2014). The RDoC framework: Facilitating transition from ICD/DSM to dimensional approaches that integrate neuroscience and psychopathology. World Psychiatry: Official Journal of the World Psychiatric Association (WPA), 13(1), 2835. ScholarPubMed
Ekman, P. E., Friesen, W. V., & Hager, J. C. (2002). Facial action coding system (FACS). Salt Lake City, UT: Research Nexus.Google Scholar
Freedman, R., Lewis, D. A., Michels, R., Pine, D. S., Schultz, S. K., Tamminga, C. A., … Yager, J. (2013). The initial field trials of DSM-5: New blooms and old thorns. The American Journal of Psychiatry, 170(1), 15. ScholarPubMed
Gaebel, W., & Wölwer, W. (1992). Facial expression and emotional face recognition in schizophrenia and depression. European Archives of Psychiatry and Clinical Neuroscience, 242(1), 4652.CrossRefGoogle ScholarPubMed
Gaebel, W., & Wölwer, W. (2004). Facial expressivity in the course of schizophrenia and depression. European Archives of Psychiatry and Clinical Neuroscience, 254(5), 335342. ScholarPubMed
Gao, S., Calhoun, V. D., & Sui, J. (2018). Machine learning in major depression: From classification to treatment outcome prediction. CNS Neuroscience & Therapeutics, 24(11), 10371052. ScholarPubMed
Gehricke, J., & Shapiro, D. (2000). Reduced facial expression and social context in major depression: Discrepancies between facial muscle activity and self-reported emotion. Psychiatry Research, 95(2), 157167.CrossRefGoogle ScholarPubMed
Gehricke, J., & Shapiro, D. (2001). Facial and autonomic activity in depression: Social context differences during imagery. International Journal of Psychophysiology: Official Journal of the International Organization of Psychophysiology, 41(1), 5364.CrossRefGoogle ScholarPubMed
Girard, J. M., Cohn, J. F., Mahoor, M. H., Mavadati, S. M., Hammal, Z., & Rosenwald, D. P. (2014). Nonverbal Social Withdrawal in Depression: Evidence from manual and automatic analysis. Image and Vision Computing, 32(10), 641647. ScholarPubMed
Greden, J. F., Genero, N., Price, H. L., Feinberg, M., & Levine, S. (1986). Facial electromyography in depression. Subgroup differences. Archives of General Psychiatry, 43(3), 269274.CrossRefGoogle ScholarPubMed
Johnston, B. A., Steele, J. D., Tolomeo, S., Christmas, D., & Matthews, K. (2015). Structural MRI-based predictions in patients with Treatment-Refractory Depression (TRD). PloS One, 10(7), e0132958. Scholar
Johnston, B. A., Tolomeo, S., Gradin, V., Christmas, D., Matthews, K., & Steele, J. D. (2015). Failure of hippocampal deactivation during loss events in treatment-resistant depression. Brain: A Journal of Neurology, 138(Pt 9), 27662776. ScholarPubMed
Kambeitz, J., Cabral, C., Sacchet, M. D., Gotlib, I. H., Zahn, R., Serpa, M. H., … Koutsouleris, N. (2017). Detecting neuroimaging biomarkers for depression: A meta-analysis of multivariate pattern recognition studies. Biological Psychiatry, 82(5), 330338. ScholarPubMed
McIntyre, R. S., Cha, D. S., Soczynska, J. K., Woldeyohannes, H. O., Gallaugher, L. A., Kudlow, P., … Baskaran, A. (2013). Cognitive deficits and functional outcomes in major depressive disorder: Determinants, substrates, and treatment interventions. Depression and Anxiety, 30(6), 515527. ScholarPubMed
McLellan, T. M., Caldwell, J. A., & Lieberman, H. R. (2016). A review of caffeine's effects on cognitive, physical and occupational performance. Neuroscience and Biobehavioral Reviews, 71, 294312. ScholarPubMed
Mwangi, B., Ebmeier, K. P., Matthews, K., & Steele, J. D. (2012). Multi-centre diagnostic classification of individual structural neuroimaging scans from patients with major depressive disorder. Brain: A Journal of Neurology, 135(Pt 5), 15081521. ScholarPubMed
Mwangi, B., Tian, T. S., & Soares, J. C. (2014). A review of feature reduction techniques in neuroimaging. Neuroinformatics, 12(2), 229244. ScholarPubMed
National Institute for Health and Care Excellence. (2009). NICE clinical guideline 90: Depression in adults: The treatment and management of depression in adults. Retrieved from Scholar
Pampouchidou, A., Simos, P. G., Marias, K., Meriaudeau, F., Yang, F., Pediaditis, M., & Tsiknakis, M. (2019). Automatic assessment of depression based on visual cues: A systematic review. IEEE Transactions on Affective Computing, 10(4), 445470. Scholar
Plant, R. R., & Turner, G. (2009). Millisecond precision psychological research in a world of commodity computers: New hardware, new problems? Behavior Research Methods, 41(3), 598614. Scholar
Radloff, L. S. (1977). The CES-D scale: A self-report depression scale for research in the general population. Applied Psychological Measurement, 1(3), 385401. Scholar
Renneberg, B., Heyn, K., Gebhard, R., & Bachmann, S. (2005). Facial expression of emotions in borderline personality disorder and depression. Journal of Behavior Therapy and Experimental Psychiatry, 36(3), 183196. ScholarPubMed
Rock, P. L., Roiser, J. P., Riedel, W. J., & Blackwell, A. D. (2014). Cognitive impairment in depression: A systematic review and meta-analysis. Psychological Medicine, 44(10), 20292040. ScholarPubMed
Rosa, M. J., Portugal, L., Hahn, T., Fallgatter, A. J., Garrido, M. I., Shawe-Taylor, J., & Mourao-Miranda, J. (2015). Sparse network-based models for patient classification using fMRI. NeuroImage, 105, 493506. ScholarPubMed
Rottenberg, J., Gross, J. J., & Gotlib, I. H. (2005). Emotion context insensitivity in major depressive disorder. Journal of Abnormal Psychology, 114(4), 627639. ScholarPubMed
Saunders, J. B., Aasland, O. G., Babor, T. F., de la Fuente, J. R., & Grant, M. (1993). Development of the Alcohol Use Disorders Identification Test (AUDIT): WHO collaborative project on early detection of persons with harmful alcohol consumption--II. Addiction (Abingdon, England), 88(6), 791804.CrossRefGoogle Scholar
Schmaal, L., Veltman, D. J., van Erp, T. G. M., Sämann, P. G., Frodl, T., Jahanshad, N., … … Hibar, D. P. (2016). Subcortical brain alterations in major depressive disorder: Findings from the ENIGMA Major Depressive Disorder working group. Molecular Psychiatry, 21(6), 806812. ScholarPubMed
Schneider, F., Heimann, H., Himer, W., Huss, D., Mattes, R., & Adam, B. (1990). Computer-based analysis of facial action in schizophrenic and depressed patients. European Archives of Psychiatry and Clinical Neuroscience, 240(2), 6776.CrossRefGoogle ScholarPubMed
Schwartz, G. E., Fair, P. L., Salt, P., Mandel, M. R., & Klerman, G. L. (1976a). Facial muscle patterning to affective imagery in depressed and nondepressed subjects. Science (New York, NY), 192(4238), 489491.CrossRefGoogle Scholar
Schwartz, G. E., Fair, P. L., Salt, P., Mandel, M. R., & Klerman, G. L. (1976b). Facial expression and imagery in depression: An electromyographic study. Psychosomatic Medicine, 38(5), 337347.CrossRefGoogle Scholar
Sloan, D. M., Strauss, M. E., Quirk, S. W., & Sajatovic, M. (1997). Subjective and expressive emotional responses in depression. Journal of Affective Disorders, 46(2), 135141.CrossRefGoogle ScholarPubMed
Steele, J. D., & Paulus, M. P. (2019). Pragmatic neuroscience for clinical psychiatry. The British Journal of Psychiatry: The Journal of Mental Science, 215(1), 404408. ScholarPubMed
Stratou, G., Scherer, S., Gratch, J., & Morency, L.-P. (2015). Automatic nonverbal behavior indicators of depression and PTSD: The effect of gender. Journal on Multimodal User Interfaces, 9(1), 1729. Scholar
Teasdale, J. D., & Bancroft, J. (1977). Manipulation of thought content as a determinant of mood and corrugator electromyographic activity in depressed patients. Journal of Abnormal Psychology, 86(3), 235241.CrossRefGoogle ScholarPubMed
Teasdale, J. D., & Rezin, V. (1978). Effect of thought-stopping on thoughts, mood and corrugator EMG in depressed patients. Behaviour Research and Therapy, 16(2), 97102.CrossRefGoogle ScholarPubMed
Wang, L., Shen, X., Wu, Y., & Zhang, D. (2016). Coffee and caffeine consumption and depression: A meta-analysis of observational studies. The Australian and New Zealand Journal of Psychiatry, 50(3), 228242. ScholarPubMed
Warriner, A. B., Kuperman, V., & Brysbaert, M. (2013). Norms of valence, arousal, and dominance for 13915 English lemmas. Behavior Research Methods, 45(4), 11911207. ScholarPubMed
Wei, M., Qin, J., Yan, R., Li, H., Yao, Z., & Lu, Q. (2013). Identifying major depressive disorder using Hurst exponent of resting-state brain networks. Psychiatry Research, 214(3), 306312. ScholarPubMed
Wexler, B. E., Levenson, L., Warrenburg, S., & Price, L. H. (1994). Decreased perceptual sensitivity to emotion-evoking stimuli in depression. Psychiatry Research, 51(2), 127138.CrossRefGoogle ScholarPubMed
Zeng, L.-L., Shen, H., Liu, L., Wang, L., Li, B., Fang, P., … Hu, D. (2012). Identifying major depression using whole-brain functional connectivity: A multivariate pattern analysis. Brain: A Journal of Neurology, 135(Pt 5), 14981507. ScholarPubMed
Zhang, Y., & Yang, Y. (2015). Cross-validation for selecting a model selection procedure. Journal of Econometrics, 187(1), 95112. Scholar
Zugal, S., & Pinggera, J. (2014). Low–cost Eye–trackers: Useful for information systems research? In Iliadis, L., Papazoglou, M., & Pohl, K. (Eds.), Advanced information systems engineering workshops (Vol. 178, pp. 159170). Cham: Springer International Publishing. Scholar
Supplementary material: PDF

Stolicyn et al. supplementary material

Stolicyn et al. supplementary material

Download Stolicyn et al. supplementary material(PDF)
PDF 335 KB