Hostname: page-component-7bb8b95d7b-qxsvm Total loading time: 0 Render date: 2024-09-10T02:12:23.342Z Has data issue: false hasContentIssue false
  • English
  • Français

Detecting suicide risk among U.S. servicemembers and veterans: a deep learning approach using social media data

Published online by Cambridge University Press:  09 September 2024

Kelly L. Zuromski Open the ORCID record for Kelly L. Zuromski [Opens in a new window] ,
Daniel M. Low ,
Noah C. Jones ,
Richard Kuzma ,
Daniel Kessler ,
Liutong Zhou ,
Erik K. Kastman ,
Jonathan Epstein ,
Carlos Madden  and
Satrajit S. Ghosh
...Show all authors
Kelly L. Zuromski*
Affiliation:
Department of Psychology, Harvard University, Cambridge, MA, USA Franciscan Children's, Brighton, MA, USA
Daniel M. Low
Affiliation:
Speech and Hearing Bioscience and Technology Program, Harvard Medical School, Boston, MA, USA McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge MA
Noah C. Jones
Affiliation:
Department of Psychology, Harvard University, Cambridge, MA, USA MIT Media Lab, Massachusetts Institute of Technology, Cambridge, MA, USA
Richard Kuzma
Affiliation:
Department of Psychology, Harvard University, Cambridge, MA, USA
Daniel Kessler
Affiliation:
Department of Psychology, Harvard University, Cambridge, MA, USA
Liutong Zhou
Affiliation:
Machine Learning Solutions Lab, Amazon Web Services, New York, NY, USA
Erik K. Kastman
Affiliation:
Department of Psychology, Harvard University, Cambridge, MA, USA RallyPoint Networks, Inc., Boston, MA, USA
Jonathan Epstein
Affiliation:
RallyPoint Networks, Inc., Boston, MA, USA
Carlos Madden
Affiliation:
RallyPoint Networks, Inc., Boston, MA, USA
Satrajit S. Ghosh
Affiliation:
Speech and Hearing Bioscience and Technology Program, Harvard Medical School, Boston, MA, USA McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge MA
David Gowel
Affiliation:
RallyPoint Networks, Inc., Boston, MA, USA
Matthew K. Nock
Affiliation:
Department of Psychology, Harvard University, Cambridge, MA, USA
*
Corresponding author: Kelly L. Zuromski; Email: kelly_zuromski@fas.harvard.edu
Get access Rights & Permissions [Opens in a new window]

Abstract

Background

Military Servicemembers and Veterans are at elevated risk for suicide, but rarely self-identify to their leaders or clinicians regarding their experience of suicidal thoughts. We developed an algorithm to identify posts containing suicide-related content on a military-specific social media platform.

Methods

Publicly-shared social media posts (n = 8449) from a military-specific social media platform were reviewed and labeled by our team for the presence/absence of suicidal thoughts and behaviors and used to train several machine learning models to identify such posts.

Results

The best performing model was a deep learning (RoBERTa) model that incorporated post text and metadata and detected the presence of suicidal posts with relatively high sensitivity (0.85), specificity (0.96), precision (0.64), F1 score (0.73), and an area under the precision-recall curve of 0.84. Compared to non-suicidal posts, suicidal posts were more likely to contain explicit mentions of suicide, descriptions of risk factors (e.g. depression, PTSD) and help-seeking, and first-person singular pronouns.

Conclusions

Our results demonstrate the feasibility and potential promise of using social media posts to identify at-risk Servicemembers and Veterans. Future work will use this approach to deliver targeted interventions to social media users at risk for suicide.

Keywords

machine learningmilitarynatural language processingsuicideveterans
Type
Original Article
Information
Psychological Medicine , First View , pp. 1 - 10
Check for updates
Copyright
Copyright © The Author(s), 2024. Published by Cambridge University Press

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

*

Equal contribution.

References

Adler, A., Jager-Hyman, S., Brown, G. K., Singh, T., Chaudhury, S., Ghahramanlou-Holloway, M., & Stanley, B. (2020). A qualitative investigation of barriers to seeking treatment for suicidal thoughts and behaviors among army soldiers with a deployment history. Archives of Suicide Research, 24, 251–268. https://doi.org/10.1080/13811118.2019.1624666CrossRefGoogle ScholarPubMed
Akiba, T., Sano, S., Yanase, T., Ohta, T., & Koyama, M. (2019). Optuna: A next-generation hyperparameter optimization framework. Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2623–2631. https://doi.org/10.1145/3292500.3330701CrossRefGoogle Scholar
Belsher, B. E., Smolenski, D. J., Pruitt, L. D., Bush, N. E., Beech, E. H., Workman, D. E., & …Skopp, N. A. (2019). Prediction models for suicide attempts and deaths: A systematic review and simulation. JAMA Psychiatry, 76, 642–651. https://doi.org/10.1001/jamapsychiatry.2019.0174CrossRefGoogle ScholarPubMed
Berry-Blunt, A. K., Holtzman, N. S., Donnellan, M. B., & Mehl, M. R. (2021). The story of “I” tracking: Psychological implications of self-referential language use. Social and Personality Psychology Compass, 15, e12647. https://doi.org/10.1111/spc3.12647CrossRefGoogle Scholar
Bryan, C. J., Butner, J. E., Sinclair, S., Bryan, A. B. O., Hesse, C. M., & Rose, A. E. (2018). Predictors of emerging suicide death among military personnel on social media networks. Suicide and Life-Threatening Behavior, 48, 413–430. https://doi.org/10.1111/sltb.12370CrossRefGoogle ScholarPubMed
Castillo-Sánchez, G., Marques, G., Dorronzoro, E., Rivera-Romero, O., Franco-Martín, M., & De la Torre-Díez, I. (2020). Suicide risk assessment using machine learning and social networks: A scoping review. Journal of Medical Systems, 44, 1–15. https://doi.org/10.1007/s10916-020-01669-5CrossRefGoogle ScholarPubMed
Colpe, L. J., Naifeh, J. A., Aliaga, P. A., Sampson, N. A., Heeringa, S. G., Stein, M. B., … Kessler, R. C. (2015). Mental health treatment among soldiers with current mental disorders in the army study to assess risk and resilience in service members (Army STARRS). Military Medicine, 180, 1041–1051. https://doi.org/10.7205/MILMED-D-14-00686CrossRefGoogle ScholarPubMed
De Choudhury, M., & De, S. (2014). Mental health discourse on reddit: Self-disclosure, social support, and anonymity. Eighth International AAAI Conference on Weblogs and Social Media, 8, 71–80. https://doi.org/10.1609/icwsm.v8i1.14526CrossRefGoogle Scholar
De Choudhury, M., Kiciman, E., Dredze, M., Coppersmith, G., & Kumar, M. (2016). Discovering shifts to suicidal ideation from mental health content in social media. Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, 2098–2110. https://doi.org/10.1145/2858036.2858207CrossRefGoogle Scholar
Franz, P. J., Nook, E. C., Mair, P., & Nock, M. K. (2020). Using topic modeling to detect and describe self-injurious and related content on a large-scale digital platform. Suicide and Life-Threatening Behavior, 50, 5–18. https://doi.org/10.1111/sltb.12569CrossRefGoogle ScholarPubMed
Gamer, M., Lemon, J., Fellows, I., & Singh, P. (2019). irr: Various coefficients of interrater reliability and agreement. Retrieved from https://CRAN.R-project.org/package=irrGoogle Scholar
Gu, K., & Budhkar, A. (2021). A package for learning on tabular and text data with transformers. Proceedings of the Third Workshop on Multimodal Artificial Intelligence, 69–73. https://doi.org/10.18653/v1/2021.maiworkshop-1.10CrossRefGoogle Scholar
Hoge, C. W., Castro, C. A., Messer, S. C., McGurk, D., Cotting, D. I., & Koffman, R. L. (2004). Combat duty in Iraq and Afghanistan, mental health problems, and barriers to care. New England Journal of Medicine, 351, 13–22. https://doi.org/10.1016/s0084-3970(08)70136-1CrossRefGoogle ScholarPubMed
Homan, S., Gabi, M., Klee, N., Bachmann, S., Moser, A.-M., Duri, M., …Kleim, B. (2022). Linguistic features of suicidal thoughts and behaviors: A systematic review. Clinical Psychology Review, 95, 102161. https://doi.org/10.1016/j.cpr.2022.102161CrossRefGoogle ScholarPubMed
Hughes, J. (2021). krippendorffsalpha: An R package for measuring agreement using Krippendorff's Alpha coefficient. The R Journal, 13, 413–425. https://doi.org/10.32614/rj-2021-046CrossRefGoogle Scholar
Jaroszewski, A. C., Morris, R. R., & Nock, M. K. (2019). Randomized controlled trial of an online machine learning-driven risk assessment and intervention platform for increasing the use of crisis services. Journal of Consulting and Clinical Psychology, 87, 370–379. https://doi.org/10.1037/ccp0000389CrossRefGoogle ScholarPubMed
Jones, N., Jaques, N., Pataranutaporn, P., Ghandeharioun, A., & Picard, R. (2019). Analysis of Online Suicide Risk with Document Embeddings and Latent Dirichlet Allocation. 2019 8th International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW), 1–5. https://doi.org/10.1109/ACIIW.2019.8925077CrossRefGoogle Scholar
Kang, H. K., Bullman, T. A., Smolenski, D. J., Skopp, N. A., Gahm, G. A., & Reger, M. A. (2015). Suicide risk among 1.3 million veterans who were on active duty during the Iraq and Afghanistan wars. Annals of Epidemiology, 25, 96–100. https://doi.org/10.1016/j.annepidem.2014.11.020CrossRefGoogle ScholarPubMed
Kessler, J. (2017). Scattertext: A browser-based tool for visualizing how corpora differ. Proceedings of ACL 2017, System Demonstrations, 85–90. https://doi.org/10.18653/v1/P17-4015CrossRefGoogle Scholar
Kessler, R. C. (2019). Clinical epidemiological research on suicide-related behaviors—where we are and where we need to go. JAMA Psychiatry, 76, 777–778. https://doi.org/10.1001/jamapsychiatry.2019.1238CrossRefGoogle ScholarPubMed
Kessler, R. C., Bossarte, R. M., Luedtke, A., Zaslavsky, A. M., & Zubizarreta, J. R. (2020). Suicide prediction models: A critical review of recent research with recommendations for the way forward. Molecular Psychiatry, 25, 168–179. https://doi.org/10.1038/s41380-019-0531-0CrossRefGoogle ScholarPubMed
Lekkas, D., Klein, R. J., & Jacobson, N. C. (2021). Predicting acute suicidal ideation on Instagram using ensemble machine learning models. Internet Interventions, 25, 100424. https://doi.org/10.1016/j.invent.2021.100424CrossRefGoogle ScholarPubMed
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., & …Stoyanov, V. (2019). RoBERTa: A robustly optimized BERT pretraining approach. ArXiv. https://doi.org/10.48550/arXiv.1907.11692CrossRefGoogle Scholar
Low, D., Rumker, L., Talkar, T., Torous, J., Cecchi, G., & Ghosh, S. S. (2020). Natural language processing reveals vulnerable mental health support groups and heightened health anxiety on Reddit during COVID-19: Observational study. Journal of Medical Internet Research, 22, e22635. https://doi.org/10.2196/22635CrossRefGoogle ScholarPubMed
Low, D., Zuromski, K., Kessler, D., Ghosh, S. S., Nock, M. K., & Dempsey, W. (2021). It's quality and quantity: The effect of the amount of comments on online suicidal posts. Proceedings of the First Workshop on Causal Inference and NLP, 95–103. https://doi.org/10.18653/v1/2021.cinlp-1.8CrossRefGoogle Scholar
MacAvaney, S., Mittu, A., Coppersmith, G., Leintz, J., & Resnik, P. (2021). Community-level research on suicidality prediction in a secure environment: Overview of the CLPsych 2021 shared task. Proceedings of the Seventh Workshop on Computational Linguistics and Clinical Psychology: Improving Access, 70–80. https://doi.org/10.18653/v1/2021.clpsych-1.7CrossRefGoogle Scholar
Monroe, B. L., Colaresi, M. P., & Quinn, K. M. (2008). Fightin’ words: Lexical feature selection and evaluation for identifying the content of political conflict. Political Analysis, 16, 372–403. https://doi.org/10.1093/pan/mpn018CrossRefGoogle Scholar
Naslund, J. A., Bondre, A., Torous, J., & Aschbrenner, K. A. (2020). Social media and mental health: Benefits, risks, and opportunities for research and practice. Journal of Technology in Behavioral Science, 5, 245–257. https://doi.org/10.1007/s41347-020-00134-xCrossRefGoogle ScholarPubMed
O'Dea, B., Wan, S., Batterham, P. J., Calear, A. L., Paris, C., & Christensen, H. (2015). Detecting suicidality on Twitter. Internet Interventions, 2, 183–188. https://doi.org/10.1016/j.invent.2015.03.005CrossRefGoogle Scholar
Ophir, Y., Tikochinski, R., Asterhan, C. S. C., Sisso, I., & Reichart, R. (2020). Deep neural networks detect suicide risk from textual Facebook posts. Scientific Reports, 10, 16685. https://doi.org/10.1038/s41598-020-73917-0CrossRefGoogle ScholarPubMed
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., … Duchesnay, E. (2011). Scikit-learn: Machine learning in python. The Journal of Machine Learning, 12, 2825–2830.Google Scholar
Ratner, A., Bach, S. H., Ehrenberg, H., Fries, J., Wu, S., & Ré, C. (2020). Snorkel: Rapid training data creation with weak supervision. The VLDB Journal, 29, 709–730. https://doi.org/10.1007/s00778-019-00552-1CrossRefGoogle ScholarPubMed
Ribeiro, J. D., Gutierrez, P. M., Joiner, T. E., Kessler, R. C., Petukhova, M. V., Sampson, N. A., & …Nock, M. K. (2017). Health care contact and suicide risk documentation prior to suicide death: Results from the Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS). Journal of Consulting and Clinical Psychology, 85, 403–408. https://doi.org/10.1037/ccp0000178CrossRefGoogle ScholarPubMed
Saito, T., & Rehmsmeier, M. (2015). The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets. PLOS ONE, 10, e0118432. https://doi.org/10.1371/journal.pone.0118432CrossRefGoogle Scholar
Sparck Jones, K. (1972). A statistical interpretation of term specificity and its application in retrieval. Journal of Documentation, 28, 11–21. https://doi.org/10.1108/eb026526CrossRefGoogle Scholar
Suitt, T. H. (2021). High suicide rates among United States service members and veterans of the post 9/11 wars. Providence, RI: Watson Institute for International and Public Affairs, Brown University. Retrieved from https://watson.brown.edu/costsofwar/files/cow/imce/papers/2021/Suitt_Suicides_Costs%20of%20War_June%2021%202021.pdf.Google Scholar
Teo, A. R., Marsh, H. E., Liebow, S. B. L., Chen, J. I., Forsberg, C. W., Nicolaidis, C., & …Dobscha, S. K. (2018). Help-seeking on Facebook versus more traditional sources of help: Cross-sectional survey of military veterans. Journal of Medical Internet Research, 20, e62. https://doi.org/10.2196/jmir.9007CrossRefGoogle ScholarPubMed
VanSickle, M., Werbel, A., Perera, K., Pak, K., DeYoung, K., & Ghahramanlou-Holloway, M. (2016). Perceived barriers to seeking mental health care among United States Marine Corps noncommissioned officers serving as gatekeepers for suicide prevention. Psychological Assessment, 28, 1020–1025. https://doi.org/10.1037/pas0000212CrossRefGoogle ScholarPubMed
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., … Rush, A. (2020). Transformers: State-of-the-art natural language processing. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, 38–45. https://doi.org/10.18653/v1/2020.emnlp-demos.6CrossRefGoogle Scholar
Zuromski, K. L., & Nock, M. K. (2023). Brief Interventions on Social Media to Reduce Suicide Risk (Intervention 3). ClinicalTrials.gov ID NCT06114849. Retrieved from https://clinicaltrials.gov/study/NCT06114849Google Scholar
Zuromski, K. L., Dempsey, C. L., Ng, T. H., Riggs-Donovan, C. A., Brent, D. A., Heeringa, S. G., & …Nock, M. K. (2019). Utilization of and barriers to treatment among suicide decedents: Results from the Army Study to Assess Risk and Resilience among Servicemembers (Army STARRS). Journal of Consulting and Clinical Psychology, 87, 671–683. https://doi.org/10.1037/ccp0000400CrossRefGoogle ScholarPubMed
Supplementary material: File

Zuromski et al. supplementary material

Zuromski et al. supplementary material
Download Zuromski et al. supplementary material(File)
File 94.2 KB