Hostname: page-component-848d4c4894-8kt4b Total loading time: 0 Render date: 2024-06-18T19:40:13.676Z Has data issue: false hasContentIssue false

Development and evaluation of a predictive algorithm and telehealth intervention to reduce suicidal behavior among university students

Published online by Cambridge University Press:  21 September 2023

Penelope A. Hasking*
Affiliation:
School of Population Health, Faculty of Health Sciences, Curtin University, Perth, Australia Faculty of Health Sciences, enAble Institute, Curtin University, Perth, Australia
Kealagh Robinson
Affiliation:
School of Population Health, Faculty of Health Sciences, Curtin University, Perth, Australia Faculty of Health Sciences, enAble Institute, Curtin University, Perth, Australia
Peter McEvoy
Affiliation:
School of Population Health, Faculty of Health Sciences, Curtin University, Perth, Australia Faculty of Health Sciences, enAble Institute, Curtin University, Perth, Australia Centre for Clinical Interventions, Perth, Australia
Glenn Melvin
Affiliation:
Centre for Social and Early Emotional Development, School of Psychology, Faculty of Health, Deakin University, Geelong, Australia
Ronny Bruffaerts
Affiliation:
University Psychiatric Center, KU Leuven, Leuven, Belgium
Mark E. Boyes
Affiliation:
School of Population Health, Faculty of Health Sciences, Curtin University, Perth, Australia Faculty of Health Sciences, enAble Institute, Curtin University, Perth, Australia
Randy P. Auerbach
Affiliation:
Department of Psychiatry, Columbia University, New York, USA Division of Clinical Developmental Neuroscience, Sackler Institute, New York, USA
Delia Hendrie
Affiliation:
School of Population Health, Faculty of Health Sciences, Curtin University, Perth, Australia
Matthew K. Nock
Affiliation:
Department of Psychology, Harvard University, Cambridge, USA
David A. Preece
Affiliation:
School of Population Health, Faculty of Health Sciences, Curtin University, Perth, Australia Faculty of Health Sciences, enAble Institute, Curtin University, Perth, Australia
Clare Rees
Affiliation:
School of Population Health, Faculty of Health Sciences, Curtin University, Perth, Australia
Ronald C. Kessler
Affiliation:
Department of Health Care Policy, Harvard Medical School, Boston, USA
*
Corresponding author: Penelope A. Hasking; Email: Penelope.Hasking@curtin.edu.au
Rights & Permissions [Opens in a new window]

Abstract

Background

Suicidal behaviors are prevalent among college students; however, students remain reluctant to seek support. We developed a predictive algorithm to identify students at risk of suicidal behavior and used telehealth to reduce subsequent risk.

Methods

Data come from several waves of a prospective cohort study (2016–2022) of college students (n = 5454). All first-year students were invited to participate as volunteers. (Response rates range: 16.00–19.93%). A stepped-care approach was implemented: (i) all students received a comprehensive list of services; (ii) those reporting past 12-month suicidal ideation were directed to a safety planning application; (iii) those identified as high risk of suicidal behavior by the algorithm or reporting 12-month suicide attempt were contacted via telephone within 24-h of survey completion. Intervention focused on support/safety-planning, and referral to services for this high-risk group.

Results

5454 students ranging in age from 17–36 (s.d. = 5.346) participated; 65% female. The algorithm identified 77% of students reporting subsequent suicidal behavior in the top 15% of predicted probabilities (Sensitivity = 26.26 [95% CI 17.93–36.07]; Specificity = 97.46 [95% CI 96.21–98.38], PPV = 53.06 [95% CI 40.16–65.56]; AUC range: 0.895 [95% CIs 0.872–0.917] to 0.966 [95% CIs 0.939–0.994]). High-risk students in the Intervention Cohort showed a 41.7% reduction in probability of suicidal behavior at 12-month follow-up compared to high-risk students in the Control Cohort.

Conclusions

Predictive risk algorithms embedded into universal screening, coupled with telehealth intervention, offer significant potential as a suicide prevention approach for students.

Type
Original Article
Creative Commons
Creative Common License - CCCreative Common License - BY
This is an Open Access article, distributed under the terms of the Creative Commons Attribution licence (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted re-use, distribution and reproduction, provided the original article is properly cited.
Copyright
Copyright © The Author(s), 2023. Published by Cambridge University Press

Suicide is a leading cause of death among 15–29 year olds worldwide (WHO, 2021). College students, many of whom are young adults (OECD, 2021), are particularly likely to experience suicidal thoughts and behaviors (Mortier et al., Reference Mortier, Auerbach, Alonso, Bantjes, Benjet, Cuijpers and Vives2018). Drawing from representative samples in nine countries, the World Health Organization's World Mental Health Surveys International College Student Initiative (WMH-ICS) estimated that 17.2% of students have experienced suicide ideation in the last 12-months, 53.4% of those who thought about suicide developed a suicide plan, and 22% of those with a plan made a suicide attempt (Mortier et al., Reference Mortier, Auerbach, Alonso, Bantjes, Benjet, Cuijpers and Vives2018). Furthermore, 30% of students who enter college with a history of suicidal thoughts and behaviors will continue to experience these thoughts and behaviors over the following two years (Kiekens et al., Reference Kiekens, Claes, Hasking, Mortier, Bootsma, Boyes and Bruffaerts2022; Mortier et al., Reference Mortier, Demyttenaere, Auerbach, Cuijpers, Green, Kiekens and Bruffaerts2017a, Reference Mortier, Kiekens, Auerbach, Cuijpers, Demyttenaere, Green and Bruffaerts2017b). College entrance, therefore, offers a strategic opportunity for suicide prevention and early intervention to reduce suicide risk.

A complication in providing these interventions is that, despite high clinical need, college students are less likely than the general public to seek help, even when they perceive a need (Bruffaerts et al., Reference Bruffaerts, Demyttenaere, Hwang, Chiu, Sampson, Kessler and Nock2011). A potential solution to low treatment seeking is to conduct universal screening of incoming students, identify those at high risk of suicidal behavior, and provide stepped referral to appropriate interventions (Mortier et al., Reference Mortier, Auerbach, Alonso, Bantjes, Benjet, Cuijpers and Vives2018). However, suicide is a complex and multifaceted behavior that is difficult to predict. Commonly studied risk factors (e.g. prior suicidal thoughts and behavior), when examined in isolation, are unreliable in identifying people at greatest future risk (Franklin et al., Reference Franklin, Ribeiro, Fox and Nock2017; Riberio et al., Reference Riberio, Franklin, Fox, Bentley, Kleiman and Chang2016). In addition, clinical risk assessment practices that attempt to categorize individuals as high v. low risk based on past suicidal behavior and psychiatric history (e.g. Manchester Self-Harm Rule, Cooper, Kapur, Dunning, Guthrie, and Appleby, Reference Cooper, Kapur, Dunning, Guthrie and Appleby2006; ReAct Self-Harm Rule, Steeg et al., Reference Steeg, Kapur, Webb, Applegate, Stewart, Hawton and Cooper2012) are generally poor (< 5% accurate) at meaningfully predicting future risk (Carter et al., Reference Carter, Milner, McGill, Pirkis, Kapur and Spittal2017; Large et al., Reference Large, Kaneson, Myles, Myles, Gunaratne and Ryan2016; Quinlivan et al., Reference Quinlivan, Cooper, Davies, Hawton, Gunnell and Kapur2016).

Given these limitations, there have been calls to shift focus from isolating individual risk factors and toward developing multivariable predictive risk algorithms (Franklin et al., Reference Franklin, Ribeiro, Fox and Nock2017; Riberio et al., Reference Riberio, Franklin, Fox, Bentley, Kleiman and Chang2016). Such algorithms have been developed to predict suicidal thoughts and behaviors over relatively short follow-up periods (Kessler et al., Reference Kessler, Warner, Ivany, Petukhova, Rose and Bromet2015; Miché et al., Reference Miché, Studerus, Meyer, Gloster, Beesdo-Baum, Wittchen and Lieb2020; Mortier et al., Reference Mortier, Kiekens, Auerbach, Cuijpers, Demyttenaere, Green and Bruffaerts2017b). For example, Mortier et al. (Reference Mortier, Demyttenaere, Auerbach, Cuijpers, Green, Kiekens and Bruffaerts2017a) developed actuarial algorithms to identify college students most at risk of suicidal thoughts and behaviors during the first year of college. Modeling indicated that targeting the top 10% of at-risk students could capture 51–66% of first-onset suicidal thoughts and behaviors. There is also a need to adapt this approach to identify incidence of suicidal behavior over later years of college.

Locally derived approaches are necessary to accurately identify and triage college students at high risk of suicidal behavior (WHO, 2014). Using previously reported procedures (Mortier et al., Reference Mortier, Demyttenaere, Auerbach, Cuijpers, Green, Kiekens and Bruffaerts2017a, Reference Mortier, Kiekens, Auerbach, Cuijpers, Demyttenaere, Green and Bruffaerts2017b), we developed a multivariable predictive screening algorithm to identify students at risk of upcoming suicidal behavior (i.e. plans and/or attempts) for use in Australian universities. Our objectives are to describe the development of the algorithm and evaluate the utility of using the algorithm to target a telehealth support intervention (Checking On Mental Health Providing Alternatives to Suicide for Students; COMPAS-S, Hasking, Chiu, Robinson, Coleman, and McEvoy, Reference Hasking, Chiu, Robinson, Coleman and McEvoy2023) for reducing suicidal behavior and increasing mental health support for students. We compare intervention outcomes with a retrospective control cohort who did not receive the intervention.

Method

Participants

Data were collected as part of the Australian arm of the WMH-ICS (Bruffaerts et al., Reference Bruffaerts, Mortier, Kiekens, Auerbach, Cuijpers, Demyttenaere and Kessler2018). In brief, all incoming first-year students at a large public Australian university were invited to take part in a baseline survey from 2016 to 2022. All students were sent an email with a personalized link to the survey with weekly reminders sent for the first five weeks of semester. Participants were offered an AUD$10 gift card in acknowledgement of their participation. Students were also invited to complete a 12-month follow-up survey.

Design

Data from the 2016–2017 cohorts were used to develop the algorithm (Development Cohort, N = 1202; invited N = 7500, 16% response rate), using data collected at baseline to predict suicide plan and/or attempt at 12-month follow-up. From 2020–2022, we integrated the algorithm into the WMH-ICS survey, providing telehealth support to students identified as being at high risk of suicide plan or attempt (Intervention Cohort; N = 2592, invited N = 13500, 19.20% response rate). These students provided baseline data, received a telehealth assessment within 24 h of completing the survey, and received a 4-week follow-up call. They then completed a 12-month follow-up survey. We used baseline data from the 2018–2019 sample, who did not receive the intervention, as a retrospective control cohort against which to compare the impact of this intervention on suicide plan and/or attempt and treatment access reported in the 12-month follow-up (Control Cohort: N = 1661, invited N = 10130, 16.40% response rate see online Supplementary Fig. S1). The samples were representative of the broader university cohorts (R-indicators: 0.87–0.95).

No students were removed from the study due to suicide risk. It is worth noting that the algorithm was not designed to identify students at imminent risk of suicide. Rather it was designed to identify students who may be at risk of suicide plan and/or attempt sometime in the coming 12 months. Our program was designed to intervene with these students before they reach crisis point. Students in the algorithm development and control cohorts who reported suicidal ideation in the past 12 months were directed to a safety planning smartphone application (Melvin et al., Reference Melvin, Gresham, Beaton, Coles, Tonge, Gordon and Stanley2019). This gold standard approach to suicide prevention allowed students to complete their own safety plan, either alone or in conjunction with a mental health professional.

Measures

The WMH-ICS baseline and 12-month follow-up surveys were developed by the World Mental Health Survey Consortium. Further details regarding the survey instruments can be found at: https://www.hcp.med.harvard.edu/wmh/college_student_survey.php. The survey was designed to take an average thirty minutes to complete.

Main outcome: suicidal behaviors

Items from the Self-Injurious Thoughts and Behaviors Interview (SITBI; Nock, Holmberg, Photos, and Michel, Reference Nock, Holmberg, Photos and Michel2007) were used to assess suicidal ideation, suicide plans, and suicide attempts. The baseline survey assessed both lifetime and 12-month suicidal thoughts and behaviors; the 12-month follow-up survey assessed 12-month suicidal thoughts and behaviors. Construct validity and test-retest reliability for the SITBI have previously been reported as good to excellent (Nock et al., Reference Nock, Holmberg, Photos and Michel2007). Our outcome variable was suicidal behaviors, operationalized as plans and/or attempt.

Predictive risk algorithm factors

The algorithm included 38 binary variables derived from the baseline survey (Table 1). These variables were selected based on their inclusion in the previous risk algorithm developed by the WMH-ICS team (Mortier et al., Reference Mortier, Demyttenaere, Auerbach, Cuijpers, Green, Kiekens and Bruffaerts2017a, Reference Mortier, Kiekens, Auerbach, Cuijpers, Demyttenaere, Green and Bruffaerts2017b), that predicted the onset of suicidal thoughts. Socio-demographic characteristics were obtained from the (university redacted for review) administration office. We included eleven mental health variables assessed at baseline. Mental health diagnoses were derived by standard scoring cut-offs on the Composite International Diagnostic Interview, 3rd version (CIDI-3.0l Kessler et al., Reference Kessler, Abelson, Demler, Escobar, Gibbon, Guyer and Zheng2004), based on the number of symptom criteria met. Adverse childhood events (prior to age 17) were assessed with items from the CIDI-3.0 and the Adverse Childhood Experience Scale (Felitti et al., Reference Felitti, Anda, Nordenberg, Williamson, Spitz, Edwards and Marks2019). Items from previously developed measures of life events (Bray & Hourani, Reference Bray and Hourani2007; Brugha & Cragg, 1990; Vogt, Rizvi, Shipherd, & Resick, Reference Vogt, Rizvi, Shipherd and Resick2008) assessed stressful events over the past 12-months.

Table 1. Pooled estimates of bivariate and multivariate associations between predictor variables and subsequent suicide plan or attempt

Note. ahigh education was defined as having a least a Bachelor's degree; w(n), weighted number of cases; w(%), weighted percentage of sample; OR, odds ratio; PARP, population attributable risk ratio; significant results (p < 0.05) are highlighted in bold.

Mental health treatment

At baseline, all participants were asked about lifetime and 12-month psychological counseling as well as use of medication for emotional or substance use problems (Bruffaerts et al., Reference Bruffaerts, Mortier, Auerbach, Alonso, Hermosillo De la Torre and Cuijpers2019). For participants identified by the algorithm, at 4-week follow-up, participants were provided with a 16-item checklist of common treatment options (e.g. general practitioner, psychologist, emergency department) and invited to indicate any resources (e.g. mental health websites) they had accessed in the past 4-weeks for mental health concerns, and (if applicable) suicidal thoughts or behaviors. These students were asked to indicate resources they had accessed specifically as a result of intervention. Treatment access was scored as a binary (accessed ⩾1 resources; did not access any resources). At 12-month follow-up, all participants were asked about both lifetime and 12-month counseling or medication for an emotional or substance use problem (Bruffaerts et al., Reference Bruffaerts, Mortier, Auerbach, Alonso, Hermosillo De la Torre and Cuijpers2019).

Intervention

Our prevention program (COMPAS-S) involved using the predictive algorithm, embedded within the WMH-ICS survey. This algorithm ran live while students completed the baseline survey and was used to identify students at increased risk of suicide plan and/or attempt in the next 12 months. A stepped-care approach was implemented. All students were provided with a comprehensive list of local and national services. Students who reported 12-month suicidal ideation were directed to a safety planning smartphone application (Melvin et al., Reference Melvin, Gresham, Beaton, Coles, Tonge, Gordon and Stanley2019). Students identified at future risk of suicide plan and/or attempt by the predictive risk algorithm (n = 184) and those reporting 12-month suicide attempt, but not identified by the algorithm (n = 19) were contacted via telephone by a mental health professional within 24-h of completing the survey (Intervention Group). This telehealth intervention focused on support and safety-planning (Melvin et al., Reference Melvin, Gresham, Beaton, Coles, Tonge, Gordon and Stanley2019), as well as personalized referral to appropriate services (e.g. emergency department, primary health care services, student support services). We worked with clinical psychology postgraduate trainees to conduct these calls, training them risk assessment, safety planning, available resources, and self-care (Hasking et al., Reference Hasking, Chiu, Robinson, Coleman and McEvoy2023). From 2020–2022 we implemented COMPAS-S, integrating the screening algorithm into the baseline survey. The protocol was approved by the (blinded for review) Human Research Ethics Committee, and all participants provided consent to participate.

Statistical analysis

Algorithm development

Our risk algorithm was developed on data from 1202 university students during their first year at college in either 2016 or 2017 (Development Cohort). Of these, 359 students (29.87%) provided data at 12-month follow-up. Missing data, including due to non-response at follow-up, were imputed using multiple imputation, generating 20 imputed datasets that were pooled for analyses. Non-propensity weights were calculated based on socio-demographic and university-related variables available for the entire first year cohorts.

Descriptive statistics are reported as weighed numbers and proportions with standard errors. We report bivariate logistic regression analyses with each of the 38 predictor variables assessed in the WMH-ICS at baseline, with the criterion variable being the 1-year follow-up survey data on a suicide plan or attempt in the past year. Next, we report multivariable analyses with all 38 variables in the model. Population level effect sizes were reported as population attributable risk factors (PARPs; Bruffaerts, Kessler, Demyttenaere, Bonnewyn, and Nock, Reference Bruffaerts, Kessler, Demyttenaere, Bonnewyn and Nock2015). Predicted probabilities were assessed against observed cases to assess sensitivity, specificity, and positive predictive values. Area under the curve estimates were calculated for the resulting model.

Retrospective cohort trial

Given high attrition, only complete cases were analyzed to test the effectiveness of COMPAS-S. Chi-squared tests compared rates of suicidal thoughts and behaviors and treatment access at baseline and 12-month follow-up by Cohort (Intervention v. Control) and Algorithm Outcome (At Risk v. Not At Risk). To evaluate associations with later suicide plan and/or attempt, we then retrospectively fit the algorithm to the 2018–2019 cohorts (Control Cohort). A hierarchical binary logistic regression tested the utility of the intervention with 12-month plans and/or attempts at follow-up as the criterion variable. Reports of lifetime and 12-month suicidal behavior at baseline were entered as covariates at Step 1. Algorithm outcome (At Risk v. Not At Risk) was entered at Step 2, cohort (Intervention v. Control) at Step 3, and the interaction between algorithm outcome and cohort condition at Step 4. A significant interaction was probed using simple slopes analysis (Aiken, West, & Reno, Reference Aiken, West and Reno1991). All analyses were conducted with SPSS v26.

Results

Algorithm development

We first explored the associations between each of our 38 predictor variables and subsequent suicidal behaviors in bivariate regression models. Almost all psychosocial variables were statistically significant, with medium to large effect sizes (Table 1). Across the 20 imputed datasets, the multivariable regression model accounted for an average of 60% (average Nagelkerke R 2 = 0.60, p < 0.001) of the variance in suicide plan amd/or attempt at follow-up. The multiple imputation resulted in reliable estimates for most predictors, with less reliable estimates for low incidence predictors (e.g. sexual abuse as a child, lifetime psychosis; Table 1). The mean relative efficiency was 0.968, suggesting an appropriate number of imputations for accurate estimates.

Predicted probability values were calculated from the regression equation for each participant. To find the optimum predicted probability cut-off point, the participants were organized into 20 ventile categories (i.e. sets of 5% of the sample) based on their predicted probability values. Examination of these ventiles highlighted that capturing the top 15% of highest risk participants according to the predicted probability values provided the best balance of true and false positives. Specifically, 76.86% of positive cases were above this cut-off, with a Positive Predictive Value of 53.06%. Extending the cut-off range beyond the top 15% highest risk predicted probability range (i.e. to the top 20% highest risk) resulted in a substantially increased false positive rate (online Supplementary Table S1; Fig. S2). As expected, rates of all predictor variables were significantly elevated among participants reporting suicidal behavior one year later (online Supplementary Table S2). Using this cut-off, area under the curve estimates across the imputed data sets ranged from 0.895 (95% CIs 0.872–0.917) to 0.966 (95% CIs 0.939–0.994; see online Supplementary Fig. S3).

Retrospective cohort trial

Baseline suicidal thoughts and behaviors

Validating the algorithm, in the 2018–2022 cohorts, 314 (7.4%) participants were identified by the algorithm as being at high risk of suicide plan and/or attempt in the upcoming 12-months, with similar proportions flagged within the Intervention (7.1%) and the Control cohorts (7.8%; χ2(1) = 0.79, p = 0.373, φ = −0.01). There were no demographic differences across Intervention and Control cohorts (online Supplementary Table S3).

Table 2 reports rates of suicide ideation, plan, and attempt, at baseline and 12-month follow-up, separated by Cohort (Intervention v. Control) and algorithm outcome (At Risk v. Not At Risk). Across both Intervention and Control cohorts, participants identified as at risk by the algorithm reported considerably higher rates of lifetime and 12-month suicide ideation, plans, and attempts than those not identified as at risk. Students in the Control cohort who were not identified as at risk reported higher rates of lifetime and 12-month suicide ideation and plans than those in the Intervention Cohort who were not identified as at risk. As such, past suicidal behaviors were statistically controlled in subsequent analyses. Suicide plans and/or attempt in the coming twelve months was our outcome variable in these analyses.

Table 2. Rates of suicide ideation, plan, attempt, at baseline and 12-month follow-up, separated by intervention cohort and algorithm outcome

Note. Asterisks signify significant within-cohort differences by flagged status: * p < 0.050, ** p < 0.001. Diamonds signify significant between-cohort differences within algorithm outcome: p < 0.050, ◊◊ p < 0.001.

Associations with outcome variables

Of students identified by the algorithm, 56.5% (n = 104) received the telehealth support intervention (32.1% did not reply, 7.1% declined the invitation, and 4.3% provided insufficient contact information; M call length = 29.38 min, s.d. = 17.53 min, range = 8–113 min). Of students we called, 23.1% were considered to be at acute suicide risk at the time of the call. Where appropriate, these students received crisis care and a follow-up call within 24 h of the initial assessment. Four weeks later, 55.8% (n = 58) of students who took part in the intervention received a follow-up call from the clinical team (41.3% did not reply and 2.9% declined the invitation; M call length = 21.78 min, s.d. = 14.92, range = 4–67 min), receiving further suicide risk assessment and updated personalized referrals to appropriate resources.

After accounting for lifetime and 12-month suicidal behavior at baseline, intervention condition significantly moderated the relationship between algorithm outcome and subsequent suicidal behavior (Table 3). Simple slopes analysis (see Fig. 1) demonstrated that within the Control Cohort, students identified as at risk by the algorithm were significantly more likely to subsequently report suicidal behavior at 12-month follow-up than were non-identified students (b = 1.15, 95% CI [0.39–1.92], z = 2.95, p = 0.003). In contrast, within the Intervention Cohort, identified students were not more likely than others to subsequently report suicide plans and/or attempt at 12-month follow-up (b = −0.09, 95% CI [−1.08 to 0.89], z = −0.18, p = 0.654), suggesting that COMPAS-S was associated with a 41.7% reduction in odds of subsequent suicidal behavior.

Table 3. Hierarchical logistic regression predicting future suicidal behavior (i.e. recent suicidal behavior at 12-month follow-up)

Note. Algorithm outcome: 1 – Not At Risk, 2 – At Risk. Intervention condition: 1 – Control Cohort, 2 – Intervention Cohort.

Figure 1. Intervention condition moderates the relationship between algorithm outcome and probability of recent suicidal behaviors at 12-month follow-up.

Treatment access

Consistent with the greater mental health need, students identified by the algorithm in both Intervention and Control cohorts were more likely to report having received lifetime, past-year, and current treatment than non-identified students at baseline (Table 4). There were no differences in lifetime, past-year, or current treatment at baseline among at-risk students in the Intervention and Control cohorts. Four-weeks following the telehealth intervention, at-risk students in the Intervention Cohorts were asked if they had accessed any mental health resources or treatment as a result of the intervention; 34.6% reported additional resource use, suggesting that students were taking up the personalized referrals provided as part of the intervention. At 12-month follow-up, at-risk students in both Intervention and Control cohorts reported similar rates of past-year treatment access and reported more access than students not at risk.

Table 4. Rates of treatment access at baseline and 12-month follow-up periods as well as mental health resource use at 4-week follow-up, separated by intervention cohort and algorithm outcome

Note. Asterisks note significant within-cohort differences: * p < 0.050, ** p < 0.001.

Discussion

Across the globe, college students continue to report high rates of suicidal thoughts and behaviors (Mortier et al., Reference Mortier, Auerbach, Alonso, Bantjes, Benjet, Cuijpers and Vives2018). In line with recent calls to develop multivariable predictive risk algorithms, rather than rely on risk assessments (Glenn & Nock, Reference Glenn and Nock2014; Riberio et al., Reference Riberio, Franklin, Fox, Bentley, Kleiman and Chang2016), we embedded an algorithm into the Australian WMH-ICS surveys to identify university students most at risk of suicidal behavior in the coming 12 months. COMPAS-S significantly outperformed existing suicide assessment tools. By allocating resources to the top 15% of students at risk, we can reach more than 50% of students who will subsequently report suicidal behavior. Echoing previous research in the field (Franklin et al., Reference Franklin, Ribeiro, Fox and Nock2017; Riberio et al., Reference Riberio, Franklin, Fox, Bentley, Kleiman and Chang2016), the accumulation of both distal and proximal factors worked together to increase risk. This underscores the need to reconsider risk assessments based primarily on examination of past suicidal behavior, and to instead consider a more comprehensive range of factors that may relate to suicide risk. Universal screening for suicide risk, coupled with targeted identification and referral of those most at risk, offers a viable approach to suicide prevention within universities, facilitating provision of appropriate support and services before a suicidal crisis is experienced.

COMPAS-S was successful in increasing short-term resource use among students identified at greatest suicide risk, although treatment access did not differ at 12-month follow-up. COMPAS-S was associated with a 41.7% reduction in the odds of suicidal behavior one year later. Although the majority of students identified were not at imminent risk of suicide at the time of the call, as the algorithm was not designed to assess acute risk, all students we contacted reported high levels of distress. This is likely a result of the variety of factors included in the algorithm (e.g. drug abuse; depression) that by themselves may not increase suicide risk but are still cause for concern and intervention.

Limitations and future directions

Although these results are encouraging, there are limitations that must be considered in interpreting our findings. First, poor response rates and significant attrition resulted in small samples and large confidence intervals around predictive accuracy for the algorithm itself. Our response rates were consistent with other WMH-ICS studies (e.g. Bruffaerts et al., Reference Bruffaerts, Mortier, Auerbach, Alonso, Hermosillo De la Torre and Cuijpers2019), and our R-indicators suggest the sample is socio-demographically representative of the student cohort. In addition, the algorithm was found in prospective validation to accurately identify students at heightened distress. However, replication with larger samples, with lower attrition rates will yield more precise estimates. Similarly, larger samples will be required to test mechanisms of action, including early access to treatment reducing associated mental health conditions such as depression, reduced barriers to treatment access (e.g. knowledge of available services), and access to university-specific resources (e.g. academic support plans). Second, our algorithm was developed for the Australian tertiary education context. Replication and adaptation for other countries and settings, considering local needs, is warranted.

Third, future work is required to assess who responds best to a telehealth intervention, and how outcomes might differ across different groups of students (e.g. domestic v. international; part time v. full time study). Finally, our algorithm development and the intervention phase overlapped with the worst of the COVID-19 pandemic. Fortunately, Western Australia, where these data were collected, did not experience significant lock down periods, school shutdowns, or restrictions in terms of isolations or mask mandates, seen in the rest of Australia at this time. In fact, students providing data during COVID were slightly less likely to report suicidal thoughts and behaviors at baseline, although the effect sizes were very small (online Supplementary Table S4). Still, it is possible that factors related to the pandemic may have affected uptake of COMPAS-S and retention rates. Relatedly, given the retrospective nature of the study we did not pre-register the study. Future trials of the effectiveness of COMPAS-S will be pre-registered to allow clear distinction between confirmatory and exploratory aspects of the research.

Conclusion

Despite these caveats, we have demonstrated the utility of embedding a multivariable predictive algorithm into a universal screening program to detect university students at greatest risk of subsequent suicidal behavior. We have also successfully used this algorithm to proactively contact students who are often reluctant to autonomously seek support, conduct support and safety planning, and link them in with appropriate services. Finally, the COMPAS-S approach has potential to expand to other community sectors. By developing a screening survey that appropriately captures risk factors among different sectors (e.g. shift work among hospital staff; combat tours among service men and women) we have the potential to screen and proactively support individuals and significantly reduce suicide rates.

Supplementary material

The supplementary material for this article can be found at https://doi.org/10.1017/S0033291723002714

Acknowledgements

We would like to acknowledge the numerous university staff, students, and mental health professionals who supported the implementation of this intervention, as well as the consumers who provided invaluable feedback on implantation of the algorithm and stepped referral processes.

Funding statement

Funding to support this project was provided from Suicide Prevention Australia and the Feilman Foundation (PH). MB is supported by the National Health and Medical Research Council (1173043); The World Mental Health International College Student project is carried out as part of the WHO World Mental Health (WMH) Survey Initiative. The WMH survey is supported by the National Institute of Mental Health NIMH R01MH070884, NIMH R56MH109566 (RPA), the John D. and Catherine T. MacArthur Foundation, the Pfizer Foundation, the US Public Health Service (R13-MH066849, R01-MH069864, and R01 DA016558), the Fogarty International Center (FIRCA R03-TW006481), the Pan American Health Organization, Eli Lilly and Company, Ortho-McNeil Pharmaceutical, GlaxoSmithKline, and Bristol-Myers Squibb; the King Baudouin Foundation (2014-J2140150-102905) (RB), and Eli Lilly (IIT-H6U-BX-I002) (RB). The content is solely the responsibility of the authors and does not necessarily represent the official views of any of the funders. A complete list of all WMH-ICS publications can be found at: http://www.hcp.med.harvard.edu/wmh/college_student_survey.php.

Competing interests

Dr Auerbach is an unpaid scientific advisor for Ksana Health, and he is a paid scientific advisor for Get Sonar, Inc. In the past 3 years, Dr Kessler was a consultant for Datastat, Inc., Holmusk, RallyPoint Networks, Inc., and Sage Pharmaceuticals. He has stock options in Mirah, PYM, and Roga Sciences.

Ethical standards

The authors assert that all procedures contributing to this work comply with the ethical standards of the relevant national and institutional committees on human experimentation and with the Helsinki Declaration of 1975, as revised in 2008.

References

Aiken, L. S., West, S. G., & Reno, R. R. (1991). Multiple regression: Testing and interpreting interactions. Newbury Park, CA: Sage.Google Scholar
Bray, R. M., & Hourani, L. L. (2007). Substance use trends among active duty military personnel: Findings from the United States department of defense health related behaviour surveys. Addiction, 102, 10921101.CrossRefGoogle Scholar
Bruffaerts, R., Demyttenaere, K., Hwang, I., Chiu, W. T., Sampson, N., Kessler, R. C., … Nock, M. K. (2011). Treatment of suicidal people around the world. The British Journal of Psychiatry, 199(1), 6470.CrossRefGoogle ScholarPubMed
Bruffaerts, R., Kessler, R. C., Demyttenaere, K., Bonnewyn, A., & Nock, M. K. (2015). Examination of the population attributable risk of different risk factor domains for suicidal thoughts and behaviors. Journal of Affective Disorders, 187, 6672.CrossRefGoogle ScholarPubMed
Bruffaerts, R., Mortier, P., Auerbach, R. P., Alonso, J., Hermosillo De la Torre, A. E., Cuijpers, P., … WHO WMH-ICS Collaborators. (2019). Lifetime and 12-month treatment for mental disorders and suicidal thoughts and behaviors among first year college students. International Journal of Methods in Psychiatric Research, 28(2), e1764.CrossRefGoogle ScholarPubMed
Bruffaerts, R., Mortier, P., Kiekens, G., Auerbach, R. P., Cuijpers, P., Demyttenaere, K., … Kessler, R. C. (2018). Mental health problems in college freshmen: Prevalence and academic functioning. Journal of Affective Disorders, 225, 97103.CrossRefGoogle ScholarPubMed
Carter, G., Milner, A., McGill, K., Pirkis, J., Kapur, N., & Spittal, M. J. (2017). Predicting suicidal behaviours using clinical instruments: Systematic review and meta-analysis of positive predictive values for risk scales. The British Journal of Psychiatry, 210(6), 387395.CrossRefGoogle ScholarPubMed
Cooper, J., Kapur, N., Dunning, J., Guthrie, E., & Appleby, L. (2006). Mackway-Jones K. A clinical tool for assessing risk after self-harm. Annals of Emergency Medicine, 48(4), 459466.CrossRefGoogle ScholarPubMed
Felitti, V. J., Anda, R. F., Nordenberg, D., Williamson, D. F., Spitz, A. M., Edwards, V., … Marks, J. S. (2019). Reprint of: Relationship of childhood abuse and household dysfunction to many of the leading causes of death in adults: The adverse childhood experiences (ACE) study. American Journal of Preventive Medicine, 56(6), 774786. doi: 10.1016/s0749-3797(98)00017-8CrossRefGoogle ScholarPubMed
Franklin, J. C., Ribeiro, J. D., Fox, K. R., & Nock, M. K. (2017). Risk factors for suicidal thoughts and behaviors: A meta-analysis of 50 years of research. Psychological Bulletin, 143(2), 187232. doi:10.1037/bul0000084CrossRefGoogle ScholarPubMed
Glenn, C. R., & Nock, M. K. (2014). Improving the short-term prediction of suicidal behaviour. American Journal of Preventative Medicine, 47(3S2), S176S180.CrossRefGoogle Scholar
Hasking, P., Chiu, V., Robinson, K., Coleman, A., & McEvoy, P. (2023). COMPAS-S Checking on mental health: Providing alternatives to suicide for students: Training manual for phone assessments. doi:10.5281/zenodo.7654149Google Scholar
Kessler, R. C., Abelson, J., Demler, O., Escobar, J. I., Gibbon, M., Guyer, M. E., … Zheng, H. (2004). Clinical calibration of DSM-IV diagnoses in the World Mental Health (WMH) version of the World Health Organization (WHO) Composite International Diagnostic Interview (WMH-CIDI). International Journal of Methods in Psychiatric Research, 13(2), 122139. doi:10.1002/mpr.168CrossRefGoogle ScholarPubMed
Kessler, R. C., Warner, C. H., Ivany, C., Petukhova, M. V., Rose, S., Bromet, E. J., … Army STARRS Collaborators. (2015). Predicting suicides after psychiatric hospitalization in US army soldiers: The army study to assess risk and resilience in servicemembers (Army STARRS). JAMA Psychiatry, 72(1), 4957.CrossRefGoogle ScholarPubMed
Kiekens, G., Claes, L., Hasking, P., Mortier, P., Bootsma, E., Boyes, M., … Bruffaerts, R. (2022). A longitudinal investigation of non-suicidal self-injury persistence patterns, risk factors, and clinical outcomes during the college period. Psychological Medicine, 116.Google ScholarPubMed
Large, M., Kaneson, M., Myles, N., Myles, H., Gunaratne, P., & Ryan, C. (2016). Meta-analysis of longitudinal cohort studies of suicide risk assessment among psychiatric patients: Heterogeneity in results and lack of improvement over time. PloS one, 11(6), e0156322.CrossRefGoogle ScholarPubMed
Melvin, G. A., Gresham, D., Beaton, S., Coles, J., Tonge, B. J., Gordon, M. S., & Stanley, B. (2019). Evaluating the feasibility and effectiveness of an Australian safety planning smartphone application: A pilot study within a tertiary mental health service. Suicide and Life-Threatening Behavior, 49(3), 846858.CrossRefGoogle ScholarPubMed
Miché, M., Studerus, E., Meyer, A. H., Gloster, A. T., Beesdo-Baum, K., Wittchen, H. U., & Lieb, R. (2020). Prospective prediction of suicide attempts in community adolescents and young adults, using regression methods and machine learning. Journal of Affective Disorders, 265, 570578.CrossRefGoogle Scholar
Mortier, P., Auerbach, R. P., Alonso, J., Bantjes, J., Benjet, C., Cuijpers, P., … Vives, M. (2018). Suicidal thoughts and behaviors among first-year college students: Results from the WMH-ICS project. Journal of the American Academy of Child & Adolescent Psychiatry, 57(4), 263273.CrossRefGoogle ScholarPubMed
Mortier, P., Demyttenaere, K., Auerbach, R. P., Cuijpers, P., Green, J. G., Kiekens, G., … Bruffaerts, R. (2017a). First onset of suicidal thoughts and behaviours in college. Journal of Affective Disorders, 207, 291299.CrossRefGoogle Scholar
Mortier, P., Kiekens, G., Auerbach, R. P., Cuijpers, P., Demyttenaere, K., Green, J. G., … Bruffaerts, R. (2017b). A risk algorithm for the persistence of suicidal thoughts and behaviors during college. The Journal of Clinical Psychiatry, 78(7), 20347.CrossRefGoogle ScholarPubMed
Nock, M. K., Holmberg, E. B., Photos, V. I., & Michel, B. D. (2007). Self-injurious thoughts and behaviors interview: Development, reliability, and validity in an adolescent sample. Psychological Assessment, 19, 309317.CrossRefGoogle Scholar
OECD. (2021). Who is expected to enter tertiary education? In Education at a glance 2021: OECD indicators. OECD, pp. 188199. doi:10.1787/b35a14e5-enGoogle Scholar
Quinlivan, L., Cooper, J., Davies, L., Hawton, K., Gunnell, D., & Kapur, N. (2016). Which are the most useful scales for predicting repeat self-harm? A systematic review evaluating risk scales using measures of diagnostic accuracy. BMJ Open, 6, e009297. doi:10.1136/bmjopen-2015-009297CrossRefGoogle ScholarPubMed
Riberio, J. D., Franklin, J. C., Fox, K. R., Bentley, K. H., Kleiman, E. M., & Chang, B. P. (2016). Self-injurious thoughts and behaviors as risk factors for future suicide ideation, attempts, and death: A mental-analysis of longitudinal stuidies. Psychological Medicine, 46, 225236.CrossRefGoogle Scholar
Steeg, S., Kapur, N., Webb, R., Applegate, E., Stewart, S. L. K., Hawton, K., … Cooper, J. (2012). The development of a population-level clinical screening tool for self-harm repetition and suicide: The ReACT self-harm rule. Psychological Medicine, 42(11), 23832394.CrossRefGoogle ScholarPubMed
Vogt, D. S., Rizvi, S. L., Shipherd, J. C., & Resick, P. A. (2008). Longitudinal investigation of reciprocal relationship between stress reactions and hardiness. Personality and Social Psychology Bulletin, 34(1), 6173.CrossRefGoogle ScholarPubMed
World Health Organization. (2014). Preventing suicide: A global imperative. World Health Organization.Google Scholar
World Health Organization. (2021). Suicide worldwide in 2019: Global health estimates. World Health Organisation. https://www.who.int/publications/i/item/9789240026643 (Accessed February 1, 2023).Google Scholar
Figure 0

Table 1. Pooled estimates of bivariate and multivariate associations between predictor variables and subsequent suicide plan or attempt

Figure 1

Table 2. Rates of suicide ideation, plan, attempt, at baseline and 12-month follow-up, separated by intervention cohort and algorithm outcome

Figure 2

Table 3. Hierarchical logistic regression predicting future suicidal behavior (i.e. recent suicidal behavior at 12-month follow-up)

Figure 3

Figure 1. Intervention condition moderates the relationship between algorithm outcome and probability of recent suicidal behaviors at 12-month follow-up.

Figure 4

Table 4. Rates of treatment access at baseline and 12-month follow-up periods as well as mental health resource use at 4-week follow-up, separated by intervention cohort and algorithm outcome

Supplementary material: File

Hasking et al. supplementary material 1

Hasking et al. supplementary material
Download Hasking et al. supplementary material 1(File)
File 24.5 KB
Supplementary material: File

Hasking et al. supplementary material 2

Hasking et al. supplementary material
Download Hasking et al. supplementary material 2(File)
File 32.5 KB
Supplementary material: File

Hasking et al. supplementary material 3

Hasking et al. supplementary material
Download Hasking et al. supplementary material 3(File)
File 23.6 KB
Supplementary material: File

Hasking et al. supplementary material 4

Hasking et al. supplementary material
Download Hasking et al. supplementary material 4(File)
File 36 KB