Skip to main content Accessibility help
×
Home
Hostname: page-component-684bc48f8b-g7stk Total loading time: 1.189 Render date: 2021-04-11T17:15:21.397Z Has data issue: true Feature Flags: { "shouldUseShareProductTool": true, "shouldUseHypothesis": true, "isUnsiloEnabled": true, "metricsAbstractViews": false, "figures": false, "newCiteModal": false, "newCitedByModal": true }

Prediction of psychosis: model development and internal validation of a personalized risk calculator

Published online by Cambridge University Press:  14 December 2020

Tae Young Lee
Affiliation:
Department of Psychiatry, Seoul National University College of Medicine, Seoul, Republic of Korea Department of Psychiatry, Pusan National University Yangsan Hospital, Yangsan, Republic of Korea
Wu Jeong Hwang
Affiliation:
Department of Brain and Cognitive Neuroscience, Seoul National University College of Natural Sciences, Seoul, Republic of Korea Center for Neuroscience Imaging Research, Institute for Basic Science, Suwon, Republic of Korea
Nahrie S. Kim
Affiliation:
Department of Psychiatry, Pusan National University Yangsan Hospital, Yangsan, Republic of Korea Department of Brain and Cognitive Neuroscience, Seoul National University College of Natural Sciences, Seoul, Republic of Korea Center for Neuroscience Imaging Research, Institute for Basic Science, Suwon, Republic of Korea
Inkyung Park
Affiliation:
Department of Brain and Cognitive Neuroscience, Seoul National University College of Natural Sciences, Seoul, Republic of Korea Center for Neuroscience Imaging Research, Institute for Basic Science, Suwon, Republic of Korea
Silvia Kyungjin Lho
Affiliation:
Department of Psychiatry, Seoul National University College of Medicine, Seoul, Republic of Korea
Sun-Young Moon
Affiliation:
Department of Psychiatry, Seoul National University College of Medicine, Seoul, Republic of Korea
Sanghoon Oh
Affiliation:
Department of Psychiatry, Seoul National University College of Medicine, Seoul, Republic of Korea
Junhee Lee
Affiliation:
Department of Psychiatry, Seoul National University College of Medicine, Seoul, Republic of Korea
Minah Kim
Affiliation:
Department of Psychiatry, Seoul National University College of Medicine, Seoul, Republic of Korea
Choong-Wan Woo
Affiliation:
Department of Brain and Cognitive Neuroscience, Seoul National University College of Natural Sciences, Seoul, Republic of Korea Center for Neuroscience Imaging Research, Institute for Basic Science, Suwon, Republic of Korea Department of Biomedical Engineering, Sungkyunkwan University, Suwon, Republic of Korea
Jun Soo Kwon
Affiliation:
Department of Psychiatry, Seoul National University College of Medicine, Seoul, Republic of Korea Department of Brain and Cognitive Neuroscience, Seoul National University College of Natural Sciences, Seoul, Republic of Korea Center for Neuroscience Imaging Research, Institute for Basic Science, Suwon, Republic of Korea
Corresponding
E-mail address:
Rights & Permissions[Opens in a new window]

Abstract

Background

Over the past two decades, early detection and early intervention in psychosis have become essential goals of psychiatry. However, clinical impressions are insufficient for predicting psychosis outcomes in clinical high-risk (CHR) individuals; a more rigorous and objective model is needed. This study aims to develop and internally validate a model for predicting the transition to psychosis within 10 years.

Methods

Two hundred and eight help-seeking individuals who fulfilled the CHR criteria were enrolled from the prospective, naturalistic cohort program for CHR at the Seoul Youth Clinic (SYC). The least absolute shrinkage and selection operator (LASSO)-penalized Cox regression was used to develop a predictive model for a psychotic transition. We performed k-means clustering and survival analysis to stratify the risk of psychosis.

Results

The predictive model, which includes clinical and cognitive variables, identified the following six baseline variables as important predictors: 1-year percentage decrease in the Global Assessment of Functioning score, IQ, California Verbal Learning Test score, Strange Stories test score, and scores in two domains of the Social Functioning Scale. The predictive model showed a cross-validated Harrell's C-index of 0.78 and identified three subclusters with significantly different risk levels.

Conclusions

Overall, our predictive model showed a predictive ability and could facilitate a personalized therapeutic approach to different risks in high-risk individuals.

Type
Original Article
Creative Commons
Creative Common License - CCCreative Common License - BY
This is an Open Access article, distributed under the terms of the Creative Commons Attribution licence (http://creativecommons.org/licenses/by/4.0), which permits unrestricted re- use, distribution and reproduction, provided the original article is properly cited.
Copyright
Copyright © The Author(s), 2020. Published by Cambridge University Press

Introduction

Over the past two decades, early detection and early intervention in psychosis have become essential goals of psychiatry (Birchwood, Todd, & Jackson, Reference Birchwood, Todd and Jackson1998; McGlashan & Johannessen, Reference McGlashan and Johannessen1996; McGorry, Killackey, & Yung, Reference McGorry, Killackey and Yung2008). Only a small proportion of cases undergo the transition to psychosis, and the process often takes place over very long periods of time; these circumstances are among the main reasons why the focus of high-risk studies has shifted from the traditional or genetic high-risk model to the clinical high-risk (CHR) model. The CHR concept has emerged to describe cases that are likely to progress to psychosis soon (Yung et al., Reference Yung, McGorry, McFarlane, Jackson, Patton and Rakkar1996). A number of prospective cohort programs have been introduced for help-seeking people who have not yet developed psychosis (Cannon et al., Reference Cannon, Cadenhead, Cornblatt, Woods, Addington, Walker and Heinssen2008; Nelson et al., Reference Nelson, Yuen, Wood, Lin, Spiliotacopoulos, Bruxner and Yung2013; Riecher-Rossler et al., Reference Riecher-Rossler, Gschwandtner, Aston, Borgwardt, Drewe, Fuhr and Stieglitz2007; Ruhrmann et al., Reference Ruhrmann, Schultze-Lutter, Salokangas, Heinimaa, Linszen, Dingemans and Klosterkotter2010); additionally, various terms, such as at-risk mental state and ultra-high-risk, in addition to CHR for psychosis, and basic symptoms have been established to describe this population (Fusar-Poli et al., Reference Fusar-Poli, Borgwardt, Bechdolf, Addington, Riecher-Rossler, Schultze-Lutter and Yung2013; Schultze-Lutter, Schimmelmann, Ruhrmann, & Michel, Reference Schultze-Lutter, Schimmelmann, Ruhrmann and Michel2013; Yung, Fusar-Poli, & Nelson, Reference Yung, Fusar-Poli and Nelson2012). However, the incidence rate of psychosis in a CHR individual decreases over time (Yung et al., Reference Yung, Yuen, Berger, Francey, Hung, Nelson and McGorry2007). Only one-fourth of CHR patients develop psychosis within 3 years (Fusar-Poli et al., Reference Fusar-Poli, Bonoldi, Yung, Borgwardt, Kempton, Valmaggia and McGuire2012, Reference Fusar-Poli, Cappucciati, Rutigliano, Schultze-Lutter, Bonoldi, Borgwardt and McGuire2015), and there are even studies that report an incidence rate of <15% (Addington et al., Reference Addington, Epstein, Liu, French, Boydell and Zipursky2011b; Katsura et al., Reference Katsura, Ohmuro, Obara, Kikuchi, Ito, Miyakoshi and Matsumoto2014; Koike et al., Reference Koike, Takano, Iwashiro, Satomura, Suga, Nagai and Kasai2013; Pruessner et al., Reference Pruessner, Faridi, Shah, Rabinovitch, Iyer, Abadi and Malla2017), although this risk state is not pluripotential but is specific to psychosis (Webb et al., Reference Webb, Addington, Perkins, Bearden, Cadenhead, Cannon and Woods2015; Woods et al., Reference Woods, Powers, Taylor, Davidson, Johannesen, Addington and McGlashan2018). Moreover, one-third of CHR patients remit from the risk state (Simon et al., Reference Simon, Borgwardt, Riecher-Rossler, Velthorst, de Haan and Fusar-Poli2013), and they showed no cognitive impairment or have good functional outcomes (Glenthoj, Kristensen, Wenneberg, Hjorthoj, & Nordentoft, Reference Glenthoj, Kristensen, Wenneberg, Hjorthoj and Nordentoft2020; Lee et al., Reference Lee, Shin, Shin, Kim, Jang, Kang and Kwon2014b). On the other hand, other non-converters also have attenuated psychotic symptoms or low levels of functioning even if they do not develop psychosis (Addington et al., Reference Addington, Cornblatt, Cadenhead, Cannon, McGlashan, Perkins and Heinssen2011a; Lee et al., Reference Lee, Kim, Correll, Byun, Kim, Jang and Kwon2014a; Lin et al., Reference Lin, Wood, Nelson, Beavan, McGorry and Yung2015). Thus, the CHR state is a heterogeneous clinical syndrome, only a small percentage is converted to psychosis and also develops other conditions than psychosis, and for this reason, the dilution of the pretest risk of psychosis due to intensive, predominantly general-population-oriented outreach campaigns and a high rate of self-referrals has been discussed (Fusar-Poli, Schultze-Lutter, & Addington, Reference Fusar-Poli, Schultze-Lutter and Addington2016b; Fusar-Poli et al., Reference Fusar-Poli, Schultze-Lutter, Cappucciati, Rutigliano, Bonoldi, Stahl and McGuire2016c; Mitter, Nah, Bong, Lee, & Chong, Reference Mitter, Nah, Bong, Lee and Chong2014). Therefore, the ability to identify true-positive patients who will later develop psychosis can immensely broaden our understanding of the pathophysiology of the long-term course of schizophrenia, and it will deepen the phenomenological, biological, and causal understanding of schizophrenia (Bentall, Jackson, & Pilgrim, Reference Bentall, Jackson and Pilgrim1988; Fusar-Poli & Schultze-Lutter, Reference Fusar-Poli and Schultze-Lutter2016; Guloksuz & van Os, Reference Guloksuz and van Os2018).

Predictive medicine is a discipline that entails predicting the probability of a disease's incidence or prognosticating its course, thus reducing the uncertainty in clinical decision making (Fusar-Poli, Hijazi, Stahl, & Steyerberg, Reference Fusar-Poli, Hijazi, Stahl and Steyerberg2018; Steyerberg, Reference Steyerberg2008; Wasson, Sox, Neff, & Goldman, Reference Wasson, Sox, Neff and Goldman1985). In contrast to classical statistics, predictive modeling has high clinical utility in that it not only indicates the average characteristics of the patient's group but also provides rich information about the onset or trajectory of illness at the individual level (Braitman & Davidoff, Reference Braitman and Davidoff1996; Hahn, Nierenberg, & Whitfield-Gabrieli, Reference Hahn, Nierenberg and Whitfield-Gabrieli2017; Lee, Bang, & Kim, Reference Lee, Bang and Kim2016). Furthermore, with the very high levels of clinical heterogeneity arising from phenotype-based diagnosis, the application of a clinical prediction model to real-world situations would facilitate better decision making in psychiatry. However, clinical impressions are insufficient for predicting psychosis outcome in CHR cases (Nelson & Yung, Reference Nelson and Yung2010); more rigorous and quantitative prediction models are needed. Thus, risk stratification has been applied in CHR studies (Fusar-Poli et al., Reference Fusar-Poli, Cappucciati, Borgwardt, Woods, Addington, Nelson and McGuire2016a, Reference Fusar-Poli, Rutigliano, Stahl, Schmidt, Ramella-Cravaro, Hitesh and McGuire2016b; Nieman et al., Reference Nieman, Ruhrmann, Dragt, Soen, van Tricht, Koelman and de Haan2014; Ruhrmann et al., Reference Ruhrmann, Schultze-Lutter, Salokangas, Heinimaa, Linszen, Dingemans and Klosterkotter2010; Schmidt et al., Reference Schmidt, Cappucciati, Radua, Rutigliano, Rocchetti, Dell'Osso and Fusar-Poli2017), this is similar to the model applied in heart failure and stroke (Goldman et al., Reference Goldman, Cook, Johnson, Brand, Rouan and Lee1996; Janes, Pepe, & Gu, Reference Janes, Pepe and Gu2008; Lip, Nieuwlaat, Pisters, Lane, & Crijns, Reference Lip, Nieuwlaat, Pisters, Lane and Crijns2010). Moreover, a number of prediction studies have been conducted (Addington et al., Reference Addington, Farris, Stowkowy, Santesteban-Echarri, Metzak and Kalathil2019; Oliver et al., Reference Oliver, Reilly, Baccaredda Boy, Petros, Davies, Borgwardt and Fusar-Poli2020; Studerus, Ramyead, & Riecher-Rossler, Reference Studerus, Ramyead and Riecher-Rossler2017), and the least absolute shrinkage and selection operator (LASSO) model, which incorporates machine learning techniques, has been introduced into such studies (Addington et al., Reference Addington, Liu, Perkins, Carrion, Keefe and Woods2017; Ciarleglio et al., Reference Ciarleglio, Brucato, Masucci, Altschuler, Colibazzi, Corcoran and Girgis2019) to overcome the problem of overfitting (Koutsouleris, Upthegrove, & Wood, Reference Koutsouleris, Upthegrove and Wood2019; Nelson, Yung, & McGorry, Reference Nelson, Yung and McGorry2019; Tibshirani, Reference Tibshirani1996; Tibshirani, Reference Tibshirani1997). More recently, several web-based personalized risk calculators have been developed (e.g. riskcalc.org/napls, psychosis-risk.net, link.konsta.com.pl/psychosis) (Cannon et al., Reference Cannon, Yu, Addington, Bearden, Cadenhead, Cornblatt and Kattan2016; Fusar-Poli et al., Reference Fusar-Poli, Rutigliano, Stahl, Davies, Bonoldi, Reilly and McGuire2017; Kotlicka-Antczak et al., Reference Kotlicka-Antczak, Karbownik, Stawiski, Pawelczyk, Zurner, Pawelczyk and Fusar-Poli2019), and their scope of use is being expanded through internal and external validation (Carrion et al., Reference Carrion, Cornblatt, Burton, Tso, Auther, Adelsheim and McFarlane2016; Fusar-Poli et al., Reference Fusar-Poli, Werbeloff, Rutigliano, Oliver, Davies, Stahl and Osborn2019b; Osborne & Mittal, Reference Osborne and Mittal2019; Zhang et al., Reference Zhang, Xu, Tang, Li, Tang, Cui and Group2019). These individual-level risk measures can be applied in real-world clinical practice to quantify the risks that patients may face and to develop appropriate treatment strategies. However, several additional considerations must be addressed to increase the predictive accuracy of personalized risk prediction models. First, to increase the accuracy of prediction, a modeling process based on longer-term follow-up is required to reduce false negatives that have not yet been developed due to insufficient observation period. Studies with a follow-up period of <3 years tend to have an incidence of less than one-quarter (Addington et al., Reference Addington, Epstein, Liu, French, Boydell and Zipursky2011b; Katsura et al., Reference Katsura, Ohmuro, Obara, Kikuchi, Ito, Miyakoshi and Matsumoto2014; Koike et al., Reference Koike, Takano, Iwashiro, Satomura, Suga, Nagai and Kasai2013; Morrison et al., Reference Morrison, French, Parker, Roberts, Stevens, Bentall and Lewis2007; Pruessner et al., Reference Pruessner, Faridi, Shah, Rabinovitch, Iyer, Abadi and Malla2017; van der Gaag et al., Reference van der Gaag, Nieman, Rietdijk, Dragt, Ising, Klaassen and Linszen2012), whereas those of more than 3 years tend to have an incidence of more than one-quarter (Nelson et al., Reference Nelson, Yuen, Wood, Lin, Spiliotacopoulos, Bruxner and Yung2013; Schultze-Lutter, Klosterkotter, & Ruhrmann, Reference Schultze-Lutter, Klosterkotter and Ruhrmann2014; Spitz et al., Reference Spitz, Studerus, Koranyi, Rapp, Ramyead, Ittig and Riecher-Rossler2017; Ziermans et al., Reference Ziermans, de Wit, Schothorst, Sprong, van Engeland, Kahn and Durston2014), with some exceptions (An et al., Reference An, Kang, Park, Kim, Lee and Lee2010; Armando et al., Reference Armando, Pontillo, De Crescenzo, Mazzone, Monducci, Lo Cascio and Schultze-Lutter2015), although meta-analytic results indicated that the incidence of psychosis in high-risk groups peaks within the first 2 years after entry (Kempton, Bonoldi, Valmaggia, McGuire, & Fusar-Poli, Reference Kempton, Bonoldi, Valmaggia, McGuire and -Poli2015). A sufficient period of observation will reveal the results of false negatives who have not yet developed and enable more accurate modeling. Second, CHR samples have different risks for psychosis, depending on their referral sources. For example, samples such as those recruited from the community via the Prodromal Questionnaire–brief form and other self-report measures, as well as patients who visit a primary clinic with non-prodromal symptoms and are suspected of having attenuated psychotic symptoms, would have different pretest risk levels than help-seeking individuals visiting a secondary or specialized clinic for CHR patients. This suggests that the diversification of referral sources and the rise of intensive outreach activity are major contributors to the increased variance of pretest risk enrichment in CHR samples and that the risk of developing psychosis is becoming more heterogeneous. Given this combination of factors, samples recruited by a fixed strategy at a single institution may allow greatly improved models to be developed if external validation can be conducted.

The primary goal of this study was to develop a model for predicting the 10-year risk of psychotic transition in patients who visited the Seoul Youth Clinic (SYC) at Seoul National University Hospital. A second goal was to perform internal validation of the resulting prediction model and stratify the CHR sample by risk level.

Methods

Participants

Two hundred and twenty-two help-seeking CHR individuals (aged 15–35 at baseline) were enrolled in the prospective, naturalistic cohort program for prodromal psychosis at the SYC between November 2004 and November 2019 (Kim et al., Reference Kim, Jang, Park, Shim, Hwang, Kim and Kwon2012; Kwon, Byun, Lee, & An, Reference Kwon, Byun, Lee and An2012; Lee et al., Reference Lee, Kim, Correll, Byun, Kim, Jang and Kwon2014a). All participants were recruited mainly from a psychiatric outpatient clinic in the Seoul National University Hospital and made initial contact with the SYC via the website (http://www.youthclinic.org). All subjects were diagnosed with the Korean version of the Structural Interview for Prodromal Syndrome (SIPS) and belonged to at least one of CHR groups: attenuated positive symptoms (APS), brief intermittent psychotic symptoms (BIPS), and genetic risk with deterioration (GRD) (Jung et al., Reference Jung, Jang, Kang, Choi, Shin, Kim and Kwon2010). The exclusion criteria for all subjects were as follows: any current or lifetime Axis I psychotic disorder or substance dependence other than cigarette smoking, past or present neurological disease or traumatic brain injury with loss of consciousness, a significant medical condition that could manifest as a psychiatric condition, past or current use of antipsychotic medications to manage attenuated psychotic symptoms, and/or an intelligence quotient (IQ) below 70. Clinical and cognitive function assessment was conducted by psychiatrists and clinical psychologists at 6-month intervals for the first 2 years and at 1-year intervals afterward for up to 10 years (online Supplementary Table S1). All subjects received case management and supportive psychotherapy monthly from psychiatrists. If the development of psychosis was suspected during these treatment sessions, the subjects also received an additional assessment session within a week, even during the regular evaluation period. The transition to psychosis was determined if the subject met the Presence Of Psychotic Syndrome (POPS) criteria of SIPS (Jung et al., Reference Jung, Jang, Kang, Choi, Shin, Kim and Kwon2010). Of the overall sample, 14 cases were excluded due to withdrawal; the final sample consisted of 208 CHR individuals. Written informed consent forms were obtained from all subjects, and if they were under the age of 18 years, guardian consent was also obtained. The present study was approved by the Institutional Review Board of the Seoul National University Hospital.

Clinical and cognitive assessments

Depressive and anxious symptoms were evaluated using the Hamilton Depression Rating Scale (HAM-D) (Yi et al., Reference Yi, Bae, Ahn, Park, Noh, Shin and Kim2005) and the Hamilton Anxiety Rating Scale (HAM-A) (Hamilton, Reference Hamilton1959), respectively. The Global Assessment of Functioning (GAF) was administered to assess the current levels of overall symptoms and functioning (Yi, Chung, Lee, & Lee, Reference Yi, Chung, Lee and Lee2003), and the percentage drop over the past year was calculated. Social functioning was also assessed using the Korean version of the Social Functioning Scale (SFS), which consists of seven domains of social behavior: social engagement/withdrawal, interpersonal behavior, independence – performance, independence – competence, recreation, prosocial activities, and employment/occupation (Kim & Lee, Reference Kim and Lee2009).

Each subject's IQ was estimated using the Korean version of the Wechsler Adult Intelligence Scale-III (WAIS) (Yeom, Park, Oh, & Lee, Reference Yeom, Park, Oh and Lee1992). We implemented four subsets of WAIS consisting of the Vocabulary, Arithmetic, Block Design, and Picture Arrangement, and assessment was performed by a trained researcher who majored in psychology with a master's degree or higher, or by a skilled clinical psychologist. The following neuropsychological tests were administered to assess cognitive function in the CHR population: Digit span, a subset of the WAIS, was used to measure attention/working memory. Processing speed was assessed with the Trail Making Test Part A (TMT-A) (Reitan, Reference Reitan1958). Measures of divided attention were assessed with the Trail Making Test Part B (TMT-B) (Reitan, Reference Reitan1958) and set-shifting from perseverative errors in the Wisconsin Card Sorting Test (WCST) (Chelune & Baer, Reference Chelune and Baer1986). Verbal fluency was evaluated with the Korean version of the verbal fluency task for semantic fluency (Kim et al., Reference Kim, Lee, Oh, Hong, Lee, Son and Kim2013) and the Controlled Oral Word Association Test (COWAT) for phonemic fluency (Kang, Chin, Na, Lee, & Park, Reference Kang, Chin, Na, Lee and Park2000). Verbal memory was assessed with the Korean version of the California Verbal Learning Test (K-CVLT), for which we examined the sum of the immediate and delayed scores (Kim & Kang, Reference Kim and Kang1999). The sum of the immediate and delayed scores on the Rey–Osterrieth Complex Figure Test (ROCF) was used to evaluate visual memory (Shin, Park, Park, Seol, & Kwon, Reference Shin, Park, Park, Seol and Kwon2006).

Statistical analysis

All analyses were conducted in Stata version 16 (Stata Corp.) and R version 3.6.0. (Comprehensive R Archive Network). Comparisons of baseline demographic and clinical characteristics were performed with χ2 tests and independent t tests. The cumulative incidence rates of transition to psychosis during the follow-up period were estimated with the Kaplan–Meier analysis. In developing the predictive model for psychosis, the LASSO-penalized Cox regression was used to developing the predictive model with subjects who had different follow-up periods (Simon, Friedman, Hastie, & Tibshirani, Reference Simon, Friedman, Hastie and Tibshirani2011). Predictors included a total of 56 candidate variables, including base demographic, clinical, and cognitive variables. Missing data were handled using the multiple imputation method with k = 10. We selected the LASSO model that resulted in minimal prediction error using 10-fold cross-validation. We then conducted a bootstrap test with 1000 iterations to estimate the 95% confidence interval of the predictive performance. For internal validation, a predictive individual prognostic index (PI) was generated, and k-means clustering analysis was performed to stratify the risk of transition to psychosis using the LASSO model with the elbow method to determine the optimal k. Kaplan–Meier analysis was performed to estimate the incidence of psychosis in each cluster. Then, the log-rank test was used to determine the different survival functions.

Results

The total SYC sample consisted of 208 CHR participants. The follow-up time ranged from a minimum of 30 days to a maximum of 12 years (online Supplementary Table S1). The mean follow-up duration was 3.5 years (s.d. 2.6 years). Thirty-eight participants developed a psychotic disorder during the follow-up period. Table 1 shows a Kaplan–Meier estimate of the survival function for the time to transition to psychotic disorders. The total cumulative incidence rate of transition was 32.6% (95% CI 21.8–46.9). Table 2 presents the baseline demographic and clinical characteristics of the participants. There was no significant difference between the groups in demographic status. Converters had higher positive symptom scale scores on the Scale of Prodromal Symptoms (SOPS), a percentage of changes in GAF scores in 1 year, and lower estimated current IQ, CVLT, and Strange Stories task scores. The social engagement/withdrawal and prosocial domains of the SFS showed trend-level differences between groups.

Table 1. Kaplan–Meier estimates of transition rates over 10 years

Table 2. Demographic and clinical characteristics of the participants

BIPS, Brief Intermittent Psychotic Syndrome; APS, Attenuated Psychosis Symptoms Syndrome; GRD, Genetic Risk and Deterioration Syndrome; SOPS, Scale of Prodromal Symptoms; HAM-D, Hamilton Depression Rating Scale; HAM-A, Hamilton Anxiety Rating Scale; GAF, Global Assessment of Functioning; SFS, Social Functioning Scale; CVLT, California Verbal Learning Test; ROCF, Rey–Osterrieth Complex Figure Test; TMT, Trail Making Test; COWAT, Controlled Oral Word Association Test; WCST, Wisconsin Card Sorting Test.

a Scores of 1–3 indicate high status and scores of 4–5 indicate low status.

In the SYC sample, 7.4% of the data overall were missing. The ROCF scores had the highest amount of missing data, with 24.5% of values missing. Category verbal fluency and COWAT scores had 21.2% of values missing. The LASSO Cox prediction model identified six baseline variables, including a percentage of changes in GAF scores in 1 year, IQ, CVLT, Strange Stories task, and two domains of the SFS (Table 3). The prediction model had a cross-validated Harrell's C-index of 0.78 (95% CI 0.777–0.783). The optimal number of clusters for the stratification of risk using the elbow method was 3. Figure 1 shows a Kaplan–Meier estimate of the survival function for the time to transition to psychotic disorders for each cluster. The cut-off values for each cluster were −1.86 for cluster 1 and −2.33 for cluster 3. The 10-year survival rate for each cluster was 15.9% for cluster 1 (n = 36), and 72.1% for cluster 2 (n = 109), 89.6% for cluster 3 (n = 99), and the clusters showed a significant difference in survival rate (cluster 1 v. 2: χ2 = 37.06, p < 0.001, cluster 2 v. 3: χ2 = 9.1, p = 0.003). Patients in cluster 1 showed an approximately 44% risk of developing psychosis at 3 years, whereas none of the patients in cluster 3 developed psychosis before 3 years (Table 4; online Supplementary Table S2).

Fig. 1. Kaplan–Meier survival estimates for the three clusters. Cluster 1 is a high-risk subgroup with 84.1% incidence (n = 36), cluster 2 is a medium-risk subgroup with 27.9% incidence (n = 109), and cluster 3 is a low-risk subgroup with 10.4% incidence within 10-year follow-up.

Table 3. The baseline variables identified by the LASSO Cox model that significantly predicted transition to psychosis

a California Verbal Learning Test.

b Global Assessment of Functioning.

c Social Functioning Scale.

Table 4. Kaplan–Meier estimates of transition rates in three clusters

Discussion

This study aimed to develop and internally validate a model for predicting the incidence of psychosis in CHR individuals to provide useful assistance in clinical practice. We developed a model that includes social functioning, social cognition, functional decline, verbal memory, and IQ; this model demonstrated fair predictive ability. Using this model, we divided the high-risk groups into three clusters, all of which showed significant differences in the incidence of psychosis. To the best of our knowledge, this is the first study using modern machine learning techniques to model a wide range of variables covering demographic, clinical, and cognitive functions in long-term cohorts spanning more than 10 years.

In this study, approximately 32% of subjects developed to psychosis during up to 10 years of follow-up. This transition rate is somewhat higher incidence compared to the results of a meta-analysis with an incidence rate of 20% (Fusar-Poli et al., Reference Fusar-Poli, Cappucciati, Borgwardt, Woods, Addington, Nelson and McGuire2016a). Although half of the total incidence occurs within the first 8 months of the 2-year follow-up period (Fusar-Poli et al., Reference Fusar-Poli, Cappucciati, Borgwardt, Woods, Addington, Nelson and McGuire2016a), in a long-term follow-up result of Nelson et al, 17% of cases are converted during the follow-up period after 3 years, and 5% are converted after 5 years (Nelson et al., Reference Nelson, Yuen, Wood, Lin, Spiliotacopoulos, Bruxner and Yung2013). This is in line with the result of our cohort, which showed a conversion rate of 21% after 3 years and 10% after 5 years. In our predictive model with a long-term follow-up duration, the estimated predictive ability had a C-index of 0.78, which is comparable to the results of other previous studies with similar designs (Addington et al., Reference Addington, Liu, Perkins, Carrion, Keefe and Woods2017; Ciarleglio et al., Reference Ciarleglio, Brucato, Masucci, Altschuler, Colibazzi, Corcoran and Girgis2019), but never meaningfully higher. Addington et al. tracked 172 subjects for 2 years and reported that 29 subjects were converted, while Ciarleglio et al. followed 199 subjects for 2 years and 64 transitions (Addington et al., Reference Addington, Liu, Perkins, Carrion, Keefe and Woods2017; Ciarleglio et al., Reference Ciarleglio, Brucato, Masucci, Altschuler, Colibazzi, Corcoran and Girgis2019). Given this, it can be assumed that long-term follow-up periods need not be mandatory to increase the predictive power of the model. However, since we have not yet performed external validation of this model, it will be necessary to examine whether the predictive accuracy of our model can be replicated in an independent sample with long-term follow-up duration. In regard to predictive models using the LASSO method, Addington et al. externally validated their model (Addington et al., Reference Addington, Liu, Perkins, Carrion, Keefe and Woods2017; Cornblatt et al., Reference Cornblatt, Carrion, Auther, McLaughlin, Olsen, John and Correll2015). Of course, predictive models developed using classical statistical techniques have been externally validated and have undergone refinements several times (Fusar-Poli et al., Reference Fusar-Poli, Davies, Rutigliano, Stahl, Bonoldi and McGuire2019a, Reference Fusar-Poli, Werbeloff, Rutigliano, Oliver, Davies, Stahl and Osborn2019b; Osborne & Mittal, Reference Osborne and Mittal2019; Zhang et al., Reference Zhang, Li, Tang, Niznikiewicz, Shenton, Keshavan and Wang2018), but it will be necessary to continuously refine the model through the optimization of coefficients and variable selection using the LASSO technique in the same sample. More recently, an advanced dynamic prognostic model that combines demographic and clinical variables to forecast the development of psychosis was introduced (Studerus, Beck, Fusar-Poli, & Riecher-Rossler, Reference Studerus, Beck, Fusar-Poli and Riecher-Rossler2020). This model can keep up with changes in symptoms over time, allowing the model to be updated at various time points as the patient is followed. In addition to the prediction models for CHR, an alternative perspective model has also been developed and validated for those who have not yet been diagnosed with CHR (Fusar-Poli et al., Reference Fusar-Poli, Rutigliano, Stahl, Davies, Bonoldi, Reilly and McGuire2017, Reference Fusar-Poli, Werbeloff, Rutigliano, Oliver, Davies, Stahl and Osborn2019b). This model can be applied to a large number of populations in the community. Therefore, in the future, it will be necessary to develop appropriate models that reflect the characteristics of the sample at each stage, such as patients who show attenuated symptoms but have not yet been diagnosed with CHR, those who have just been diagnosed with CHR, and those who are being followed up.

Moreover, based on the results of the clustering analysis, our predictive model classified the CHR into three subgroups characterized by different levels of risks. Risk stratification has already been implemented several times (Addington et al., Reference Addington, Liu, Perkins, Carrion, Keefe and Woods2017; Michel, Ruhrmann, Schimmelmann, Klosterkotter, & Schultze-Lutter, Reference Michel, Ruhrmann, Schimmelmann, Klosterkotter and Schultze-Lutter2014; Ruhrmann et al., Reference Ruhrmann, Schultze-Lutter, Salokangas, Heinimaa, Linszen, Dingemans and Klosterkotter2010). Ruhrmann et al. first classified the samples into four PIs (Ruhrmann et al., Reference Ruhrmann, Schultze-Lutter, Salokangas, Heinimaa, Linszen, Dingemans and Klosterkotter2010). At 18 months, the incidence of class I was 3.5%, and that of class IV was 85.1%. They also developed a new predictive model that was divided into four classes using diagnostic criteria and cognitive function (Michel et al., Reference Michel, Ruhrmann, Schimmelmann, Klosterkotter and Schultze-Lutter2014). In this study, the cumulative hazard rate of class I was 0, and that of class IV was 1.29. However, these studies were not cross-validated; thus, it is difficult to ignore the risk of overfitting problems. Recently, Addington et al. introduced a cross-validated predictive model (Addington et al., Reference Addington, Liu, Perkins, Carrion, Keefe and Woods2017). They divided CHR into three risk groups, but there was no data on the incidence of each group. In contrast, in our model, we found that the three clusters that were stratified using the LASSO model had significantly different degrees of risk. Interestingly, in cluster 1, approximately half of the cases transitioned to psychosis within 3 years of follow-up, whereas in cluster 3, no transition occurred within that time period. Moreover, unlike simply distinguishing between the converters and the non-converters, there were a distinct demographic, symptom, and cognitive differences in each subgroup (online Supplementary Table S3). Our result will help facilitate a personalized therapeutic approach to different degrees of risks among high-risk individuals and will enrich future recruitment efforts, such as targeting only CHR patients with moderate to high risk.

Consistent with previous studies, our model included general functioning, social cognition, social functioning, and verbal memory as significant predictors (Cannon et al., Reference Cannon, Yu, Addington, Bearden, Cadenhead, Cornblatt and Kattan2016; Cornblatt et al., Reference Cornblatt, Carrion, Auther, McLaughlin, Olsen, John and Correll2015; Malda et al., Reference Malda, Boonstra, Barf, de Jong, Aleman, Addington and Pijnenborg2019; Zhang et al., Reference Zhang, Xu, Tang, Li, Tang, Cui and Group2019). However, demographic and prodromal symptom variables were not significant in our analyses and were not included in our model. A recent meta-analysis revealed that the demographic variables of subjects are significant predictors of transition to psychosis. In this study, each of the predictors was stratified into groups ranging from convincing evidence (class I) to weak evidence (class IV) (Oliver et al., Reference Oliver, Reilly, Baccaredda Boy, Petros, Davies, Borgwardt and Fusar-Poli2020). However, out of 26 putative risk/protective factors, there were no class I predictors, while only global functioning and attenuated positive psychotic symptoms are in line with previous findings that the dynamic functional change, rather than the static functional status at a single timepoint, better predicts prognosis (Tarbox et al., Reference Tarbox, Addington, Cadenhead, Cannon, Cornblatt, Perkins and Woods2013; Velthorst et al., Reference Velthorst, Nelson, Wiltink, de Haan, Wood, Lin and Yung2013; Zhang et al., Reference Zhang, Xu, Tang, Li, Tang, Cui and Group2019). Similarly, we observed no significance for the SOPS score. Unlike our model, some other predictive models include both cognitive function and clinical symptoms (Addington et al., Reference Addington, Liu, Perkins, Carrion, Keefe and Woods2017; Cornblatt et al., Reference Cornblatt, Carrion, Auther, McLaughlin, Olsen, John and Correll2015). However, these studies used the rescaled SOPS scores for their models and not the raw scores, which may have resulted in the discrepancy of the results. Further investigation into whether the cognitive function is more influential and essential than clinical symptom scores for predicting psychosis is needed.

The present study has several limitations. First, we did not utilize the variables that were acquired during the patient follow-up, including pharmacotherapy, cognitive-behavioral therapy, and compliance, in our model. Our primary goal was to predict the risk of psychosis using only baseline information. This is reasonable, as at the time of the patients' first evaluation, no interventions had been administered. Although we used 10-year follow-up data, we assumed that this long duration might compensate for the delay in the transition to psychosis related to the use of medication, a dynamic model that reflects both the baseline and longitudinal change may be a better way to solve this problem in the future (Studerus et al., Reference Studerus, Beck, Fusar-Poli and Riecher-Rossler2020; Yuen et al., Reference Yuen, Mackinnon, Hartmann, Amminger, Markulev, Lavoie and Nelson2018). Second, external validation of the model was not performed in our current study. For external validation, the variables used in the model should be identical. However, the actual tasks used to assess specific cognitive domains may differ from study to study. To overcome this problem, a model needs to be developed that utilizes only the variables that overlap across studies. Third, this study was conducted as a data-driven study. Given the number of variables being injected, the sample size may not be large enough. This is a frequently mentioned issue in clinical prediction. Even with the application of data science technique to the current study, it is still probable that the model developed could be biased against the sample's characteristics. It is still necessary to be conducted with the selection of variables based on hypotheses and the development of a model. Last, it has not been explored what life events have affected the transition to psychosis. Further research on this will be needed, especially since the appraisal of life experiences is an important axis of the bio-physico-social model of schizophrenia (Garety et al., Reference Garety, Waller, Emsley, Jolley, Kuipers, Bebbington and Freeman2015).

In summary, we developed a predictive model with 10-year follow-up data of CHR patients. One-third of the CHR patients developed psychosis over a sufficient follow-up period. Our model showed that verbal ability, social cognition, social functioning, functional decline, and general intelligence are important predictors of the transition to psychosis. CHR sample was divided into three clusters according to the degree of risk. We believe that our model could facilitate a personalized therapeutic approach to different risks in high-risk individuals.

Supplementary material

The supplementary material for this article can be found at https://doi.org/10.1017/S0033291720004675

Acknowledgment

This research was supported by the Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Science, ICT and Future Planning (grant no. 2019R1A2B5B03100844; 2017M3C7A1029610). We declare that there is no potential conflict of interest in this study.

References

Addington, J., Cornblatt, B. A., Cadenhead, K. S., Cannon, T. D., McGlashan, T. H., Perkins, D. O., … Heinssen, R. (2011a). At clinical high risk for psychosis: Outcome for nonconverters. The American Journal of Psychiatry, 168(8), 800805. doi: 10.1176/appi.ajp.2011.10081191CrossRefGoogle Scholar
Addington, J., Epstein, I., Liu, L., French, P., Boydell, K. M., & Zipursky, R. B. (2011b). A randomized controlled trial of cognitive behavioral therapy for individuals at clinical high risk of psychosis. Schizophrenia Research, 125(1), 5461. doi: 10.1016/j.schres.2010.10.015CrossRefGoogle Scholar
Addington, J., Farris, M., Stowkowy, J., Santesteban-Echarri, O., Metzak, P., & Kalathil, M. S. (2019). Predictors of transition to psychosis in individuals at clinical high risk. Current Psychiatry Reports, 21(6), 39. doi: 10.1007/s11920-019-1027-yCrossRefGoogle ScholarPubMed
Addington, J., Liu, L., Perkins, D. O., Carrion, R. E., Keefe, R. S., & Woods, S. W. (2017). The role of cognition and social functioning as predictors in the transition to psychosis for youth with attenuated psychotic symptoms. Schizophrenia Bulletin, 43(1), 5763. doi: 10.1093/schbul/sbw152CrossRefGoogle ScholarPubMed
An, S. K., Kang, J. I., Park, J. Y., Kim, K. R., Lee, S. Y., & Lee, E. (2010). Attribution bias in ultra-high risk for psychosis and first-episode schizophrenia. Schizophrenia Research, 118(1–3), 5461. doi: 10.1016/j.schres.2010.01.025CrossRefGoogle ScholarPubMed
Armando, M., Pontillo, M., De Crescenzo, F., Mazzone, L., Monducci, E., Lo Cascio, N., … Schultze-Lutter, F. (2015). Twelve-month psychosis-predictive value of the ultra-high risk criteria in children and adolescents. Schizophrenia Research, 169(1–3), 186192. doi: 10.1016/j.schres.2015.10.033CrossRefGoogle ScholarPubMed
Bentall, R. P., Jackson, H. F., & Pilgrim, D. (1988). Abandoning the concept of ‘schizophrenia’: Some implications of validity arguments for psychological research into psychotic phenomena. British Journal of Clinical Psychology, 27(4), 303324. doi: 10.1111/j.2044-8260.1988.tb00795.xCrossRefGoogle ScholarPubMed
Birchwood, M., Todd, P., & Jackson, C. (1998). Early intervention in psychosis. The critical period hypothesis. The British Journal of Psychiatry Suppl, 172(33), 5359.CrossRefGoogle ScholarPubMed
Braitman, L. E., & Davidoff, F. (1996). Predicting clinical states in individual patients. Annals of Internal Medicine, 125(5), 406412. doi: 10.7326/0003-4819-125-5-199609010-00008CrossRefGoogle ScholarPubMed
Cannon, T. D., Cadenhead, K., Cornblatt, B., Woods, S. W., Addington, J., Walker, E., … Heinssen, R. (2008). Prediction of psychosis in youth at high clinical risk: A multisite longitudinal study in North America. Archives of General Psychiatry, 65(1), 2837. doi: 10.1001/archgenpsychiatry.2007.3CrossRefGoogle ScholarPubMed
Cannon, T. D., Yu, C., Addington, J., Bearden, C. E., Cadenhead, K. S., Cornblatt, B. A., … Kattan, M. W. (2016). An individualized risk calculator for research in prodromal psychosis. The American Journal of Psychiatry, 173(10), 980988. doi: 10.1176/appi.ajp.2016.15070890CrossRefGoogle ScholarPubMed
Carrion, R. E., Cornblatt, B. A., Burton, C. Z., Tso, I. F., Auther, A. M., Adelsheim, S., … McFarlane, W. R. (2016). Personalized prediction of psychosis: External validation of the NAPLS-2 psychosis risk calculator with the EDIPPP project. The American Journal of Psychiatry, 173(10), 989996. doi: 10.1176/appi.ajp.2016.15121565CrossRefGoogle ScholarPubMed
Chelune, G. J., & Baer, R. A. (1986). Developmental norms for the Wisconsin Card Sorting test. Journal of Clinical and Experimental Neuropsychology, 8(3), 219228.CrossRefGoogle ScholarPubMed
Ciarleglio, A. J., Brucato, G., Masucci, M. D., Altschuler, R., Colibazzi, T., Corcoran, C. M., … Girgis, R. R. (2019). A predictive model for conversion to psychosis in clinical high-risk patients. Psychological Medicine, 49(7), 11281137. doi: 10.1017/S003329171800171XCrossRefGoogle ScholarPubMed
Cornblatt, B. A., Carrion, R. E., Auther, A., McLaughlin, D., Olsen, R. H., John, M., & Correll, C. U. (2015). Psychosis prevention: A modified clinical high risk perspective from the Recognition and Prevention (RAP) program. The American Journal of Psychiatry, 172(10), 986994. doi: 10.1176/appi.ajp.2015.13121686CrossRefGoogle ScholarPubMed
Fusar-Poli, P., Bonoldi, I., Yung, A. R., Borgwardt, S., Kempton, M. J., Valmaggia, L., … McGuire, P. (2012). Predicting psychosis: Meta-analysis of transition outcomes in individuals at high clinical risk. Archives of General Psychiatry, 69(3), 220229. doi: 10.1001/archgenpsychiatry.2011.1472CrossRefGoogle ScholarPubMed
Fusar-Poli, P., Borgwardt, S., Bechdolf, A., Addington, J., Riecher-Rossler, A., Schultze-Lutter, F., … Yung, A. (2013). The psychosis high-risk state: A comprehensive state-of-the-art review. JAMA Psychiatry, 70(1), 107120. doi: 10.1001/jamapsychiatry.2013.269CrossRefGoogle ScholarPubMed
Fusar-Poli, P., Cappucciati, M., Borgwardt, S., Woods, S. W., Addington, J., Nelson, B., … McGuire, P. K. (2016a). Heterogeneity of psychosis risk within individuals at clinical high risk: A meta-analytical stratification. JAMA Psychiatry, 73(2), 113120. doi: 10.1001/jamapsychiatry.2015.2324CrossRefGoogle Scholar
Fusar-Poli, P., Cappucciati, M., Rutigliano, G., Schultze-Lutter, F., Bonoldi, I., Borgwardt, S., … McGuire, P. (2015). At risk or not at risk? A meta-analysis of the prognostic accuracy of psychometric interviews for psychosis prediction. World Psychiatry, 14(3), 322332. doi: 10.1002/wps.20250CrossRefGoogle ScholarPubMed
Fusar-Poli, P., Davies, C., Rutigliano, G., Stahl, D., Bonoldi, I., & McGuire, P. (2019a). Transdiagnostic individualized clinically based risk calculator for the detection of individuals at risk and the prediction of psychosis: Model refinement including nonlinear effects of age. Frontiers in Psychiatry, 10, 313. doi: 10.3389/fpsyt.2019.00313CrossRefGoogle Scholar
Fusar-Poli, P., Hijazi, Z., Stahl, D., & Steyerberg, E. W. (2018). The science of prognosis in psychiatry: A review. JAMA Psychiatry, 75(12), 12891297. doi: 10.1001/jamapsychiatry.2018.2530CrossRefGoogle ScholarPubMed
Fusar-Poli, P., Rutigliano, G., Stahl, D., Davies, C., Bonoldi, I., Reilly, T., & McGuire, P. (2017). Development and validation of a clinically based risk calculator for the transdiagnostic prediction of psychosis. JAMA Psychiatry, 74(5), 493500. doi: 10.1001/jamapsychiatry.2017.0284CrossRefGoogle ScholarPubMed
Fusar-Poli, P., Rutigliano, G., Stahl, D., Schmidt, A., Ramella-Cravaro, V., Hitesh, S., & McGuire, P. (2016a). Deconstructing pretest risk enrichment to optimize prediction of psychosis in individuals at clinical high risk. JAMA Psychiatry, 73(12), 12601267. doi: 10.1001/jamapsychiatry.2016.2707CrossRefGoogle Scholar
Fusar-Poli, P., & Schultze-Lutter, F. (2016). Predicting the onset of psychosis in patients at clinical high risk: Practical guide to probabilistic prognostic reasoning. Evidence-Based Mental Health, 19(1), 1015. doi: 10.1136/eb-2015-102295CrossRefGoogle ScholarPubMed
Fusar-Poli, P., Schultze-Lutter, F., & Addington, J. (2016b). Intensive community outreach for those at ultra high risk of psychosis: Dilution, not solution. Lancet Psychiatry, 3(1), 18. doi: 10.1016/S2215-0366(15)00491-5CrossRefGoogle Scholar
Fusar-Poli, P., Schultze-Lutter, F., Cappucciati, M., Rutigliano, G., Bonoldi, I., Stahl, D., … McGuire, P. (2016c). The dark side of the moon: Meta-analytical impact of recruitment strategies on risk enrichment in the clinical high risk state for psychosis. Schizophrenia Bulletin, 42(3), 732743. doi: 10.1093/schbul/sbv162CrossRefGoogle Scholar
Fusar-Poli, P., Werbeloff, N., Rutigliano, G., Oliver, D., Davies, C., Stahl, D., … Osborn, D. (2019b). Transdiagnostic risk calculator for the automatic detection of individuals at risk and the prediction of psychosis: Second replication in an independent national health service trust. Schizophrenia Bulletin, 45(3), 562570. doi: 10.1093/schbul/sby070CrossRefGoogle Scholar
Garety, P., Waller, H., Emsley, R., Jolley, S., Kuipers, E., Bebbington, P., … Freeman, D. (2015). Cognitive mechanisms of change in delusions: An experimental investigation targeting reasoning to effect change in paranoia. Schizophrenia Bulletin, 41(2), 400410. doi: 10.1093/schbul/sbu103CrossRefGoogle ScholarPubMed
Glenthoj, L. B., Kristensen, T. D., Wenneberg, C., Hjorthoj, C., & Nordentoft, M. (2020). Predictors of remission from the ultra-high risk state for psychosis. Early Intervention in Psychiatry. doi: 10.1111/eip.12917CrossRefGoogle Scholar
Goldman, L., Cook, E. F., Johnson, P. A., Brand, D. A., Rouan, G. W., & Lee, T. H. (1996). Prediction of the need for intensive care in patients who come to emergency departments with acute chest pain. The New England Journal of Medicine, 334(23), 14981504. doi: 10.1056/NEJM199606063342303CrossRefGoogle ScholarPubMed
Guloksuz, S., & van Os, J. (2018). The slow death of the concept of schizophrenia and the painful birth of the psychosis spectrum. Psychological Medicine, 48(2), 229244. doi: 10.1017/S0033291717001775CrossRefGoogle ScholarPubMed
Hahn, T., Nierenberg, A. A., & Whitfield-Gabrieli, S. (2017). Predictive analytics in mental health: Applications, guidelines, challenges and perspectives. Molecular Psychiatry, 22(1), 3743. doi: 10.1038/mp.2016.201CrossRefGoogle Scholar
Hamilton, M. (1959). The assessment of anxiety states by rating. British Journal of Medical Psychology, 32(1), 5055. doi: 10.1111/j.2044-8341.1959.tb00467.xCrossRefGoogle ScholarPubMed
Janes, H., Pepe, M. S., & Gu, W. (2008). Assessing the value of risk predictions by using risk stratification tables. Annals of Internal Medicine, 149(10), 751760. doi: 10.7326/0003-4819-149-10-200811180-00009CrossRefGoogle ScholarPubMed
Jung, M. H., Jang, J. H., Kang, D. H., Choi, J. S., Shin, N. Y., Kim, H. S., … Kwon, J. S. (2010). The reliability and validity of the Korean version of the structured interview for prodromal syndrome. Psychiatry Investigation, 7(4), 257263. doi: 10.4306/pi.2010.7.4.257CrossRefGoogle ScholarPubMed
Kang, Y., Chin, J.-H., Na, D., Lee, J., & Park, J. (2000). A normative study of the Korean version of Controlled Oral Word Association Test (COWAT) in the elderly. Korean Journal of Clinical Psychology, 19(2), 385392.Google Scholar
Katsura, M., Ohmuro, N., Obara, C., Kikuchi, T., Ito, F., Miyakoshi, T., … Matsumoto, K. (2014). A naturalistic longitudinal study of at-risk mental state with a 2.4 year follow-up at a specialized clinic setting in Japan. Schizophrenia Research, 158(1–3), 3238. doi: 10.1016/j.schres.2014.06.013CrossRefGoogle Scholar
Kempton, M. J., Bonoldi, I., Valmaggia, L., McGuire, P., & -Poli, P. (2015). Speed of psychosis progression in people at ultra-high clinical risk: A complementary meta-analysis. JAMA Psychiatry, 72(6), 622623. doi: 10.1001/jamapsychiatry.2015.0094CrossRefGoogle ScholarPubMed
Kim, E., Jang, J. H., Park, H. Y., Shim, G., Hwang, J. Y., Kim, S. N., & Kwon, J. S. (2012). Pharmacotherapy and clinical characteristics of ultra-high-risk for psychosis according to conversion status: A naturalistic observational study. Early Intervention in Psychiatry, 6(1), 3037. doi: 10.1111/j.1751-7893.2011.00295.xCrossRefGoogle ScholarPubMed
Kim, J. K., & Kang, Y. (1999). Brief report normative study of the Korean-California Verbal Learning Test (K-CVLT). The Clinical Neuropsychologist, 13(3), 365369.CrossRefGoogle Scholar
Kim, C. K., & Lee, J. A. (2009). Development of the Korean version of the social functioning scale in the schizophrenics: A study on the reliability and validity. Korean Journal of Biological Psychiatry, 16(2), 76.Google Scholar
Kim, B. J., Lee, C. S., Oh, B. H., Hong, C. H., Lee, K. S., Son, S. J., … Kim, K. W. (2013). A normative study of lexical verbal fluency in an educationally-diverse elderly population. Psychiatry Investigation, 10(4), 346351. doi: 10.4306/pi.2013.10.4.346CrossRefGoogle Scholar
Koike, S., Takano, Y., Iwashiro, N., Satomura, Y., Suga, M., Nagai, T., … Kasai, K. (2013). A multimodal approach to investigate biomarkers for psychosis in a clinical setting: The integrative neuroimaging studies in schizophrenia targeting for early intervention and prevention (IN-STEP) project. Schizophrenia Research, 143(1), 116124. doi: 10.1016/j.schres.2012.11.012CrossRefGoogle Scholar
Kotlicka-Antczak, M., Karbownik, M. S., Stawiski, K., Pawelczyk, A., Zurner, N., Pawelczyk, T., … Fusar-Poli, P. (2019). Short clinically-based prediction model to forecast transition to psychosis in individuals at clinical high risk state. European Psychiatry, 58, 7279. doi: 10.1016/j.eurpsy.2019.02.007CrossRefGoogle ScholarPubMed
Koutsouleris, N., Upthegrove, R., & Wood, S. J. (2019). Importance of variable selection in multimodal prediction models in patients at clinical high risk for psychosis and recent onset depression-reply. JAMA Psychiatry, 76(3), 339340. doi: 10.1001/jamapsychiatry.2018.4237CrossRefGoogle ScholarPubMed
Kwon, J. S., Byun, M. S., Lee, T. Y., & An, S. K. (2012). Early intervention in psychosis: Insights from Korea. Asian Journal of Psychiatry, 5(1), 98105. doi: 10.1016/j.ajp.2012.02.007CrossRefGoogle ScholarPubMed
Lee, Y. H., Bang, H., & Kim, D. J. (2016). How to establish clinical prediction models. Endocrinology and Metabolism (Seoul), 31(1), 3844. doi: 10.3803/EnM.2016.31.1.38CrossRefGoogle ScholarPubMed
Lee, T. Y., Kim, S. N., Correll, C. U., Byun, M. S., Kim, E., Jang, J. H., … Kwon, J. S. (2014a). Symptomatic and functional remission of subjects at clinical high risk for psychosis: A 2-year naturalistic observational study. Schizophrenia Research, 156(2–3), 266271. doi: 10.1016/j.schres.2014.04.002CrossRefGoogle Scholar
Lee, T. Y., Shin, Y. S., Shin, N. Y., Kim, S. N., Jang, J. H., Kang, D. H., & Kwon, J. S. (2014b). Neurocognitive function as a possible marker for remission from clinical high risk for psychosis. Schizophrenia Research, 153(1–3), 4853. doi: 10.1016/j.schres.2014.01.018CrossRefGoogle Scholar
Lin, A., Wood, S. J., Nelson, B., Beavan, A., McGorry, P., & Yung, A. R. (2015). Outcomes of nontransitioned cases in a sample at ultra-high risk for psychosis. The American Journal of Psychiatry, 172(3), 249258. doi: 10.1176/appi.ajp.2014.13030418CrossRefGoogle Scholar
Lip, G. Y., Nieuwlaat, R., Pisters, R., Lane, D. A., & Crijns, H. J. (2010). Refining clinical risk stratification for predicting stroke and thromboembolism in atrial fibrillation using a novel risk factor-based approach: The euro heart survey on atrial fibrillation. Chest, 137(2), 263272. doi: 10.1378/chest.09-1584CrossRefGoogle ScholarPubMed
Malda, A., Boonstra, N., Barf, H., de Jong, S., Aleman, A., Addington, J., … Pijnenborg, G. H. M. (2019). Individualized prediction of transition to psychosis in 1676 individuals at clinical high risk: Development and validation of a multivariable prediction model based on individual patient data meta-analysis. Frontiers in Psychiatry, 10, 345. doi: 10.3389/fpsyt.2019.00345CrossRefGoogle ScholarPubMed
McGlashan, T. H., & Johannessen, J. O. (1996). Early detection and intervention with schizophrenia: Rationale. Schizophrenia Bulletin, 22(2), 201222. doi: 10.1093/schbul/22.2.201CrossRefGoogle ScholarPubMed
McGorry, P. D., Killackey, E., & Yung, A. (2008). Early intervention in psychosis: Concepts, evidence and future directions. World Psychiatry, 7(3), 148156. doi: 10.1002/j.2051-5545.2008.tb00182.xCrossRefGoogle ScholarPubMed
Michel, C., Ruhrmann, S., Schimmelmann, B. G., Klosterkotter, J., & Schultze-Lutter, F. (2014). A stratified model for psychosis prediction in clinical practice. Schizophrenia Bulletin, 40(6), 15331542. doi: 10.1093/schbul/sbu025CrossRefGoogle ScholarPubMed
Mitter, N., Nah, G. Q., Bong, Y. L., Lee, J., & Chong, S. A. (2014). Longitudinal Youth-At-Risk Study (LYRIKS): Outreach strategies based on a community-engaged framework. Early Intervention in Psychiatry, 8(3), 298303. doi: 10.1111/eip.12049CrossRefGoogle ScholarPubMed
Morrison, A. P., French, P., Parker, S., Roberts, M., Stevens, H., Bentall, R. P., & Lewis, S. W. (2007). Three-year follow-up of a randomized controlled trial of cognitive therapy for the prevention of psychosis in people at ultrahigh risk. Schizophrenia Bulletin, 33(3), 682687. doi: 10.1093/schbul/sbl042CrossRefGoogle ScholarPubMed
Nelson, B., Yuen, H. P., Wood, S. J., Lin, A., Spiliotacopoulos, D., Bruxner, A., … Yung, A. R. (2013). Long-term follow-up of a group at ultra high risk (‘prodromal’) for psychosis: The PACE 400 study. JAMA Psychiatry, 70(8), 793802. doi: 10.1001/jamapsychiatry.2013.1270CrossRefGoogle Scholar
Nelson, B., & Yung, A. R. (2010). Can clinicians predict psychosis in an ultra high risk group? Australian and New Zealand Journal of Psychiatry, 44(7), 625630. doi: 10.3109/00048671003620210CrossRefGoogle Scholar
Nelson, B., Yung, A. R., & McGorry, P. D. (2019). Importance of variable selection in multimodal prediction models in patients at clinical high risk for psychosis and recent-onset depression. JAMA Psychiatry, 76(3), 339. doi: 10.1001/jamapsychiatry.2018.4234CrossRefGoogle ScholarPubMed
Nieman, D. H., Ruhrmann, S., Dragt, S., Soen, F., van Tricht, M. J., Koelman, J. H., … de Haan, L. (2014). Psychosis prediction: Stratification of risk estimation with information-processing and premorbid functioning variables. Schizophrenia Bulletin, 40(6), 14821490. doi: 10.1093/schbul/sbt145CrossRefGoogle ScholarPubMed
Oliver, D., Reilly, T. J., Baccaredda Boy, O., Petros, N., Davies, C., Borgwardt, S., … Fusar-Poli, P. (2020). What causes the onset of psychosis in individuals at clinical high risk? A meta-analysis of risk and protective factors. Schizophrenia Bulletin, 46(1), 110120. doi: 10.1093/schbul/sbz039CrossRefGoogle Scholar
Osborne, K. J., & Mittal, V. A. (2019). External validation and extension of the NAPLS-2 and SIPS-RC personalized risk calculators in an independent clinical high-risk sample. Psychiatry Research, 279, 914. doi: 10.1016/j.psychres.2019.06.034CrossRefGoogle Scholar
Pruessner, M., Faridi, K., Shah, J., Rabinovitch, M., Iyer, S., Abadi, S., … Malla, A. K. (2017). The Clinic for Assessment of Youth at Risk (CAYR): 10 years of service delivery and research targeting the prevention of psychosis in Montreal, Canada. Early Intervention in Psychiatry, 11(2), 177184. doi: 10.1111/eip.12300CrossRefGoogle ScholarPubMed
Reitan, R. M. (1958). Validity of the Trail Making Test as an indicator of organic brain damage. Perceptual and Motor Skills, 8(3), 271276.CrossRefGoogle Scholar
Riecher-Rossler, A., Gschwandtner, U., Aston, J., Borgwardt, S., Drewe, M., Fuhr, P., … Stieglitz, R. D. (2007). The Basel early-detection-of-psychosis (FEPSY)-study – design and preliminary results. Acta Psychiatrica Scandinavica, 115(2), 114125. doi: 10.1111/j.1600-0447.2006.00854.xCrossRefGoogle ScholarPubMed
Ruhrmann, S., Schultze-Lutter, F., Salokangas, R. K., Heinimaa, M., Linszen, D., Dingemans, P., … Klosterkotter, J. (2010). Prediction of psychosis in adolescents and young adults at high risk: Results from the prospective European prediction of psychosis study. Archives of General Psychiatry, 67(3), 241251. doi: 10.1001/archgenpsychiatry.2009.206CrossRefGoogle Scholar
Schmidt, A., Cappucciati, M., Radua, J., Rutigliano, G., Rocchetti, M., Dell'Osso, L., … Fusar-Poli, P. (2017). Improving prognostic accuracy in subjects at clinical high risk for psychosis: Systematic review of predictive models and meta-analytical sequential testing simulation. Schizophrenia Bulletin, 43(2), 375388. doi: 10.1093/schbul/sbw098Google ScholarPubMed
Schultze-Lutter, F., Klosterkotter, J., & Ruhrmann, S. (2014). Improving the clinical prediction of psychosis by combining ultra-high risk criteria and cognitive basic symptoms. Schizophrenia Research, 154(1–3), 100106. doi: 10.1016/j.schres.2014.02.010CrossRefGoogle ScholarPubMed
Schultze-Lutter, F., Schimmelmann, B. G., Ruhrmann, S., & Michel, C. (2013). ‘A rose is a rose is a rose’, but at-risk criteria differ. Psychopathology, 46(2), 7587. doi: 10.1159/000339208CrossRefGoogle Scholar
Shin, M. S., Park, S. Y., Park, S. R., Seol, S. H., & Kwon, J. S. (2006). Clinical and empirical applications of the Rey-Osterrieth Complex Figure Test. Nature Protocol, 1(2), 892899. doi: 10.1038/nprot.2006.115CrossRefGoogle ScholarPubMed
Simon, A. E., Borgwardt, S., Riecher-Rossler, A., Velthorst, E., de Haan, L., & Fusar-Poli, P. (2013). Moving beyond transition outcomes: Meta-analysis of remission rates in individuals at high clinical risk for psychosis. Psychiatry Research, 209(3), 266272. doi: 10.1016/j.psychres.2013.03.004CrossRefGoogle ScholarPubMed
Simon, N., Friedman, J., Hastie, T., & Tibshirani, R. (2011). Regularization paths for Cox's proportional hazards model via coordinate descent. Journal of Statistical Software, 39(5), 1.CrossRefGoogle ScholarPubMed
Spitz, A., Studerus, E., Koranyi, S., Rapp, C., Ramyead, A., Ittig, S., … Riecher-Rossler, A. (2017). Correlations between self-rating and observer-rating of psychopathology in at-risk mental state and first-episode psychosis patients: Influence of disease stage and gender. Early Intervention in Psychiatry, 11(6), 461470. doi: 10.1111/eip.12270CrossRefGoogle ScholarPubMed
Steyerberg, E. W. (2008). Clinical prediction models: A practical approach to development, validation, and updating. Berlin: Springer Science & Business Media.Google Scholar
Studerus, E., Beck, K., Fusar-Poli, P., & Riecher-Rossler, A. (2020). Development and validation of a dynamic risk prediction model to forecast psychosis onset in patients at clinical high risk. Schizophrenia Bulletin, 46(2), 252260. doi: 10.1093/schbul/sbz059Google ScholarPubMed
Studerus, E., Ramyead, A., & Riecher-Rossler, A. (2017). Prediction of transition to psychosis in patients with a clinical high risk for psychosis: A systematic review of methodology and reporting. Psychological Medicine, 47(7), 11631178. doi: 10.1017/S0033291716003494CrossRefGoogle ScholarPubMed
Tarbox, S. I., Addington, J., Cadenhead, K. S., Cannon, T. D., Cornblatt, B. A., Perkins, D. O., … Woods, S. W. (2013). Premorbid functional development and conversion to psychosis in clinical high-risk youths. Development and Psychopathology, 25(4 Pt 1), 11711186. doi: 10.1017/S0954579413000448CrossRefGoogle ScholarPubMed
Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological), 58(1), 267288.Google Scholar
Tibshirani, R. (1997). The lasso method for variable selection in the Cox model. Statistics in Medicine, 16(4), 385395. doi: 10.1002/(sici)1097-0258(19970228)16:4<385::aid-sim380>3.0.co;2-33.0.CO;2-3>CrossRefGoogle ScholarPubMed
van der Gaag, M., Nieman, D. H., Rietdijk, J., Dragt, S., Ising, H. K., Klaassen, R. M., … Linszen, D. H. (2012). Cognitive behavioral therapy for subjects at ultrahigh risk for developing psychosis: A randomized controlled clinical trial. Schizophrenia Bulletin, 38(6), 11801188. doi: 10.1093/schbul/sbs105CrossRefGoogle ScholarPubMed
Velthorst, E., Nelson, B., Wiltink, S., de Haan, L., Wood, S. J., Lin, A., & Yung, A. R. (2013). Transition to first episode psychosis in ultra high risk populations: Does baseline functioning hold the key? Schizophrenia Research, 143(1), 132137. doi: 10.1016/j.schres.2012.10.025CrossRefGoogle ScholarPubMed
Wasson, J. H., Sox, H. C., Neff, R. K., & Goldman, L. (1985). Clinical prediction rules. Applications and methodological standards. The New England Journal of Medicine, 313(13), 793799. doi: 10.1056/NEJM198509263131306CrossRefGoogle ScholarPubMed
Webb, J. R., Addington, J., Perkins, D. O., Bearden, C. E., Cadenhead, K. S., Cannon, T. D., … Woods, S. W. (2015). Specificity of incident diagnostic outcomes in patients at clinical high risk for psychosis. Schizophrenia Bulletin, 41(5), 10661075. doi: 10.1093/schbul/sbv091CrossRefGoogle ScholarPubMed
Woods, S. W., Powers, A. R III, Taylor, J. H., Davidson, C. A., Johannesen, J. K., Addington, J., … McGlashan, T. H. (2018). Lack of diagnostic pluripotentiality in patients at clinical high risk for psychosis: Specificity of comorbidity persistence and search for pluripotential subgroups. Schizophrenia Bulletin, 44(2), 254263. doi: 10.1093/schbul/sbx138.CrossRefGoogle ScholarPubMed
Yeom, T., Park, Y., Oh, K., & Lee, Y. (1992). Korean Version Wechsler adult intelligence scale. Seoul: Korean Guidance, 4, 1328.Google Scholar
Yi, J.-S., Bae, S.-O., Ahn, Y.-M., Park, D.-B., Noh, K.-S., Shin, H.-K., … Kim, Y.-S. (2005). Validity and reliability of the Korean version of the Hamilton Depression Rating Scale (K-HDRS). Journal of Korean Neuropsychiatric Association, 44(4), 456465.Google Scholar
Yi, J., Chung, M., Lee, B., & Lee, S. (2003). Reliability and validity of the Korean version of global assessment of functioning scale. Korean Journal of Schizophrenia Research, 6, 118124.Google Scholar
Yuen, H. P., Mackinnon, A., Hartmann, J., Amminger, G. P., Markulev, C., Lavoie, S., … Nelson, B. (2018). Dynamic prediction of transition to psychosis using joint modelling. Schizophrenia Research, 202, 333340. doi: 10.1016/j.schres.2018.07.002CrossRefGoogle ScholarPubMed
Yung, A. R., Fusar-Poli, P., & Nelson, B. (2012). The ultra high risk approach to define psychosis risk. Current Pharmaceutical Design, 18(4), 346350. doi: 10.2174/138161212799316299CrossRefGoogle ScholarPubMed
Yung, A. R., McGorry, P. D., McFarlane, C. A., Jackson, H. J., Patton, G. C., & Rakkar, A. (1996). Monitoring and care of young people at incipient risk of psychosis. Schizophrenia Bulletin, 22(2), 283303. doi: 10.1093/schbul/22.2.283CrossRefGoogle ScholarPubMed
Yung, A. R., Yuen, H. P., Berger, G., Francey, S., Hung, T. C., Nelson, B., … McGorry, P. (2007). Declining transition rate in ultra high risk (prodromal) services: Dilution or reduction of risk? Schizophrenia Bulletin, 33(3), 673681. doi: 10.1093/schbul/sbm015CrossRefGoogle ScholarPubMed
Zhang, T., Li, H., Tang, Y., Niznikiewicz, M. A., Shenton, M. E., Keshavan, M. S., … Wang, J. (2018). Validating the predictive accuracy of the NAPLS-2 psychosis risk calculator in a clinical high-risk sample from the SHARP (Shanghai At Risk for Psychosis) program. The American Journal of Psychiatry, 175(9), 906908. doi: 10.1176/appi.ajp.2018.18010036CrossRefGoogle Scholar
Zhang, T., Xu, L., Tang, Y., Li, H., Tang, X., Cui, H., … Group, S. S. (2019). Prediction of psychosis in prodrome: Development and validation of a simple, personalized risk calculator. Psychological Medicine, 49(12), 19901998. doi: 10.1017/S0033291718002738CrossRefGoogle ScholarPubMed
Ziermans, T., de Wit, S., Schothorst, P., Sprong, M., van Engeland, H., Kahn, R., & Durston, S. (2014). Neurocognitive and clinical predictors of long-term outcome in adolescents at ultra-high risk for psychosis: A 6-year follow-up. PLoS ONE, 9(4), e93994. doi: 10.1371/journal.pone.0093994CrossRefGoogle ScholarPubMed

Lee et al. supplementary material

Table S1-S3

File 23 KB

Altmetric attention score

Full text views

Full text views reflects PDF downloads, PDFs sent to Google Drive, Dropbox and Kindle and HTML full text views.

Total number of HTML views: 430
Total number of PDF views: 333 *
View data table for this chart

* Views captured on Cambridge Core between 14th December 2020 - 11th April 2021. This data will be updated every 24 hours.

Access Access
Open access Open access