Data from neurocognitive assessments may not be accurate in the context of factors impacting validity, such as disengagement, unmotivated responding, or intentional underperformance. Performance validity tests (PVTs) were developed to address these phenomena and assess underperformance on neurocognitive tests. However, PVTs can be burdensome, rely on cutoff scores that reduce information, do not examine potential variations in task engagement across a battery, and are typically not well-suited to acquisition of large cognitive datasets. Here we describe the development of novel performance validity measures that could address some of these limitations by leveraging psychometric concepts using data embedded within the Penn Computerized Neurocognitive Battery (PennCNB).

We first developed these validity measures using simulations of invalid response patterns with parameters drawn from real data. Next, we examined their application in two large, independent samples: 1) children and adolescents from the Philadelphia Neurodevelopmental Cohort (n = 9498); and 2) adult servicemembers from the Marine Resiliency Study-II (n = 1444).

Our performance validity metrics detected patterns of invalid responding in simulated data, even at subtle levels. Furthermore, a combination of these metrics significantly predicted previously established validity rules for these tests in both developmental and adult datasets. Moreover, most clinical diagnostic groups did not show reduced validity estimates.

These results provide proof-of-concept evidence for multivariate, data-driven performance validity metrics. These metrics offer a novel method for determining the performance validity for individual neurocognitive tests that is scalable, applicable across different tests, less burdensome, and dimensional. However, more research is needed into their application.

Assessment of risks of illnesses has been an important part of medicine for decades. We now have hundreds of ‘risk calculators’ for illnesses, including brain disorders, and these calculators are continually improving as more diverse measures are collected on larger samples.

We first replicated an existing psychosis risk calculator and then used our own sample to develop a similar calculator for use in recruiting ‘psychosis risk’ enriched community samples. We assessed 632 participants age 8–21 (52% female; 48% Black) from a community sample with longitudinal data on neurocognitive, clinical, medical, and environmental variables. We used this information to predict psychosis spectrum (PS) status in the future. We selected variables based on lasso, random forest, and statistical inference relief; and predicted future PS using ridge regression, random forest, and support vector machines.

Cross-validated prediction diagnostics were obtained by building and testing models in randomly selected sub-samples of the data, resulting in a distribution of the diagnostics; we report the mean. The strongest predictors of later PS status were the Children's Global Assessment Scale; delusions of predicting the future or having one's thoughts/actions controlled; and the percent married in one's neighborhood. Random forest followed by ridge regression was most accurate, with a cross-validated area under the curve (AUC) of 0.67. Adjustment of the model including only six variables reached an AUC of 0.70.

Results support the potential application of risk calculators for screening and identification of at-risk community youth in prospective investigations of developmental trajectories of the PS.