The ability to remotely monitor cognitive skills is increasing with the ubiquity of smartphones. The Mobile Toolbox (MTB) is a new measurement system that includes measures assessing Executive Functioning (EF) and Processing Speed (PS): Arrow Matching, Shape-Color Sorting, and Number-Symbol Match. The purpose of this study was to assess their psychometric properties.

MTB measures were developed for smartphone administration based on constructs measured in the NIH Toolbox® (NIHTB). Psychometric properties of the resulting measures were evaluated in three studies with participants ages 18 to 90. In Study 1 (N = 92), participants completed MTB measures in the lab and were administered both equivalent NIH TB measures and other external measures of similar cognitive constructs. In Study 2 (N = 1,021), participants completed the equivalent NIHTB measures in the lab and then took the MTB measures on their own, remotely. In Study 3 (N = 168), participants completed MTB measures twice remotely, two weeks apart.

All three measures exhibited very high internal consistency and strong test-retest reliability, as well as moderately high correlations with comparable NIHTB tests and moderate correlations with external measures of similar constructs. Phone operating system (iOS vs. Android) had a significant impact on performance for Arrow Matching and Shape-Color Sorting, but no impact on either validity or reliability.

Results support the reliability and convergent validity of MTB EF and PS measures for use across the adult lifespan in remote, self-administered designs.

To present validation evidence for the first eight cognitive measures available through Mobile Toolbox (MTB). These measures use a remote self-administered platform to assess language, working memory, episodic memory, executive function, and processing speed.

We used two separate samples, recruited as part of a larger study, to validate MTB measures. Sample I, comprised of 92 English-speaking adults ages 18-85, was used to assess internal consistency and construct validity. Participants were first administered “gold standard” cognitive measures (Wechsler Memory Scale-IV Verbal Paired Associates I and II; Wechsler Adult Intelligence Scale-IV Symbol Search, Digit Span, Coding, and Letter-Number Sequencing; Delis-Kaplan Executive Function System Color-Word Interference Test, Peabody Picture Vocabulary Test, Wechsler Individual Achievement Test-4 Spelling, and the Wisconsin Card Sorting Test), after which they completed MTB (pre-loaded on a study-provided smartphone) on their own. Internal consistency was evaluated using measure-appropriate indices (split-half reliability, Cronbach’s alpha or IRT-based indices). Pearson correlation coefficients between MTB tests and measures of similar constructs were used to evaluate concurrent validity. For two tests with timing-dependent scores, Arrow Matching and Shape-Color Sorting, separate analyses were performed for iOS and Android devices. Sample II, with 1,120 English-speaking participants ages 18-90, was used to evaluate age-related change. Participants completed MTB measures remotely on their own smartphones, in a preset order, within a 14-day period. Spearman correlation coefficients, corrected for education, were calculated to evaluate relationships between age and test scores.

Sample I participants were 67% female, 52% white, 99% non-Hispanic; average age=48 (SD= 17). Education was: < high school (1%); high school (55%); some college (21%); college (15%); graduate degree (8%). Internal consistency estimates ranged from 0.81 to 0.99. Pearson correlations between MTB and external measures ranged from 0.41 to 0.86 (all p < .01). Of the timed tests, only Shape-Color sorting showed significant score differences between Android and iOS devices. Sample II was 57% female, 13% Hispanic, 72% white, mean age = 45 (SD = 21). Education distribution was: < high school (2%); high school (34%); some college (34%), college (20%); graduate degree (11%). Measures of executive function (r = -0.50; r=-0.57) and processing speed (r= -0.61) showed the expected negative correlation with age (all p <0.001). Negative correlations, although weaker, were also seen on measures of working memory (r=-0.2) and episodic memory (r=-0.2, r=-0.37; p.<.001). Vocabulary performance improved with age (r=0.4; p<.001), while spelling scores remained stable (r=0.09).

Initial studies support the validity and reliability of the first eight MTB cognitive measures in two diverse samples. MTB tests showed satisfactory construct validity, as demonstrated by the associations between MTB and well-established tests. Furthermore, most MTB measures correlated with age in the expected directions. Executive function, processing speed and memory typically decrease with age and this decrease was reflected in MTB test performance. In contrast, spelling and vocabulary, typically preserved as we age, did not decrease in our sample. Our results support the use of MTB in cognitive aging research.