Skip to main content Accessibility help
×
Home

Prospective, Head-to-Head Study of Three Computerized Neurocognitive Assessment Tools (CNTs): Reliability and Validity for the Assessment of Sport-Related Concussion

  • Lindsay D. Nelson (a1), Ashley A. LaRoche (a1), Adam Y. Pfaller (a1), E. Brooke Lerner (a1), Thomas A. Hammeke (a1) (a2), Christopher Randolph (a3), William B. Barr (a4), Kevin Guskiewicz (a5) and Michael A. McCrea (a1) (a2)...

Abstract

Limited data exist comparing the performance of computerized neurocognitive tests (CNTs) for assessing sport-related concussion. We evaluated the reliability and validity of three CNTs—ANAM, Axon Sports/Cogstate Sport, and ImPACT—in a common sample. High school and collegiate athletes completed two CNTs each at baseline. Concussed (n=165) and matched non-injured control (n=166) subjects repeated testing within 24 hr and at 8, 15, and 45 days post-injury. Roughly a quarter of each CNT’s indices had stability coefficients (M=198 day interval) over .70. Group differences in performance were mostly moderate to large at 24 hr and small by day 8. The sensitivity of reliable change indices (RCIs) was best at 24 hr (67.8%, 60.3%, and 47.6% with one or more significant RCIs for ImPACT, Axon, and ANAM, respectively) but diminished to near the false positive rates thereafter. Across time, the CNTs’ sensitivities were highest in those athletes who became asymptomatic within 1 day before neurocognitive testing but was similar to the tests’ false positive rates when including athletes who became asymptomatic several days earlier. Test–retest reliability was similar among these three CNTs and below optimal standards for clinical use on many subtests. Analyses of group effect sizes, discrimination, and sensitivity and specificity suggested that the CNTs may add incrementally (beyond symptom scores) to the identification of clinical impairment within 24 hr of injury or within a short time period after symptom resolution but do not add significant value over symptom assessment later. The rapid clinical recovery course from concussion and modest stability probably jointly contribute to limited signal detection capabilities of neurocognitive tests outside a brief post-injury window. (JINS, 2016, 22, 24–37)

  • View HTML
    • Send article to Kindle

      To send this article to your Kindle, first ensure no-reply@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about sending to your Kindle. Find out more about sending to your Kindle.

      Note you can select to send to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be sent to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

      Find out more about the Kindle Personal Document Service.

      Prospective, Head-to-Head Study of Three Computerized Neurocognitive Assessment Tools (CNTs): Reliability and Validity for the Assessment of Sport-Related Concussion
      Available formats
      ×

      Send article to Dropbox

      To send this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Dropbox.

      Prospective, Head-to-Head Study of Three Computerized Neurocognitive Assessment Tools (CNTs): Reliability and Validity for the Assessment of Sport-Related Concussion
      Available formats
      ×

      Send article to Google Drive

      To send this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Google Drive.

      Prospective, Head-to-Head Study of Three Computerized Neurocognitive Assessment Tools (CNTs): Reliability and Validity for the Assessment of Sport-Related Concussion
      Available formats
      ×

Copyright

Corresponding author

Correspondence and reprint requests to: Lindsay Nelson, 8701 West Watertown Plank Road, Milwaukee, WI. E-mail: linelson@mcw.edu.

References

Hide All
Belanger, H.G., & Vanderploeg, R.D. (2005). The neuropsychological impact of sports-related concussion: A meta-analysis. Journal of the International Neuropsychological Society, 11, 345357.
Benjamini, Y., & Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society: Series B (Statistical Mehodology), 57, 289300.
Benjamini, Y., & Yekutieli, D. (2001). The control of the false discovery rate in multiple testing under dependency. The Annals of Statistics, 29, 11651188.
Bleiberg, J., Cernich, A.N., Cameron, K., Sun, W., Peck, K., Ecklund, P.J., & Warden, D.L. (2004). Duration of cognitive impairment after sports concussion. Neurosurgery, 54, 10731078.
Broglio, S.P., Ferrara, M.S., Macciocchi, S.N., Baumgartner, T.A., & Elliott, R. (2007). Test-retest reliability of computerized concussion assessment programs. Journal of Athletic Training, 42, 509514.
Broglio, S.P., Macciocchi, S.N., & Ferrara, M.S. (2007). Sensitivity of the concussion assessment battery. Neurosurgery, 60, 10501057. doi:10.1227/01.NEU.0000255479.90999.C0
Broglio, S.P., Pontifex, M.B., O’Connor, P., & Hillman, C.H. (2009). The persistent effects of concussion on neuroelectric indices of attention. Journal of Neurotrauma, 26, 14631470. doi:10.1089/neu.2008-0766
Broglio, S.P., & Puetz, T.W. (2008). The effect of sport concussion on neurocognitive function, self-report symptoms and postural control: A meta-analysis. Sports Medicine, 38, 5367.
Bruce, J., Echemendia, R., Meeuwisse, W., Comper, P., & Sisco, A. (2014). 1 year test-retest reliability of ImPACT in professional ice hockey players. The Clinical Neuropsychologist, 28, 1425. doi:10.1080/13854046.2013.866272
Cernich, A., Reeves, D., Sun, W., & Bleiberg, J. (2007). Automated Neuropsychological Assessment Metrics sports medicine battery. Archives of Clinical Neuropsychology, 22(Suppl. 1), S101S114. doi:10.1016/j.acn.2006.10.008
Chelune, G.J., Naugle, R.I., Lüders, H., Sedlak, J., & Awad, I.A. (1993). Individual change after epilepsy surgery: Practice effects and base-rate information. Neuropsychology, 7, 4152.
Cole, W.R., Arrieux, J.P., Schwab, K., Ivins, B.J., Qashu, F.M., & Lewis, S.C. (2013). Test-retest reliability of four computerized neurocognitive assessment tools in an active duty military population. Archives of Clinical Neuropsychology, 28, 732742. doi:10.1093/arclin/act040
Collie, A., Darby, D., & Maruff, P. (2001). Computerised cognitive assessment of athletes with sports related head injury. British Journal of Sports Medicine, 35, 297302.
Collie, A., Maruff, P., Makdissi, M., McCrory, P., McStephen, M., & Darby, D. (2003). CogSport: Reliability and correlation with conventional cognitive tests used in postconcussion medical evaluations. Clinical Journal of Sport Medicine, 13, 2832.
Covassin, T., Elbin, R. III, & Stiller-Ostrowski, J.L. (2009). Current sport-related concussion teaching and clinical practices of sports medicine professionals. Journal of Athletic Training, 44, 400404. doi:10.4085/1062-6050-44.4.400
Covassin, T., Elbin, R.J. III, Stiller-Ostrowski, J.L., & Kontos, A.P. (2009). Immediate post-concussion assessment and cognitive testing (ImPACT) practices of sports medicine professionals. Journal of Athletic Training, 44, 639644. doi:10.4085/1062-6050-44.6.639
Crawford, J.R., Garthwaite, P.H., & Gault, C.B. (2007). Estimating the percentage of the population with abnormally low scores (or abnormally large score differences) on standardized neuropsychological test batteries: A generic method with applications. Neuropsychology, 21, 419430. doi:10.1037/0894-4105.21.4.419
Derogatis, L.R. (2001). Brief Symptom Inventory 18 (BSI-18): Administration, scoring, and procedures manual. Bloomington, MN: Pearson.
Dettwiler, A., Murugavel, M., Putukian, M., Cubon, V., Furtado, J., & Osherson, D. (2014). Persistent differences in patterns of brain activation after sports-related concussion: A longitudinal functional magnetic resonance imaging study. Journal of Neurotrauma, 31, 180188. doi:10.1089/neu.2013.2983
Diener, E., Emmons, R.A., Larsen, R.J., & Griffin, S. (1985). The satisfaction with life scale. Journal of Personality Assessment, 49, 7175. doi:10.1207/s15327752jpa4901_13
Echemendia, R.J., Iverson, G.L., McCrea, M., Macciocchi, S.N., Gioia, G.A., Putukian, M., & Comper, P. (2013). Advances in neuropsychological assessment of sport-related concussion. British Journal of Sports Medicine, 47, 294298. doi:10.1136/bjsports-2013-092186
Eckner, J.T., Kutcher, J.S., & Richardson, J.K. (2011). Between-seasons test-retest reliability of clinically measured reaction time in National Collegiate Athletic Association Division I athletes. Journal of Athletic Training, 46, 409414.
Elbin, R.J., Schatz, P., & Covassin, T. (2011). One-year test-retest reliability of the online version of ImPACT in high school athletes. American Journal of Sports Medicine, 39, 23192324. doi:10.1177/0363546511417173
Green, P. (2003). Green’s Medical Symptom Validity Test for Windows. Edmonton, Alberta, Canada: Green’s Publishing, Inc.
Guskiewicz, K.M., Ross, S.E., & Marshall, S.W. (2001). Postural stability and neuropsychological deficits after concussion in collegiate athletes. Journal of Athletic Training, 36, 263273.
Helmick, K., Guskiewicz, K., Barth, J., Cantu, R., Kelly, J., McDonald, E., & Warden, D. (2006). Defense and Veterans Brain Injury Center Working Group on the acute management of mild traumatic brain injury in military operational settings: Clinical practice guideline and recommendations. Washington, DC: Defense and Veteran Brain Injury Center. Retrieved from http://www.pdhealth.mil/downloads/clinical_practice_guideline_recommendations.pdf
Iverson, G.L., Brooks, B.L., Collins, M.W., & Lovell, M.R. (2006). Tracking neuropsychological recovery following concussion in sport. Brain Injury, 20, 245252. doi:10.1080/02699050500487910
Iverson, G.L., Lovell, M.R., & Collins, M.W. (2003). Interpreting change on ImPACT following sport concussion. Clinical Neuropsychologist, 17, 460467. doi:10.1076/clin.17.4.460.27934
Jacobson, N.S., & Truax, P. (1991). Clinical significance: A statistical approach to defining meaningful change in psychotherapy research. Journal of Consulting and Clinical Psychology, 59, 1219.
LaRoche, A.A., Nelson, L.D., Connelly, P.K., Walter, K.D., & McCrea, M.A. (2015). Sport-related concussion reporting and state legislative effects. Clinical Journal of Sport Medicine. doi:10.1097/JSM.0000000000000192
Lichtenstein, J.D., Moser, R.S., & Schatz, P. (2014). Age and test setting affect the prevalence of invalid baseline scores on neurocognitive tests. American Journal of Sports Medicine, 42, 479484. doi:10.1177/0363546513509225
Louey, A.G., Cromer, J.A., Schembri, A.J., Darby, D.G., Maruff, P., Makdissi, M., &McCrory, P. (2014). Detecting cognitive impairment after concussion: Sensitivity of change from baseline and normative data methods using the CogSport/Axon cognitive test battery. Archives of Clinical Neuropsychology, 29, 432441. doi:10.1093/arclin/acu020
MacDonald, J., & Duerson, D. (2015). Reliability of a computerized neurocognitive test in baseline concussion testing of high school athletes. Clinical Journal of Sport Medicine, 25, 367372. doi:10.1097/JSM.0000000000000139
McClincy, M.P., Lovell, M.R., Pardini, J., Collins, M.W., & Spore, M.K. (2006). Recovery from sports concussion in high school and collegiate athletes. Brain Injury, 20, 3339. doi:10.1080/02699050500309817
McCrea, M., Guskiewicz, K.M., Marshall, S.W., Barr, W., Randolph, C., Cantu, R.C., & Kelly, J.P. (2003). Acute effects and recovery time following concussion in collegiate football players: The NCAA Concussion Study. Journal of the American Medical Association, 290, 25562563. doi:10.1001/jama.290.19.2556
McCrea, M., Kelly, J.P., Randolph, C., Kluge, J., Bartolic, E., Finn, G., & Baxter, B. (1998). Standardized assessment of concussion (SAC): On-site mental status evaluation of the athlete. Journal of Head Trauma Rehabilitation, 13, 2735.
McCrory, P., Meeuwisse, W., Aubry, M., Cantu, B., Dvorak, J., Echemendia, R.J., & Tator, C.H. (2013). Consensus statement on concussion in sport--the 4th International Conference on Concussion in Sport held in Zurich, November 2012. Clinical Journal of Sport Medicine, 23, 89117. doi:10.1097/JSM.0b013e31828b67cf
Meehan, W.P. III, d’Hemecourt, P., Collins, C.L., Taylor, A.M., & Comstock, R.D. (2012). Computerized neurocognitive testing for the management of sport-related concussions. Pediatrics, 129, 3844. doi:10.1542/peds.2011-1972
Mondello, S., Schmid, K., Berger, R., Kobeissy, F., Italiono, D., Jeromin, A., & Buki, A. (2014). The challenge of mild traumatic brain injury: Role of biochemical markers in diagnosis of brain damage. Medical Research Reviews, 34, 503531.
Moser, R.S., Iverson, G.L., Echemendia, R.J., Lovell, M.R., Schatz, P., Webbe, F.M., & Barth, J.T. (2007). Neuropsychological evaluation in the diagnosis and management of sports-related concussion. Archives of Clinical Neuropsychology, 22, 909916. doi:10.1016/j.acn.2007.09.004
Moser, R.S., Schatz, P., & Lichtenstein, J.D. (2015). The importance of proper administration and interpretation of neuropsychological baseline and postconcussion computerized testing. Applied Neuropsychology. Child, 4, 4148. doi:10.1080/21622965.2013.791825
Moser, R.S., Schatz, P., Neidzwski, K., & Ott, S.D. (2011). Group versus individual administration affects baseline neurocognitive test performance. American Journal of Sports Medicine, 39, 23252330. doi:10.1177/0363546511417114
Nakayama, Y., Covassin, T., Schatz, P., Nogle, S., & Kovan, J. (2014). Examination of the test-retest reliability of a computerized neurocognitive test battery. American Journal of Sports Medicine, 42, 20002005. doi:10.1177/0363546514535901
Nelson, L.D. (in press). False positive rates of reliable change indices for concussion test batteries: A Monte Carlo simulation. Journal of Athletic Training.
Nelson, L.D., Janecek, J.K., & McCrea, M.A. (2013). Acute clinical recovery from sport-related concussion. Neuropsychology Review, 23, 285299. doi:10.1007/s11065-013-9240-7
Nelson, L.D., Pfaller, A.Y., Rein, L., & McCrea, M.A. (2015). Rates and predictors of invalid baseline test performance for three computerized neurocognitive tests (CNTs): ANAM, Axon, and ImPACT. American Journal of Sports Medicine, 43, 20182026. doi:10.1177/0363546515587714
Prichep, L.S., McCrea, M., Barr, W., Powell, M., & Chabot, R.J. (2013). Time course of clinical and electrophysiological recovery after sport-related concussion. Journal of Head Trauma Rehabilitation, 28, 266273. doi:10.1097/HTR.0b013e318247b54e
Rahman-Filipiak, A.A.M., & Woodward, J.L. (2014). Administration and environment considerations in computer-based sports-concussion assessment. Neuropsychology Review, 23, 314334.
Randolph, C., McCrea, M., & Barr, W.B. (2005). Is neuropsychological testing useful in the management of sport-related concussion? Journal of Athletic Training, 40, 139152.
Register-Mihalik, J.K., Guskiewicz, K.M., Mihalik, J.P., Schmidt, J.D., Kerr, Z.Y., & McCrea, M.A. (2012). Reliable change, sensitivity, and specificity of a multidimensional concussion assessment battery: Implications for caution in clinical practice. Journal of Head Trauma Rehabilitation. doi:10.1097/HTR.0b013e3182585d37
Register-Mihalik, J.K., Guskiewicz, K.M., Mihalik, J.P., Schmidt, J.D., Kerr, Z.Y., & McCrea, M.A. (2013). Reliable change, sensitivity, and specificity of a multidimensional concussion assessment battery: Implications for caution in clinical practice. Journal of Head Trauma Rehabilitation, 28, 274283. doi:10.1097/HTR.0b013e3182585d37
Register-Mihalik, J.K., Kontos, D.L., Guskiewicz, K.M., Mihalik, J.P., Conder, R., & Shields, E.W. (2012). Age-related differences and reliability on computerized and paper-and-pencil neurocognitive assessment batteries. Journal of Athletic Training, 47, 297305. doi:10.4085/1062-6050-47.3.13
Resch, J.E., Driscoll, A., McCaffrey, N., Brown, C., Ferrara, M.S., Macciocchi, S., & Walpert, K. (2013). ImPact test-retest reliability: Reliably unreliable? Journal of Athletic Training, 48, 506511. doi:10.4085/1062-6050-48.3.09
Resch, J.E., McCrea, M.A., & Cullum, C.M. (2013). Computerized neurocognitive testing in the management of sport-related concussion: An update. Neuropsychology Review, 23, 335349. doi:10.1007/s11065-013-9242-5
Rousson, V., Gasser, T., & Seifert, B. (2002). Assessing intrarater, interrater and test-retest reliability of continuous measurements. Statistics in Medicine, 21, 34313446. doi:10.1002/sim.1253
Schatz, P. (2010). Long-term test-retest reliability of baseline cognitive assessments using ImPACT. American Journal of Sports Medicine, 38, 4753. doi:10.1177/0363546509343805
Schatz, P., & Ferris, C.S. (2013). One-month test-retest reliability of the ImPACT test battery. Archives of Clinical Neuropsychology, 28, 499504. doi:10.1093/arclin/act034
Schatz, P., Pardini, J.E., Lovell, M.R., Collins, M.W., & Podell, K. (2006). Sensitivity and specificity of the ImPACT Test Battery for concussion in athletes. Archives of Clinical Neuropsychology, 21, 9199. doi:10.1016/j.acn.2005.08.001
Schatz, P., & Sandel, N. (2013). Sensitivity and specificity of the online version of ImPACT in high school and collegiate athletes. American Journal of Sports Medicine, 41, 321326. doi:10.1177/0363546512466038
Schretlen, D.J., Testa, S.M., Winicki, J.M., Pearlson, G.D., & Gordon, B. (2008). Frequency and bases of abnormal performance by healthy adults on neuropsychological testing. Journal of the International Neuropsychological Society, 14, 436445. doi:10.1017/S1355617708080387
Segalowitz, S.J., Mahaney, P., Santesso, D.L., MacGregor, L., Dywan, J., & Willer, B. (2007). Retest reliability in adolescents of a computerized neuropsychological battery used to assess recovery from concussion. NeuroRehabilitation, 22, 243251.
Sim, A., Terryberry-Spohr, L., & Wilson, K.R. (2008). Prolonged recovery of memory functioning after mild traumatic brain injury in adolescent athletes. Journal of Neurosurgery, 108, 511516. doi:10.3171/JNS/2008/108/3/0511
Straume-Naesheim, T.M., Andersen, T.E., & Bahr, R. (2005). Reproducibility of computer based neuropsychological testing among Norwegian elite football players. British Journal of Sports Medicine, 39(Suppl. 1), i64i69. doi:10.1136/bjsm.2005.019620
Thomas, D.G., Apps, J.N., Hoffmann, R.G., McCrea, M., & Hammeke, T. (2015). Benefits of strict rest after acute concussion: A randomized controlled trial. Pediatrics, 135, 213223. doi:10.1542/peds.2014-0966
Van Kampen, D.A., Lovell, M.R., Pardini, J.E., Collins, M.W., & Fu, F.H. (2006). The “value added” of neurocognitive testing after sports-related concussion. American Journal of Sports Medicine, 34, 16301635. doi:10.1177/0363546506288677
Vincent, A.S., Roebuck-Spencer, T., Lopez, M.S., Twillie, D.A., Logan, B.W., Grate, S.J., & Gilliland, K. (2012). Effects of military deployment on cognitive functioning. Military Medicine, 177, 248255.
Wechsler, D. (2001). Wechsler Test of Adult Reading: WTAR. San Antonio, TX: The Psychological Corporation.
Weir, J.P. (2005). Quantifying test-retest reliability using the intraclass correlation coefficient and the SEM. Journal of Strength & Conditioning Research, 19, 231240. doi:10.1519/15184.1
Yuh, E.L., Hawryluk, G.W., & Manley, G.T. (2014). Imaging concussion: A review. Neurosurgery, 75(Suppl. 4), S50S63. doi:10.1227/NEU.0000000000000491
Zhu, D.C., Covassin, T., Nogle, S., Doyle, S., Russell, D., Pearson, R.L., & Kaufman, D.I. (2015). A potential biomarker in sports-related concussion: Brain functional connectivity alteration of the default-mode network measured with longitudinal resting-state fMRI over thirty days. Journal of Neurotrauma, 32, 327341. doi:10.1089/neu.2014.3413

Keywords

Type Description Title
WORD
Supplementary materials

Nelson Supplementary Material
Tables S1-S8

 Word (69 KB)
69 KB

Metrics

Altmetric attention score

Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed