Skip to main content Accessibility help

Increasing the reliability of ipsative interpretations in neuropsychology: A comparison of reliable components analysis and other factor analytic methods



Ipsative approaches to neuropsychological assessment typically involve interpreting difference scores between individual test scores. The utility of these methods is limited by the reliability of neuropsychological difference scores and the number of comparisons between scores. The present study evaluated the utility of difference scores using factor analytic methods, including reliable components analysis (RCA), equally weighted composites and individual neuropsychological measures. Data from 1,364 individuals referred for neuropsychological assessment were factor analyzed and the resulting solutions were used to compute composite scores. Reliabilities and confidence intervals were derived for each method. Results indicated that RCA outperformed other factor analytic methods, but produced a slightly different factor structure. Difference scores derived using orthogonal solutions were slightly more reliable than oblique methods, and both were more reliable than those from equally weighted composites and individual measures. Confidence intervals for difference scores were considerably smaller for factor methods relative to those for individual test comparisons, due to the greater reliability of factor based difference scores and the smaller number of comparisons required. These findings suggest that difference scores derived from orthogonal factor solutions, particularly RCA solutions, may improve reliability for clinical assessment purposes. (JINS, 2004, 10, 578–589.)


Corresponding author

Reprint requests to: Thomas W. Frazier, Department of Psychiatry and Psychology (P57), Cleveland Clinic Foundation, 9500 Euclid Avenue, Cleveland, OH 44195. E-mail:,


Hide All


Bauer, R.M. (2000). The flexible battery approach to neuropsychological assessment. In R.D. Vanderploeg (Ed.), Clinician's guide to neuropsychological assessment (pp. 419448). Mahwah, NJ: Lawrence Erlbaum Associates.
Benton, A.L., Hamsher, K., & de S.Sivan, A.B. (1994). Multilingual aphasia examination. Iowa City, IA: AJA Associates.
Bowden, S.C., Fowler, K.S., Bell, R.C., Whelan, G., Clifford, C.C., Ritter, A.J., & Long, C.M. (1998). The reliability and internal validity of the Wisconsin Card Sorting Test. Neuropsychological Rehabilitation, 8, 243254.
Buja, A. & Eyuboglu, N. (1992). Remarks on parallel analysis. Multivariate Behavioral Research, 27, 509540.
Carroll, J.B. (1993). Human cognitive abilities: A survey of factor-analytic studies. New York: Cambridge University Press.
Caruso, J.C. (2001a). Increasing the reliability of the Fluid/Crystallized Difference Score from the Kaufman Adolescent and Adult Intelligence Test with reliable component analysis. Assessment, 8, 155166.
Caruso, J.C. (2001b). Reliable components analysis of the Stanford-Binet: Fourth Edition for 2- to 6-year olds. Psychological Assessment, 13, 261266.
Caruso, J.C. & Cliff, N. (1998). The factor structure of the WAIS–R: Replicability across age groups. Multivariate Behavioral Research, 33, 291308.
Caruso, J.C. & Cliff, N. (1999). The properties of equally and differentially weighted WAIS–III factor scores. Psychological Assessment, 11, 198206.
Caruso, J.C. & Cliff, N. (2000). Increasing the reliability of Wechsler Intelligence Scale for Children–Third edition difference scores with reliable component analysis. Psychological Assessment, 12, 8996.
Caruso, J.C. & Cliff, N. (2002). Reliable component analysis: A new/old method for exploratory data reduction. Unpublished manuscript, University of Montana at Missoula and University of Southern California at Los Angeles.
Caruso, J.C. & Witkiewitz, K. (2001). Memory and reasoning abilities assessed by the Universal Nonverbal Intelligence Test: A reliable component analysis (RCA) study. Educational and Psychological Measurement, 61, 522.
Caruso, J.C. & Witkiewitz, K. (2002). Increasing the reliability of ability–achievement difference scores: An example using the Kaufman Assessment Battery for Children. Journal of Educational Measurement, 39, 3958.
Cerhan, J.H., Ivnik, R.J., Smith, G.E., Tangalos, E.C., Petersen, R.C., & Boeve, B.F. (2002). Diagnostic utility of letter fluency, category fluency, and fluency difference scores in Alzheimer's disease. Clinical Peuropsychologist, 16, 3542.
Cliff, N. & Caruso, J.C. (1998). Reliable component analysis through maximizing composite reliability. Psychological Methods, 3, 291308.
Corey, D.M., Dunlap, W.P., & Burke, M.J. (1998). Averaging correlations: Expected values and bias in combining Pearson's r and Fisher's z transformation. Journal of General Psychology, 125, 245262.
Dikmen, S.S., Heaton, R.K., Grant, I., & Temkin, N.R. (1999). Test–retest reliability and practice effects of expanded Halstead-Reitan Neuropsychological Test Battery. Journal of the International Neuropsychological Society, 5, 346356.
Fabrigar, L.R., Wegener, D.T., MacCallum, R.C., & Strahan, E.J. (1999). Evaluating the use of exploratory factor analysis in psychological research. Psychological Methods, 4, 272299.
Fastenau, P.S., Denburg, N.L., & Mauer, B.A. (1998). Parallel short forms for the Boston Naming Test: Psychometric properties and norms for older adults. Journal of Clinical and Experimental Neuropsychology, 20, 828834.
Fennell, E.B. (1995). The role of neuropsychological assessment in learning disabilities. Journal of Child Neurology, 10, S36S41.
Franzen, M.D., Haut, M.W., Rankin, E., & Keefover, R. (1995). Empirical comparison of alternate forms of the Boston Naming Test. Clinical Neuropsychologist, 9, 225229.
Franzen, M.D., Paul, D., & Iverson, G.L. (1996). Reliability of alternate forms of the Trail Making Test. Clinical Neuropsychologist, 10, 125129.
Graham, J.W. & Schafer, J.L. (1999). On the performance of multiple imputation for multivariate data with small sample size. In R.H. Hoyle (Ed.), Statistical strategies for small sample research (pp. 129). Thousand Oaks, CA: Sage Publications, Inc.
Halstead, W.C. (1947). Brain and intelligence. Chicago: University of Chicago Press.
Heaton, R.K., Chelune, G.J., Talley, J.L., Kay, G.G., & Curtiss, G. (1993). Wisconsin Card Sorting Test manual: Revised and expanded. Odessa, FL: Psychological Assessment Resources.
Ingram, F., Greve, K.W., Ingram, P.T.F., & Soukup, V.M. (1999). Temporal stability of the Wisconsin Card Sorting Test in an untreated patient sample. British Journal of Clinical Psychology, 38, 209211.
Ivnik, R.J., Smith, G.E., Petersen, R.C., Boeve, B.F., Kokmen, E., & Tangalos, E.G. (2000). Diagnostic accuracy of four approaches to interpreting neuropsychological test data. Neuropsychology, 14, 163177.
Jacobson, M.W., Delis, D.C., Bondi, M.W., & Salmon, D.P. (2002). Do neuropsychological tests detect preclinical Alzheimer's disease: Individual–test versus cognitive-discrepancy score analyses. Neuropsychology, 16, 132139.
Johnstone, B. & Wilhelm, K.L. (1996). The longitudinal stability of the WRAT–R reading subtest: Is it an appropriate estimate of premorbid intelligence? Journal of the International Neuropsychological Society, 2, 282285.
Kaplan, E., Fein, D., Morris, R., & Delis, D.C. (1991). The WAIS–R as a neuropsychological instrument. Manual. San Antonio, TX: Psychological Corporation.
Kaplan, E., Goodglass, H., & Weintraub, S. (1983). The Boston Naming Test. Philadelphia: Lea and Febiger.
Kaufman, A.S. (1994). Intelligent testing with the WISC–III. New York: Wiley.
Keilp, J.G., Gorlyn, M., Alexander, G.E., Stern, Y., & Prohovnik, I. (1999). Cerebral blood flow patterns underlying the differential impairment in category vs. letter fluency in Alzheimer's disease. Neuropsychologia, 37, 12511261.
Kelley, T.L. (1927). Interpretation of educational measurements. Yonkers, NY: World Books.
Langeluddecke, P.M. & Lucas, S.K. (2003). Quantitative measures of memory malingering on the Wechsler Memory Scale–Third Edition in mild head injury litigants. Archives of Clinical Neuropsychology, 18, 181197.
Lees-Haley, P.R., Smith, H.H., Williams, C.W., & Dunn, J.T. (1996). Forensic neuropsychological test usage: An empirical survey. Archives of Clinical Neuropsychology, 11, 4551.
Lezak, M.D. (1995). Neuropsychological assessment (3rd ed.). New York: Oxford University Press.
Lord, F.M. & Novick, M.R. (1968). Statistical theories of mental test scores. Reading, MA: Addison-Wesley.
Macmann, G.M. & Barnett, D.W. (1997). Myth of the master detective: Reliability of interpretations for Kaufman's “intelligent testing” approach to the WISC–III. School Psychology Quarterly, 12, 197234.
Matarazzo, J.D. (1972). Measurement and appraisal of adult intelligence. Baltimore: Williams & Wilkins.
McCrae, R.R., Zonderman, A.B., Costa, P.T., Bond, M.H., & Paunonen, S.V. (1996). Evaluating Replicability of factors in the revised NEO Personality Inventory: Confirmatory factor analysis versus procrustes rotation. Journal of Personality and Social Psychology, 70, 552566.
McDermott, P.A., Fantuzzo, J.W., Glutting, J.J., Watkins, M.W., & Baggaley, A.R. (1992). Illusions of meaning in the ipsative assessment of children's ability. Journal of Special Education, 25, 504526.
Millis, S.R., Ross, S.R., & Ricker, J.H. (1998). Detection of incomplete effort on the Wechsler Adult Intelligence Scale–Revised: A cross-validation. Journal of Clinical and Experimental Neuropsychology, 20, 167173.
Mitrushina, M.N., Boone, K.B., & D'Elia, L.F. (1999). Handbook of normative data for neuropsychological assessment. New York: Oxford University Press.
Nunnally, J.C. & Bernstein, I.H. (1994). Psychometric theory (3rd ed.). New York: McGraw-Hill, Inc.
O'Connor, B.P. (2000). SPSS and SAS programs for determining the number of components using parallel analysis and Velicer's MAP test. Behavior Research Methods Instruments and Computers, 32, 396402.
Reitan, R.M. (1955). Investigation of the validity of Halstead's measures of biological intelligence. Archives of Neurology and Psychiatry, 73, 2835.
Reitan, R.M. (1958). Validity of the trail making test as an indicator of organic brain damage. Perceptual and Motor Skills, 8, 271276.
Russell, E.W. (2000). The cognitive-metric, fixed battery approach to neuropsychological assessment. In R.D. Vanderploeg (Ed.), Clinician's guide to neuropsychological assessment (pp. 449482). Mahwah, NJ: Lawrence Erlbaum Associates.
Schafer, J.L. (2002). Multiple imputation with PAN. In L. Collins & G. Aline (Eds.), New methods for the analysis of change (pp. 357377). Washington, DC: American Psychological Association.
Scott, J.G., Krull, K.R., Williamson, D.J.G., Adams, R.L., & Iverson, G.L. (1997). Oklahoma Premorbid Intelligence Estimation (OPIE): Utilization in clinical samples. Clinical Neuropsychologist, 11, 146154.
Soper, H.V., Cicchetti, D.V., Satz, P., Light, R., & Orsini, D.L. (1988). Null hypothesis disrespect in neuropsychology: Dangers of alpha and beta errors. Journal of Clinical and Experimental Neuropsychology, 10, 255270.
Spreen, O. & Benton, A.L. (1969). Neurosensory Center Comprehensive Examination for Aphasia: Manual of directions. Victoria, British Columbia, Canada: Neuropsychology Laboratory, University of Victoria.
Spreen, O. & Strauss, E. (1998). A compendium of neuropsychological tests: Administration, norms, and commentary (2nd ed.). New York: Oxford University Press.
SPSS, Inc. (2002). SPSS Professional Statistical Package (Version 11.0). Chicago: Author.
Streiner, D.L. & Norman, G.R. (1995). Health measurement scales: A practical guide to their development and use (2nd ed.). New York: Oxford University Press.
Tarter, R.E. & Edwards, K.L. (1986). Neuropsychological assessment. In T. Incagnoli, G. Goldstein, & C.J. Golden (Eds.), Clinical application of neuropsychological test batteries (pp. 135153). New York: Plenum Press.
Tate, R.L., Perdices, M., & Maggiotto, S. (1998). Stability of the Wisconsin Card Sorting Test and the determination of reliability of change in scores. Clinical Neuropsychologist, 12, 348357.
Tucker, L.R. (1951). A method for synthesis of factor analysis studies (Personnel Research Section Report No. 984). Washington, DC: Department of the Army.
Wechsler, D. (1997a). Manual for the Wechsler Adult Intelligence Scale–Third Edition. San Antonio, TX: The Psychological Corporation.
Wechsler, D. (1997b). Manual for the Wechsler Memory Scale–Third Edition. San Antonio, TX: The Psychological Corporation.
Wechsler, D. (1997c). WAIS–III WMS–III technical manual. San Antonio, TX: The Psychological Corporation.
Widaman, K.F. (1993). Common factor analysis versus principal component analysis: Differential bias in representing model parameters? Multivariate Behavioral Research, 28, 263311.
Wilde, N., Strauss, E., Chelune, G.J., Loring, D.W., Martin, R.C., Hermann, B.P., Sherman, E.M.S., & Hunter, M. (2001). WMS–III performance in patients with temporal lobe epilepsy: Group differences and individual classification. Journal of the International Neuropsychological Society, 7, 881891.
Wilkinson, G.S. (1993). The Wide Range Achievement Test–3. San Antonio, TX: The Psychological Corporation.
Williams, V.S.L., Jones, L.V., & Tukey, J.W. (1999). Controlling error in multiple comparisons, with examples from state-to-state differences in educational achievement. Journal of Educational and Behavioral Statistics, 24, 4269.
Zwick, W.R. & Velicer, W.F. (1986). Comparison of five rules for determining the number of components to retain. Psychological Bulletin, 99, 432442.



Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed