Hostname: page-component-78c5997874-s2hrs Total loading time: 0 Render date: 2024-11-18T10:30:05.499Z Has data issue: false hasContentIssue false
  • English
  • Français

Increasing the reliability of ipsative interpretations in neuropsychology: A comparison of reliable components analysis and other factor analytic methods

Published online by Cambridge University Press:  01 July 2004

THOMAS W. FRAZIER ,
ERIC A. YOUNGSTROM ,
GORDON J. CHELUNE ,
RICHARD I. NAUGLE  and
TARA T. LINEWEAVER
THOMAS W. FRAZIER
Affiliation:
Department of Psychology, Case Western Reserve University, Cleveland, Ohio
ERIC A. YOUNGSTROM
Affiliation:
Department of Psychology, Case Western Reserve University, Cleveland, Ohio
GORDON J. CHELUNE
Affiliation:
Mellen Center for Multiple Sclerosis Treatment and Research, Cleveland Clinic Foundation, Cleveland, Ohio
RICHARD I. NAUGLE
Affiliation:
Section of Neuropsychology, Cleveland Clinic Foundation, Cleveland, Ohio
TARA T. LINEWEAVER
Affiliation:
Section of Neuropsychology, Cleveland Clinic Foundation, Cleveland, Ohio
Get access Rights & Permissions [Opens in a new window]

Abstract

Ipsative approaches to neuropsychological assessment typically involve interpreting difference scores between individual test scores. The utility of these methods is limited by the reliability of neuropsychological difference scores and the number of comparisons between scores. The present study evaluated the utility of difference scores using factor analytic methods, including reliable components analysis (RCA), equally weighted composites and individual neuropsychological measures. Data from 1,364 individuals referred for neuropsychological assessment were factor analyzed and the resulting solutions were used to compute composite scores. Reliabilities and confidence intervals were derived for each method. Results indicated that RCA outperformed other factor analytic methods, but produced a slightly different factor structure. Difference scores derived using orthogonal solutions were slightly more reliable than oblique methods, and both were more reliable than those from equally weighted composites and individual measures. Confidence intervals for difference scores were considerably smaller for factor methods relative to those for individual test comparisons, due to the greater reliability of factor based difference scores and the smaller number of comparisons required. These findings suggest that difference scores derived from orthogonal factor solutions, particularly RCA solutions, may improve reliability for clinical assessment purposes. (JINS, 2004, 10, 578–589.)

Keywords

IpsativeBattery interpretationReliable components analysisDifference scores
Type
Research Article
Information
Journal of the International Neuropsychological Society , Volume 10 , Issue 4 , July 2004 , pp. 578 - 589
Copyright
2004 The International Neuropsychological Society

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

REFERENCES

Bauer, R.M. (2000). The flexible battery approach to neuropsychological assessment. In R.D. Vanderploeg (Ed.), Clinician's guide to neuropsychological assessment (pp. 419–448). Mahwah, NJ: Lawrence Erlbaum Associates.
Benton, A.L., Hamsher, K., & de S.Sivan, A.B. (1994). Multilingual aphasia examination. Iowa City, IA: AJA Associates.
Bowden, S.C., Fowler, K.S., Bell, R.C., Whelan, G., Clifford, C.C., Ritter, A.J., & Long, C.M. (1998). The reliability and internal validity of the Wisconsin Card Sorting Test. Neuropsychological Rehabilitation, 8, 243–254.CrossRefGoogle Scholar
Buja, A. & Eyuboglu, N. (1992). Remarks on parallel analysis. Multivariate Behavioral Research, 27, 509–540.CrossRefGoogle Scholar
Carroll, J.B. (1993). Human cognitive abilities: A survey of factor-analytic studies. New York: Cambridge University Press.CrossRef
Caruso, J.C. (2001a). Increasing the reliability of the Fluid/Crystallized Difference Score from the Kaufman Adolescent and Adult Intelligence Test with reliable component analysis. Assessment, 8, 155–166.Google Scholar
Caruso, J.C. (2001b). Reliable components analysis of the Stanford-Binet: Fourth Edition for 2- to 6-year olds. Psychological Assessment, 13, 261–266.Google Scholar
Caruso, J.C. & Cliff, N. (1998). The factor structure of the WAIS–R: Replicability across age groups. Multivariate Behavioral Research, 33, 291–308.Google Scholar
Caruso, J.C. & Cliff, N. (1999). The properties of equally and differentially weighted WAIS–III factor scores. Psychological Assessment, 11, 198–206.CrossRefGoogle Scholar
Caruso, J.C. & Cliff, N. (2000). Increasing the reliability of Wechsler Intelligence Scale for Children–Third edition difference scores with reliable component analysis. Psychological Assessment, 12, 89–96.CrossRefGoogle Scholar
Caruso, J.C. & Cliff, N. (2002). Reliable component analysis: A new/old method for exploratory data reduction. Unpublished manuscript, University of Montana at Missoula and University of Southern California at Los Angeles.
Caruso, J.C. & Witkiewitz, K. (2001). Memory and reasoning abilities assessed by the Universal Nonverbal Intelligence Test: A reliable component analysis (RCA) study. Educational and Psychological Measurement, 61, 5–22.CrossRefGoogle Scholar
Caruso, J.C. & Witkiewitz, K. (2002). Increasing the reliability of ability–achievement difference scores: An example using the Kaufman Assessment Battery for Children. Journal of Educational Measurement, 39, 39–58.CrossRefGoogle Scholar
Cerhan, J.H., Ivnik, R.J., Smith, G.E., Tangalos, E.C., Petersen, R.C., & Boeve, B.F. (2002). Diagnostic utility of letter fluency, category fluency, and fluency difference scores in Alzheimer's disease. Clinical Peuropsychologist, 16, 35–42.CrossRefGoogle Scholar
Cliff, N. & Caruso, J.C. (1998). Reliable component analysis through maximizing composite reliability. Psychological Methods, 3, 291–308.CrossRefGoogle Scholar
Corey, D.M., Dunlap, W.P., & Burke, M.J. (1998). Averaging correlations: Expected values and bias in combining Pearson's r and Fisher's z transformation. Journal of General Psychology, 125, 245–262.CrossRefGoogle Scholar
Dikmen, S.S., Heaton, R.K., Grant, I., & Temkin, N.R. (1999). Test–retest reliability and practice effects of expanded Halstead-Reitan Neuropsychological Test Battery. Journal of the International Neuropsychological Society, 5, 346–356.Google Scholar
Fabrigar, L.R., Wegener, D.T., MacCallum, R.C., & Strahan, E.J. (1999). Evaluating the use of exploratory factor analysis in psychological research. Psychological Methods, 4, 272–299.CrossRefGoogle Scholar
Fastenau, P.S., Denburg, N.L., & Mauer, B.A. (1998). Parallel short forms for the Boston Naming Test: Psychometric properties and norms for older adults. Journal of Clinical and Experimental Neuropsychology, 20, 828–834.CrossRefGoogle Scholar
Fennell, E.B. (1995). The role of neuropsychological assessment in learning disabilities. Journal of Child Neurology, 10, S36–S41.CrossRefGoogle Scholar
Franzen, M.D., Haut, M.W., Rankin, E., & Keefover, R. (1995). Empirical comparison of alternate forms of the Boston Naming Test. Clinical Neuropsychologist, 9, 225–229.CrossRefGoogle Scholar
Franzen, M.D., Paul, D., & Iverson, G.L. (1996). Reliability of alternate forms of the Trail Making Test. Clinical Neuropsychologist, 10, 125–129.CrossRefGoogle Scholar
Graham, J.W. & Schafer, J.L. (1999). On the performance of multiple imputation for multivariate data with small sample size. In R.H. Hoyle (Ed.), Statistical strategies for small sample research (pp. 1–29). Thousand Oaks, CA: Sage Publications, Inc.
Halstead, W.C. (1947). Brain and intelligence. Chicago: University of Chicago Press.
Heaton, R.K., Chelune, G.J., Talley, J.L., Kay, G.G., & Curtiss, G. (1993). Wisconsin Card Sorting Test manual: Revised and expanded. Odessa, FL: Psychological Assessment Resources.
Ingram, F., Greve, K.W., Ingram, P.T.F., & Soukup, V.M. (1999). Temporal stability of the Wisconsin Card Sorting Test in an untreated patient sample. British Journal of Clinical Psychology, 38, 209–211.CrossRefGoogle Scholar
Ivnik, R.J., Smith, G.E., Petersen, R.C., Boeve, B.F., Kokmen, E., & Tangalos, E.G. (2000). Diagnostic accuracy of four approaches to interpreting neuropsychological test data. Neuropsychology, 14, 163–177.CrossRefGoogle Scholar
Jacobson, M.W., Delis, D.C., Bondi, M.W., & Salmon, D.P. (2002). Do neuropsychological tests detect preclinical Alzheimer's disease: Individual–test versus cognitive-discrepancy score analyses. Neuropsychology, 16, 132–139.CrossRefGoogle Scholar
Johnstone, B. & Wilhelm, K.L. (1996). The longitudinal stability of the WRAT–R reading subtest: Is it an appropriate estimate of premorbid intelligence? Journal of the International Neuropsychological Society, 2, 282–285.Google Scholar
Kaplan, E., Fein, D., Morris, R., & Delis, D.C. (1991). The WAIS–R as a neuropsychological instrument. Manual. San Antonio, TX: Psychological Corporation.
Kaplan, E., Goodglass, H., & Weintraub, S. (1983). The Boston Naming Test. Philadelphia: Lea and Febiger.
Kaufman, A.S. (1994). Intelligent testing with the WISC–III. New York: Wiley.
Keilp, J.G., Gorlyn, M., Alexander, G.E., Stern, Y., & Prohovnik, I. (1999). Cerebral blood flow patterns underlying the differential impairment in category vs. letter fluency in Alzheimer's disease. Neuropsychologia, 37, 1251–1261.Google Scholar
Kelley, T.L. (1927). Interpretation of educational measurements. Yonkers, NY: World Books.
Langeluddecke, P.M. & Lucas, S.K. (2003). Quantitative measures of memory malingering on the Wechsler Memory Scale–Third Edition in mild head injury litigants. Archives of Clinical Neuropsychology, 18, 181–197.Google Scholar
Lees-Haley, P.R., Smith, H.H., Williams, C.W., & Dunn, J.T. (1996). Forensic neuropsychological test usage: An empirical survey. Archives of Clinical Neuropsychology, 11, 45–51.CrossRefGoogle Scholar
Lezak, M.D. (1995). Neuropsychological assessment (3rd ed.). New York: Oxford University Press.
Lord, F.M. & Novick, M.R. (1968). Statistical theories of mental test scores. Reading, MA: Addison-Wesley.
Macmann, G.M. & Barnett, D.W. (1997). Myth of the master detective: Reliability of interpretations for Kaufman's “intelligent testing” approach to the WISC–III. School Psychology Quarterly, 12, 197–234.CrossRefGoogle Scholar
Matarazzo, J.D. (1972). Measurement and appraisal of adult intelligence. Baltimore: Williams & Wilkins.
McCrae, R.R., Zonderman, A.B., Costa, P.T., Bond, M.H., & Paunonen, S.V. (1996). Evaluating Replicability of factors in the revised NEO Personality Inventory: Confirmatory factor analysis versus procrustes rotation. Journal of Personality and Social Psychology, 70, 552–566.CrossRefGoogle Scholar
McDermott, P.A., Fantuzzo, J.W., Glutting, J.J., Watkins, M.W., & Baggaley, A.R. (1992). Illusions of meaning in the ipsative assessment of children's ability. Journal of Special Education, 25, 504–526.CrossRefGoogle Scholar
Millis, S.R., Ross, S.R., & Ricker, J.H. (1998). Detection of incomplete effort on the Wechsler Adult Intelligence Scale–Revised: A cross-validation. Journal of Clinical and Experimental Neuropsychology, 20, 167–173.CrossRefGoogle Scholar
Mitrushina, M.N., Boone, K.B., & D'Elia, L.F. (1999). Handbook of normative data for neuropsychological assessment. New York: Oxford University Press.
Nunnally, J.C. & Bernstein, I.H. (1994). Psychometric theory (3rd ed.). New York: McGraw-Hill, Inc.
O'Connor, B.P. (2000). SPSS and SAS programs for determining the number of components using parallel analysis and Velicer's MAP test. Behavior Research Methods Instruments and Computers, 32, 396–402.CrossRefGoogle Scholar
Reitan, R.M. (1955). Investigation of the validity of Halstead's measures of biological intelligence. Archives of Neurology and Psychiatry, 73, 28–35.CrossRefGoogle Scholar
Reitan, R.M. (1958). Validity of the trail making test as an indicator of organic brain damage. Perceptual and Motor Skills, 8, 271–276.CrossRefGoogle Scholar
Russell, E.W. (2000). The cognitive-metric, fixed battery approach to neuropsychological assessment. In R.D. Vanderploeg (Ed.), Clinician's guide to neuropsychological assessment (pp. 449–482). Mahwah, NJ: Lawrence Erlbaum Associates.
Schafer, J.L. (2002). Multiple imputation with PAN. In L. Collins & G. Aline (Eds.), New methods for the analysis of change (pp. 357–377). Washington, DC: American Psychological Association.
Scott, J.G., Krull, K.R., Williamson, D.J.G., Adams, R.L., & Iverson, G.L. (1997). Oklahoma Premorbid Intelligence Estimation (OPIE): Utilization in clinical samples. Clinical Neuropsychologist, 11, 146–154.CrossRefGoogle Scholar
Soper, H.V., Cicchetti, D.V., Satz, P., Light, R., & Orsini, D.L. (1988). Null hypothesis disrespect in neuropsychology: Dangers of alpha and beta errors. Journal of Clinical and Experimental Neuropsychology, 10, 255–270.CrossRefGoogle Scholar
Spreen, O. & Benton, A.L. (1969). Neurosensory Center Comprehensive Examination for Aphasia: Manual of directions. Victoria, British Columbia, Canada: Neuropsychology Laboratory, University of Victoria.
Spreen, O. & Strauss, E. (1998). A compendium of neuropsychological tests: Administration, norms, and commentary (2nd ed.). New York: Oxford University Press.
SPSS, Inc. (2002). SPSS Professional Statistical Package (Version 11.0). Chicago: Author.
Streiner, D.L. & Norman, G.R. (1995). Health measurement scales: A practical guide to their development and use (2nd ed.). New York: Oxford University Press.
Tarter, R.E. & Edwards, K.L. (1986). Neuropsychological assessment. In T. Incagnoli, G. Goldstein, & C.J. Golden (Eds.), Clinical application of neuropsychological test batteries (pp. 135–153). New York: Plenum Press.
Tate, R.L., Perdices, M., & Maggiotto, S. (1998). Stability of the Wisconsin Card Sorting Test and the determination of reliability of change in scores. Clinical Neuropsychologist, 12, 348–357.CrossRefGoogle Scholar
Tucker, L.R. (1951). A method for synthesis of factor analysis studies (Personnel Research Section Report No. 984). Washington, DC: Department of the Army.CrossRef
Wechsler, D. (1997a). Manual for the Wechsler Adult Intelligence Scale–Third Edition. San Antonio, TX: The Psychological Corporation.
Wechsler, D. (1997b). Manual for the Wechsler Memory Scale–Third Edition. San Antonio, TX: The Psychological Corporation.
Wechsler, D. (1997c). WAIS–III WMS–III technical manual. San Antonio, TX: The Psychological Corporation.
Widaman, K.F. (1993). Common factor analysis versus principal component analysis: Differential bias in representing model parameters? Multivariate Behavioral Research, 28, 263–311.Google Scholar
Wilde, N., Strauss, E., Chelune, G.J., Loring, D.W., Martin, R.C., Hermann, B.P., Sherman, E.M.S., & Hunter, M. (2001). WMS–III performance in patients with temporal lobe epilepsy: Group differences and individual classification. Journal of the International Neuropsychological Society, 7, 881–891.Google Scholar
Wilkinson, G.S. (1993). The Wide Range Achievement Test–3. San Antonio, TX: The Psychological Corporation.
Williams, V.S.L., Jones, L.V., & Tukey, J.W. (1999). Controlling error in multiple comparisons, with examples from state-to-state differences in educational achievement. Journal of Educational and Behavioral Statistics, 24, 42–69.CrossRefGoogle Scholar
Zwick, W.R. & Velicer, W.F. (1986). Comparison of five rules for determining the number of components to retain. Psychological Bulletin, 99, 432–442.CrossRefGoogle Scholar