Skip to main content Accessibility help

We use cookies to distinguish you from other users and to provide you with a better experience on our websites. Close this message to accept cookies or find out how to manage your cookie settings.

Close cookie message
×
Hostname: page-component-8448b6f56d-xtgtn Total loading time: 0 Render date: 2024-04-20T02:58:23.996Z Has data issue: false hasContentIssue false

10 - The Fusion Model Skills Diagnosis System

Published online by Cambridge University Press:  23 November 2009

By
Louis A. Roussos ,
Louis V. DiBello ,
William Stout ,
Sarah M. Hartz ,
Robert A. Henson  and
Jonathan L. Templin
Edited by
Jacqueline Leighton  and
Mark Gierl
Louis A. Roussos
Affiliation:
Senior Psychometrician, Measured Progress
Louis V. DiBello
Affiliation:
Research Professor of Psychology, University of Illinois at Chicago
William Stout
Affiliation:
Professor of Statistics, University of Illinois at Urbana-Champaign
Sarah M. Hartz
Affiliation:
Resident, Department of Psychiatry, University of Iowa
Robert A. Henson
Affiliation:
Assistant Professor of Education Research and Methodology, University of North Carolina at Greensboro
Jonathan L. Templin
Affiliation:
Assistant Professor of Psychology, University of Kansas
Jacqueline Leighton
Affiliation:
University of Alberta
Mark Gierl
Affiliation:
University of Alberta
Get access

Summary

INTRODUCTION

There is a long history of calls for combining cognitive science and psychometrics (Cronbach, 1975; Snow & Lohman, 1989). The U.S. standards movement, begun more than 20 years ago (McKnight et al., 1987; National Council of Teachers of Mathematics, 1989), sought to articulate public standards for learning that would define and promote successful performance by all students; establish a common base for curriculum development and instructional practice; and provide a foundation for measuring progress for students, teachers and programs. The standards movement provided the first widespread call for assessment systems that directly support learning. For success, such systems must satisfy a number of conditions having to do with cognitive-science–based design, psychometrics, and implementation. This chapter focuses on the psychometric aspects of one particular system that builds on a carefully designed test and a user-selected set of relevant skills measured by that test to assess student mastery of each of the chosen skills. This type of test-based skills level assessment is called skills diagnosis. The system that the chapter describes in detail is called the Fusion Model system.

This chapter focuses on the statistical and psychometric aspects of the Fusion Model system, with skills diagnosis researchers and practitioners in mind who may be interested in working with this system. We view the statistical and psychometric aspects as situated within a comprehensive framework for diagnostic assessment test design and implementation.

Type
Chapter
Information
Cognitive Diagnostic Assessment for Education
Theory and Applications
 , pp. 275 - 318
Publisher: Cambridge University Press
Print publication year: 2007

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Birenbaum, M., Tatsuoka, C., & Yamada, Y. (2004). Diagnostic assessment in TIMMS-R: Between countries and within country comparisons of eighth graders' mathematics performance. Studies in Educational Evaluation, 30, 151–173.Google Scholar
Carlin, B., Xia, H., Devine, O., Tollbert, P., & Mulholland, J. (1999). Spatio-temporal hierarchical models for analyzing Atlanta pediatric asthma ER visit rates. In Gatsonis, C., Kass, R., Carlin, B., Carriquiry, A., Gelman, A., Verdinelli, I., & West, M. (Eds.), Case studies in Bayesian statistics (Vol. IV, pp. 303–320). New York: Springer-Verlag.CrossRefGoogle Scholar
The Commission on Instructionally Supportive Assessment. (2001, October). Building tests to support instruction and accountability: A guide for policymakers. Retrieved November 29, 2006, from http://www.testaccountability.org/
Cowles, M.K., & Carlin, B.P. (1996). Markov chain Monte Carlo convergence diagnostics: A comparative review. Journal of the American Statistical Association, 91, 883–904.CrossRefGoogle Scholar
Cronbach, L.J. (1975). Beyond the two disciplines of scientific psychology. American Psychologist, 30, 116–127.CrossRefGoogle Scholar
Torre, J., & Douglas, J. A. (2004). Higher-order latent trait models for cognitive diagnosis. Psychometrika, 69, 333–353.CrossRefGoogle Scholar
DiBello, L.V., Stout, W.F., & Hartz, S.M. (2000, June). Model parameterization and identifiability issues. Paper presented at the international meeting of the Psychometric Society, University of British Columbia, Vancouver.Google Scholar
DiBello, L.V., Stout, W.F., & Roussos, L.A. (1995). Unified cognitive/psychometric diagnostic assessment likelihood-based classification techniques. In Nichols, P.D., Chipman, S.F., & Brennan, R.L. (Eds.), Cognitively diagnostic assessment (pp. 361–389). Mahwah, NJ: Erlbaum.Google Scholar
DiBello, L.V., Stout, W.F., Roussos, L.A., Henson, R.A., Templin, J.L., & Fife, J. (2006). Skill profiles for groups of students at a given NAEP scale level: Development and demonstration. Unpublished project report, Educational Testing Service, Princeton, NJ.Google Scholar
Dorans, N.J. (1989). Two new approaches to assessing differential item functioning: Standardization and the Mantel-Haenszel method. Applied Measurement in Education, 2, 217–233.CrossRefGoogle Scholar
Embretson, S. (1984). A general latent trait model for response processes. Psychometrika, 49, 175–186.CrossRefGoogle Scholar
Gelman, A., Carlin, J.B., Stern, H.S., & Rubin, D.R. (1995). Bayesian data analysis. London: Chapman & Hall.Google Scholar
Gelman, A., & Meng, X.-L. (1996). Model checking and model improvement. In Gilks, W.R., Richardson, S., & Spiegelhalter, D.J. (Eds.), Markov chain Monte Carlo in practice. Boca Raton, FL: Chapman & Hall/CRC.Google Scholar
Hambleton, R.K., & Swaminathan, H. (1985). Item response theory. New York: Kluwer-Nijhoff.CrossRefGoogle Scholar
Hartz, S.M. (2002). A Bayesian framework for the unified model for assessing cognitive abilities: Blending theory with practicality. Unpublished doctoral dissertation, University of Illinois, Champaign.Google Scholar
Hartz, S.M., & Roussos, L.A. (in press). The fusion model for skills diagnosis: Blending theory with practice (ETS Research Report). Princeton, NJ: Educational Testing Service.Google Scholar
Henson, R.A., He, X., & Roussos, L.A. (in press). Cognitive diagnosis attribute level discrimination indices. Applied Psychological Measurement.Google Scholar
Henson, R.A., Roussos, L.A., & Templin, J.L. (2004). Cognitive diagnostic “fit” indices. Unpublished ETS Project Report, Princeton, NJ.Google Scholar
Henson, R.A., Roussos, L.A., & Templin, J.L. (2005). Fusion model “fit” indices. Unpublished ETS Project Report, Princeton, NJ.Google Scholar
Henson, R.A., Stout, W.F., & Templin, J. (2005). Using cognitive model-based sum-scores to do skills diagnosis. Unpublished ETS External Team Project Report, Educational Testing Service, Princeton, NJ.Google Scholar
Henson, R.A., Templin, J.L., & Porch, F. (2004). Description of the underlying algorithm of the improved Arpeggio. Unpublished ETS External Team Project Report, Educational Testing Service, Princeton, NJ.Google Scholar
Henson, R.A., & Templin, J.L. (in press). Using model-based subscoring to do skills diagnosis. Journal of Educational Measurement.Google Scholar
Jang, E.E. (2005). A validity narrative: Effects of reading skills diagnosis on teaching and learning in the context of NG TOEFL. Unpublished doctoral dissertation, University of Illinois, Champaign.Google Scholar
Jang, E.E. (2006, April). Pedagogical implications of cognitive skills diagnostic assessment for teaching and learning. Paper presented at the annual meeting of the American Educational Research Association, San Francisco.Google Scholar
Jiang, H. (1996). Applications of computational statistics in cognitive diagnosis and IRT modeling. Unpublished doctoral dissertation, University of Illinois, Champaign.Google Scholar
Junker, B.W. (1999, November 30). Some statistical models and computational methods that may be useful for cognitively-relevant assessment. Prepared for the Committee on the Foundations of Assessment, National Research Council. Retrieved November 29, 2006, from http://www.stat.cmu.edu/∼brian/nrc/cfa/.Google Scholar
Junker, B. (2000). Some topics in nonparametric and parametric IRT, with some thoughts about the future. Unpublished manuscript, Carnegie Mellon University, Pittsburgh.Google Scholar
Landis, J.R., & Koch, G.G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33, 159–174.CrossRefGoogle ScholarPubMed
Maris, E. (1999). Estimating multiple classification latent class models. Psychometrika, 64, 187–212.CrossRefGoogle Scholar
McKnight, C., Crosswhite, F., Dossey, J., Kifer, E., Swafford, J., Travers, K., & Cooney, T. (1987). The underachieving curriculum: Assessing U.S. school mathematics from an international perspective. Champaign, IL: Stripes.Google Scholar
National Council of Teachers of Mathematics. (1989). Assessment standards for school mathematics. Reston, VA: Author.
Patz, R.J., & Junker, B. (1999a). A straightforward approach to Markov chain Monte Carlo methods for item response models. Journal of Educational and Behavioral Statistics, 24, 146–178.CrossRefGoogle Scholar
Patz, R.J., & Junker, B. (1999b). Applications and extensions of MCMC in IRT: Multiple item types, missing data, and rated responses. Journal of Educational and Behavioral Statistics, 24, 342–366.CrossRefGoogle Scholar
Pellegrino, J.W., & Chudowsky, N. (2003). The foundations of assessment. Measurement: Interdisciplinary Research and Perspectives, 1(2), 103–148.Google Scholar
Popham, W.J., Keller, T., Moulding, B., Pellegrino, J.W., & Sandifer, P. (2005). Instructionally supportive accountability tests in science: A viable assessment option?Measurement: Interdisciplinary Research and Perspectives, 3(3), 121–179.Google Scholar
Rasch, G. (1961). On general laws and the meaning of measurement in psychology. In Neyman, J. (Ed.), Proceedings of the fourth Berkeley symposium on mathematical statistics and probability (Vol. 4). Berkeley: University of California Press.Google Scholar
DiBello, L.V., Roussos, L.A., & Stout, W. (2007). Review of cognitively diagnostic assessment and a summary of psychometric models. In Rao, C.R. & Sinharay, S. (Eds.), Handbook of statistics: Volume 26, psychometrics (pp. 979–1030). Amsterdam: Elsevier.Google Scholar
Roussos, L.A., DiBello, L.V., Henson, R.A., Jang, E.E., & Templin, J.L. (in press). Skills diagnosis for education and psychology with IRT-based parametric latent class models. In Embretson, S. & Roberts, J. (Eds.), New directions in psychological measurement with model-based approaches. Washington, DC: American Psychological Association.Google Scholar
Samejima, F. (1994, April). Cognitive diagnosis using latent trait models. Paper presented at the annual meeting of the National Council on Measurement in Education, New Orleans.Google Scholar
Samejima, F. (1995). A cognitive diagnosis method using latent trait models: Competency space approach and its relationship with DiBello and Stout's unified cognitive-psychometric diagnosis model. In Nichols, P.D., Chipman, S.F., & Brennan, R.L. (Eds.), Cognitively diagnostic assessment (pp. 391–410). Mahwah, NJ: Erlbaum.Google Scholar
Shavelson, R.J., Black, P.J., William, D., & Coffey, J. (in press). On linking formative and summative functions in the design of large-scale assessment systems. Educational Evaluation and Policy Analysis.Google Scholar
Shepard, L.A. (2000). The role of classroom assessment in teaching and learning (CSE Technical Report 517). Los Angeles: National Center for Research on Evaluation, Standards, and Student Testing, University of California, Los Angeles.Google Scholar
Sinharay, S. (2004). Experiences with Markov chain Monte Carlo convergence assessment in two psycho-metric examples. Journal of Educational and Behavioral Statistics, 29, 461–488.CrossRefGoogle Scholar
Sinharay, S. (2005). Assessing of unidimensional IRT models using a Bayesian approach. Journal of Educational Measurement, 42, 375–394.CrossRefGoogle Scholar
Sinharay, S. (2006). Model diagnostics for Bayesian networks. Journal of Educational and Behavioral Statistics, 31, 1–33.CrossRefGoogle Scholar
Sinharay, S., & Johnson, M.S. (2003). Simulation studies applying posterior model checking for assessment of the common item response theory models. (ETS Research Report No. RR-03–28). Princeton, NJ: Educational Testing Service.Google Scholar
Snow, R.E., & Lohman, D.F. (1989). Implications of cognitive psychology for educational measurement. In Linn, R.L. (Ed.), Educational measurement (3rd ed., pp. 263–331). New York: Macmillan.Google Scholar
Tatsuoka, K.K. (1990). Toward an integration of item-response theory and cognitive error diagnoses. In Frederiksen, N., Glaser, R.L., Lesgold, A.M., & Shafto, M.G. (Eds.), Diagnostic monitoring of skill and knowledge acquisition. (pp 453–488). Hillsdale, NJ: Erlbaum.Google Scholar
Tatsuoka, K.K., Corter, J., & Guerrero, A. (2003). Manual of attribute-coding for general mathematics in TIMSS studies. New York: Columbia University, Teachers College.Google Scholar
Tatsuoka, K.K., Corter, J., & Tatsuoka, C. (2004). Patterns of diagnosed mathematical content and process skills in TIMMS-R across a sample of 20 countries. American Educational Research Journal, 41, 901–926.CrossRefGoogle Scholar
Xin, T. (2004). Latent trait and latent class models for multi-rater, multi-attribute agreement (UMI No. 3129052). Dissertation Abstracts International, 65(04), 1337A.Google Scholar
Xin, T., Xu, Z., & Tatsuoka, K.K. (2004). Linkage between teacher quality, student achievement, and cognitive skills: A rule-space model. Studies in Educational Evaluation, 30, 205–223.CrossRefGoogle Scholar

Save book to Kindle

To save this book to your Kindle, first ensure coreplatform@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about saving to your Kindle.

Note you can select to save to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be saved to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

Available formats
×

Save book to Dropbox

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Dropbox.

Available formats
×

Save book to Google Drive

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Google Drive.

Available formats
×