Skip to main content Accessibility help
×
Home
  • Print publication year: 2011
  • Online publication date: June 2012

7 - Cognitively Based Statistical Methods – Technical Illustrations

Summary

Research focused on the application of cognitive principles to assessment practices is thriving in educational measurement, particularly in the area of cognitively based statistical methods. Since the publication of key articles, chapters, and books two decades ago (e.g. Frederiksen, Glaser, Lesgold, & Shafto, 1990; Nichols, 1994; Nichols, Chipman, & Brennan, 1995; Ronning, Glover, Conoley, & Witt, 1990; Snow & Lohman, 1989), there has been clear recognition that inferences about examinees' knowledge and skills require detailed information on the organization, representation, and production of attributes from a cognitive model. The form that these models may take, their learning scientific foundations, and their applicability to the design and development of large-scale educational assessment are topics in this book. The desire to integrate cognition with assessment has also spawned a range of ambitious research activities designed to identify and evaluate examinees' knowledge and skills using new statistical methods. Many recent examples can be cited to document these focused research efforts. For example, an American Educational Research Association (AERA) special interest group, Cognition and Assessment, was formed in 2007 to provide a platform for scholars presenting cutting-edge research that combines the fields of cognitive psychology, cognitive science, educational psychology, educational assessment, and statistics to solve complex assessment problems using a multi-disciplinary approach. A special issue of the Journal of Educational Measurement was published in 2007 devoted to cognitively based statistical methods.

References
Almond, R. & Shute, A. (2009, April). Calibration of Bayesian network-based diagnostic assessment. In A. Rupp (Chair), Software for calibrating diagnostic classification models: An overview of the current state-of-the-art. Symposium conducted at the meeting of the American Educational Research Association, San Diego, CA.
Bolt, D., Chen, H., DiBello, L., Hartz, S., Henson, R., Roussos, L., Stout, W., & Templin, J. (April 2009). Cognitive diagnostic psychometric modelling: Fitting the fusion model with the Arpeggio system software. In A. Rupp (Chair), Software for calibrating diagnostic classification models: An overview of the current state-of-the-art. Symposium conducted at the meeting of the American Educational Research Association, San Diego, CA.
Cui, Y., Gierl, M.J., & Leighton, J.P. (2009, April). Estimating the attribute hierarchy method with Mathematica. In A. Rupp (Chair), Software for calibrating diagnostic classification models: An overview of the current state-of-the-art. Symposium conducted at the meeting of the American Educational Research Association, San Diego, CA.
Cui, Y. & Leighton, J.P. (2009). The hierarchy consistency index: Evaluating person fit for cognitive diagnostic assessment. Journal of Educational Measurement, 46, 429–449.
Torre, J. (2008, July). The generalized DINA model. Paper presented at the annual International Meeting of the Psychometric Society (IMPS), Durham, NH.
Torre, J. (2009, April). Estimation code for the G-DINA model. In A. Rupp (Chair), Software for calibrating diagnostic classification models: An overview of the current state-of-the-art. Symposium conducted at the meeting of the American Educational Research Association, San Diego, CA.
Torre, J. & Douglas, J.A. (2004). Higher-order latent trait models for cognitive diagnosis. Psychometrika, 69, 333–353.
Frederiksen, N., Glaser, R.L., Lesgold, A.M., & Shafto, M.G. (1990). Diagnostic monitoring of skills and knowledge acquisition. Hillsdale, NJ: Erlbaum.
Fu, J. & Li, Y. (2007, April). Cognitively diagnostic psychometric models: An integrated review. Paper presented at the annual meeting of the National Council on Measurement in Education, Chicago, IL.
Gierl, M.J., Cui, Y., & Hunka, S. (2008a). Using connectionist models to evaluate examinees' response patterns on tests. Journal of Modern Applied Statistical Methods, 7, 234–245.
Gierl, M.J., Cui, Y., & Zhou, J. (2009). Reliability of attribute-based scoring in cognitive diagnostic assessment. Journal of Educational Measurement, 46, 293–313.
Gierl, M.J., Wang, C., & Zhou, J. (2008b). Using the attribute hierarchy method to make diagnostic inferences about examinees' cognitive skills in algebra on the SAT©. Journal of Technology, Learning, and Assessment, 6 (6). Retrieved [date] from http://www.jtla.org.
Gierl, M.J., Alves, C., & Taylor-Majeau, R. (2010). Using the Attribute Hierarchy Method to make diagnostic inferences about examinees' skills in mathematics: An operational implementation of cognitive diagnostic assessment. International Journal of Testing, 10, 318–341.
Hartz, S. (2002). A Bayesian framework for the unified model for assessing cognitive abilities: Blending theory with practicality. Unpublished doctoral dissertation, University of Illinois at Urbana-Champaign, Urbana-Champaign, IL.
Henson, R.A., Templin, J.L., & Willse, J.T. (2008). Defining a family of cognitive diagnostic model using log-linear models with latent variables. Psychometrika, 74, 191–210.
Jang, E.E. (2009). Cognitive diagnostic assessment of L2 reading comprehension ability: Validity arguments for fusion model application to LanguEdge assessment. Language Testing, 26, 31–73.
Junker, B.W. & Sijtsma, K. (2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25, 258–272.
Lee, Y.W. & Sawaki, Y. (2009). Cognitive diagnosis approaches to language assessment: An overview. Language Assessment Quarterly, 6, 172–189.
Leighton, J.P. (2009). Where's the psychology?: A commentary on “Unique Characteristics of Diagnostic Classification Models: A Comprehensive Review of the Current State-of-the-Art.”Measurement: Interdisciplinary Research and Perspectives, 6 (4), 272–275.
Leighton, J.P. & Gierl, M.J. (2007a). Defining and evaluating models of cognition used in educational measurement to make inferences about examinees' thinking processes. Educational Measurement: Issues and Practice, 26, 3–16.
Leighton, J.P. & Gierl, M.J. (Eds.). (2007b). Cognitive diagnostic assessment for education: Theory and applications. Cambridge, UK: Cambridge University Press.
Leighton, J.P., Gierl, M.J., & Hunka, S. (2004). The attribute hierarchy method for cognitive assessment: A variation on Tatsuoka's rule-space approach. Journal of Educational Measurement, 41, 205–236.
Maris, E. (1995). Psychometric latent response models. Psychometrika, 60, 523–547.
McClelland, J.L. (1998). Connectionist models and Bayesian inference. In Oaksford, M. & Chater, N. (Eds.), Rational models of cognition (pp. 21–53). Oxford: Oxford University Press.
Mislevy, R.J. & Levy, R. (2007). Bayesian psychometric modeling from an evidence-centered design perspective. In Rao, C. R. & Sinharay, S. (Eds.), Handbook of statistics, Vol. 26 (pp. 839–865). Amsterdam, The Netherlands: Elsevier.
Nichols, P.D. (1994). A framework for developing cognitively diagnostic assessments. Review of Educational Research, 64, 575–603.
Nichols, P.D., Chipman, S.F., & Brennan, R.L. (1995). Cognitively diagnostic assessment. Hillsdale, NJ: Erlbaum.
Software, Norsys (2007). Netica manual. Http://www.norsys.com.
Pearl, J. (1988). Probabilistic reasoning in intelligent systems: Networks of plausible inference. San Mateo, CA: Kaufmann.
Ronning, R., Glover, J., Conoley, J.C., & Witt, J. (1990). The influence of cognitive psychology on testing and measurement: The Buros-Nebraska symposium on measurement and testing (Vol. 3.). Hillsdale, NJ: Erlbaum.
Roussos, L., DiBello, L.V., Stout, W., Hartz, S., Henson, R.A., & Templin, J.H. (2007). The fusion model skills diagnosis system. In Leighton, J. P., & Gierl, M. J. (Eds.), Cognitive diagnostic assessment for education: Theory and applications (pp. 275–318). Cambridge, UK: Cambridge University Press.
Rumelhart, D.E., Hinton, G.E., & Williams, R.J. (1986a). Learning representations by back-propagating errors. Nature, 323, 533–536.
Rumelhart, D.E., Hinton, G.E., & Williams, R.J. (1986b). Parallel distributed processing (Vol. 1). Cambridge, MA: MIT Press.
Rupp, A. A. (2009, April). Software for calibrating diagnostic classification models: An overview of the current state-of-the-art. Symposium conducted at the meeting of the American Educational Research Association, San Diego, CA.
Rupp, A. A. & Templin, J. (2008). Unique characteristics of diagnostic classification models: A comprehensive review of the current state-of-the-art. Measurement: Interdisciplinary Research and Perspectives, 6, 219–262.
Rupp, A.A., Templin, J.L., & Henson, R.A. (2010). Diagnostic measurement: Theory, methods, and applications. New York: The Guilford Press.
Snow, R.E. & Lohman, D.F. (1989). Implications of cognitive psychology for educational measurement. In Linn, R. L. (Ed.), Educational measurement (3rd ed., pp. 263–331). New York: American Council on Education, Macmillan.
Tatsuoka, K.K. (1983). Rule space: An approach for dealing with misconceptions based on item response theory. Journal of Educational Measurement, 20, 345–354.
Tatsuoka, K.K. (1995). Architecture of knowledge structures and cognitive diagnosis: A statistical pattern recognition and classification approach. In Nichols, P.D., Chipman, S.F., & Brennan, R.L. (Eds.), Cognitively diagnostic assessment (pp. 327–359). Hillsdale, NJ: Erlbaum.
Tatsuoka, K.K. & Tatsuoka, C. (2009, April). The rule space methodology: The Q matrix theory, rule space classification space and POSET model. In A. Rupp (Chair), Software for calibrating diagnostic classification models: An overview of the current state-of-the-art. Symposium conducted at the meeting of the American Educational Research Association, San Diego, CA.
Templin, J.L. (2006). CDM user's guide. Unpublished manuscript.
Templin, J.L. & Henson, R.A. (2006). Measurement of psychological disorders using cognitive diagnosis models. Psychological Methods, 11, 287–305.
Templin, J.L, Henson, R.A., Douglas, J., & Hoffman, L. (2009, April). Estimating log-linear diagnostic classification models with Mplus and SAS. In A. Rupp (Chair), Software for calibrating diagnostic classification models: An overview of the current state-of-the-art. Symposium conducted at the meeting of the American Educational Research Association, San Diego, CA.
Thissen, D. & Steinberg, L. (1986). A taxonomy of item response models. Psychometrika, 51, 567–577.
Davier, M. (2005). A general diagnostic model applied to language testing data (ETS Research Report No. RR-05–16). Princeton, NJ: Educational Testing Service.
Davier, M. (2007). Hierarchical general diagnostic models (Research Report No. RR-07–19). Princeton, NJ: Educational Testing Service.
Davier, M. & Xu, X. (2009, April). Estimating latent structure models (including diagnostic classification models) with mdltm: A software for multidimensional discrete latent traits models. In A. Rupp (Chair), Software for calibrating diagnostic classification models: An overview of the current state-of-the-art. Symposium conducted at the meeting of the American Educational Research Association, San Diego, CA.
West, P., Rutstein, D.W., Mislevy, R.J., Liu, J., Levy, R., DiCerbo, K.E., Crawford, A., Choi, Y., & Behrens, J.T. (2009, June). A bayes net approach to modeling learning progressions and task performances. Paper presented at the Learning Progression in Science Conference, Iowa City, IA.
Yan, D., Mislevy, R.J., & Almond, R.G. (2003). Design and analysis in a cognitive assessment (ETS Research Report No. RR-03–32). Princeton, NJ: Educational Testing Service.