Skip to main content Accessibility help
  • Print publication year: 2011
  • Online publication date: June 2012

2 - Evaluating Cognitive Models in Large-Scale Educational Assessments


In educational assessment, an idea whose time has come is that tests can be designed according to advances in the learning sciences. Currently, researchers and practitioners are studying how learning science outcomes can inform the design, development, and use of large-scale educational assessments. To date, however, there are few examples of large-scale educational tests designed from a learning science perspective and, hence, few empirical studies exist to evaluate the strengths and weaknesses of this approach. Moreover, developing the first generation of assessments using outcomes from the learning sciences is hampered by the limited information available on the types of cognitive models that can facilitate the design, validation, and administration of educational tests. Yet the role of cognitive models is paramount to the success of this endeavor. The National Research Council (NRC, 2001) asserted in their seminal publication, Knowing What Students Know: The Science and Design of Educational Assessments, that cognitive models have a foundational role in educational assessment that is currently poorly developed and under-valued:

A model of cognition and learning, or a description of how people represent knowledge and develop competence in a subject domain, is a cornerstone of the assessment development enterprise. Unfortunately, the model of learning is not made explicit in most assessment development efforts, is not empirically derived, and/or is impoverished relative to what it could be. (p. 176)

,Alberta Education (2007). The Alberta K–9 mathematics program of studies with achievement indicators. Edmonton, AB: Alberta Education.
,Alberta Education (2008). Alberta provincial achievement testing, subject bulletin 2008–2009: Grade 3 mathematics. Edmonton, AB: Alberta Education.
,American Educational Research Association, American Psychological Association, & National Council on Measurement in Education (1999). Standards for educational and psychological testing. Washington DC: Author.
Bloom, B., Englehart, M. Furst, E., Hill, W., & Krathwohl, D. (1956). Taxonomy of educational objectives: The classification of educational goals. Handbook I: Cognitive domain. New York: Longmans, Green.
Brown, J.S. & Burton, R.R. (1978). Diagnostic models for procedural bugs in basic mathematics skills. Cognitive Science, 2, 155–192.
Busemeyer, J.R. & Diederich, A. (2010). Cognitive modeling. Thousand Oaks, CA: Sage.
Case, R. (1992). The mind's staircase: Exploring the conceptual underpinnings of children's thought and knowledge. Hillsdale, NJ: Erlbaum.
Case, R. (1996). Reconceptualizing the nature of children's conceptual structures and their development in middle childhood. Monographs of the Society for Research in Child Development, 61 (1–26), Serial 246.
Case, R. & Griffin, S., (1990). Child cognitive development: The role of central conceptual structures in the development of scientific and social thought. In Hauert, E. A. (Ed.), Developmental psychology: Cognitive, perceptuo-motor, and neurological perspectives (pp. 193–230). North-Holland: Elsevier.
Chi, M.T.H. (1997). Quantifying qualitative analyses of verbal data: A practical guide. The Journal of the Learning Sciences, 6, 271–315.
Dawson, M.R.W. (1998). Understanding cognitive science. Blackwell.
Dawson, M.R.W. (2004). Minds and machines: Connectionism and psychological modeling. Malden, MA: Blackwell.
Downing, S.M. & Haladyna, T.M. (2006). Handbook of test development. Mahwah, NJ: Erlbaum.
Ericsson, K.A. & Simon, H.A. (1993). Protocol analysis: Verbal reports as data. Cambridge, MA: The MIT Press.
Gierl, M.J. (1997). Comparing the cognitive representations of test developers and students on a mathematics achievement test using Bloom's taxonomy. Journal of Educational Research, 91, 26–32.
Gierl, M.J., Alves, C., Roberts, M., & Gotzmann, A. (2009, April). Using judgments from content specialists to develop cognitive models for diagnostic assessments. In Gorin, J. (Chair), How to build a cognitive model for educational assessments. Symposium conducted at the meeting of the National Council on Measurement in Education, San Diego, CA.
Huff, K. & Goodman, D. (2007). The demand for cognitive diagnostic assessment. In Leighton, J. P. & Gierl, M. J. (Eds.), Cognitive diagnostic assessment for education: Theory and applications (pp. 19–60). New York: NY: Cambridge University Press.
Griffin, S. (2002). The development of math competence in the preschool and early school years: Cognitive foundations and instructional strategies. In Royer, J. (Ed.), Mathematical cognition (pp. 1–32). Greenwich, CT: Information Age Publishing.
Griffin, S. (2004). Building number sense with number worlds: A mathematics program for young children. Early Childhood Research Quarterly, 19, 173–180.
Griffin, S. & Case, R. (1997). Re-thinking the primary school math curriculum: An approach based on cognitive science. Issues in Education, 3, 1–49.
Griffin, S., Case, R., & Sandieson, R. (1992). Synchrony and asynchrony in the acquisition of children's everyday mathematical knowledge. In Case, R. (Ed.), The mind's staircase: Exploring the conceptual underpinnings of children's thought and knowledge (pp. 75–97). Hillsdale, NJ: Erlbaum.
Griffin, S., Case, R., & Siegler, R. (1994). Rightstart: Providing the central conceptual prerequisites for first formal learning of arithmetic to students at-risk for school failure. In McGilly, K. (Ed.), Classroom lessons: Integrating cognitive theory and classroom practice (pp.24–49). Cambridge, MA: Bradford Books MIT Press.
Leighton, J.P. (2004). Avoiding misconceptions, misuse, and missed opportunities: The collection of verbal reports in educational achievement testing. Educational Measurement: Issues and Practice, 4, 1–10.
Leighton, J.P. & Gierl, M.J. (2007a). Defining and evaluating models of cognition used in educational measurement to make inferences about examinees' thinking processes. Educational Measurement: Issues and Practice, 26, 3–16.
Leighton, J.P. & Gierl, M.J. (2007b). Verbal reports as data for cognitive diagnostic assessment. In Leighton, J. P. & Gierl, M. J. (Eds.), Cognitive diagnostic assessment for education: Theory and applications. (pp. 146–172). Cambridge, UK: Cambridge University Press.
Leighton, J.P. & Gokiert, R.J. (2008). Identifying test item misalignment using verbal reports of item misinterpretation and uncertainty. Educational Assessment, 13, 215–242.
Lohman, D. & Nichols, P. (1990). Training spatial abilities: Effects of practice on rotation and synthesis tasks. Learning and Individual Differences, 2, 67–93.
Marr, D. (1982). Vision. San Francisco, CA: W. H. Freeman.
,National Research Council (2001). Knowing what students know: The science and design of educational assessment. Committee on the Foundations of Assessment. Pellegrino, J., Chudowsky, N., and Glaser, R. (Eds.). Board on Testing and Assessment, Center for Education. Washington, DC: National Academy Press.
Newell, A. & Simon, H.A. (1972). Human problem solving. NJ: Prentice Hall.
Norris, S.P., Leighton, J.P., & Phillips, L.M. (2004). What is at stake in knowing the content and capabilities of children's minds? A case for basing high stakes tests on cognitive models. Theory and Research in Education, 2, 283–308.
Okamoto, Y. & Case, R. (1996). Exploring the microstructure of children's central conceptual structures in the domain of number. Monographs of the Society for Research in Child Development, 61 (27–58), Serial 246.
Poggio, A., Clayton, D.B., Glasnapp, D., Poggio, J., Haack, P., & Thomas, J. (April, 2005). Revisiting the item format question: Can the multiple choice format meet the demand for monitoring higher-order skills? Paper presented at the annual meeting of the National Council on Measurement in Education, Montreal, Canada.
Schmeiser, C.B. & Welch, C.J. (2006). Test development. In Brennan, R. L. (Ed.), Educational measurement (4th edition, pp. 307–353). Westport, CT: Praeger.
Snow, R.E. & Lohman, D.F. (1989). Implications of cognitive psychology for educational measurement. In Linn, R. L. (Ed.), Educational measurement (3rd ed., 263–331). New York: American Council on Education, Macmillan.
Taylor, K.L. & Dionne, J-P. (2000). Accessing problem-solving strategy knowledge: The complementary use of concurrent verbal protocols and retrospective debriefing. Journal of Educational Psychology, 92, 413–425.
Wellman, H.M. & Lagattuta, K.H. (2004). Theory of mind for learning and teaching: The nature and role of explanation. Cognitive Development, 19, 479–497.