In educational assessment, an idea whose time has come is that tests can be designed according to advances in the learning sciences. Currently, researchers and practitioners are studying how learning science outcomes can inform the design, development, and use of large-scale educational assessments. To date, however, there are few examples of large-scale educational tests designed from a learning science perspective and, hence, few empirical studies exist to evaluate the strengths and weaknesses of this approach. Moreover, developing the first generation of assessments using outcomes from the learning sciences is hampered by the limited information available on the types of cognitive models that can facilitate the design, validation, and administration of educational tests. Yet the role of cognitive models is paramount to the success of this endeavor. The National Research Council (NRC, 2001) asserted in their seminal publication, Knowing What Students Know: The Science and Design of Educational Assessments, that cognitive models have a foundational role in educational assessment that is currently poorly developed and under-valued:

Science and scientific progress are products of skilled methods of thinking about and acting on the world. The innovation, systematicness, and logic comprising scientific reasoning and discovery are therefore natural goals for governments interested in cultivating a literate, technologically savvy, and economically productive citizenry in the twenty-first century (Hanushek, 2005). As noted in Chapter 1, recent international assessments indicate variability in how well students of different nations have mastered scientific knowledge and skills. For example, U.S. examinees scored below average on two out of three content area sub-scales in the 2006 Programme for International Student Assessment (PISA), namely, explaining phenomena scientifically and using scientific evidence (see Baldi et al., 2007; see also Organization Economic Cooperation Development [OECD], 2006). This finding is not new. In 1982–1983, the National Commission on Excellence in Education, composed of a select group of university presidents, professors, and K–12 teachers, compiled a grim report on the state of science and mathematics' education in the United States: A Nation at Risk. Moreover, the National Research Council's (2007) Taking Science to School indicates that at the time A Nation at Risk (1983) was disseminated, U.S. President Ronald Reagan had already cautioned the National Academies of Sciences and Engineering of an impending decline in science and mathematics achievement, and of the erosion this decline could eventually cause the American people in their standards of living.

It is an understatement to say that mathematical knowledge and skill are valuable for the jobs and careers of the twenty-first century. In fact, they are essential for individuals who want to have the widest array of career options available and a high quality of life. Learners who shun mathematics shut themselves off from many lucrative career paths. According to a recently published article in the Wall Street Journal (Needleman, January 26, 2009), “Doing the Math to Find Good Jobs,” the best occupations in America all required advanced mathematics. According to data compiled by the U.S. Bureau of Labor Statistics and Census, the top five jobs in a list of two hundred included mathematician, actuary, statistician, biologist, and software engineer. These jobs were rated highest because they combined large salaries with desirable working conditions, namely, indoor office environments, unadulterated air, absence of heavy lifting and physical hardship, and conveniences such as controlling one's work schedule. The worst jobs were lumberjack, dairy farmer, taxi driver, seaman, and emergency medical technician. Most of the jobs at the lower end of the list did not require advanced mathematics.

The importance of mathematics for maximizing the likelihood of obtaining a desirable job in the future would make one think that students, desirous of having an edge for a future career, would be clamoring to learn and perform as well as possible in mathematics. Yet this is not the case.

Reading comprehension is a central skill students must learn to become successful learners in other content domains. For example, adequate reading skills are required to acquire scientific and mathematical literacy skills. In the present volume, then, we begin by describing cognitive models of task performance in reading comprehension, because reading skills function as a gatekeeper for the acquisition of most other academic skills.

Although students typically prefer reading over mathematics (OECD, 2004), many students continue to struggle with their reading performance. According to 2003 PISA results (Lemke et al., 2004), students in the United States scored 495 on the reading component, which is not measurably different from the OECD average of 494 (PISA results are scaled to have a mean of approximately 500 with a standard deviation of 100). This is good news for American students. However, it should be noted that among the thirty-eight countries participating in the assessment, eleven countries scored higher in reading than the United States.

More recently, the U.S. National Center for Education Statistics (2009) reported that although students' reading scores on the National Assessment of Educational Progress (NAEP) have increased since 1992, they remain unchanged from 2007 to 2009. In particular, 67 percent of students in grade four read at a basic level, and only 33 percent were classified as either proficient or advanced.

In this penultimate chapter, we summarize our impetus for identifying and evaluating diagrammatic cognitive models in reading, science, and mathematics and offer some conclusions about where we go from here. Borrowing the definition from Leighton and Gierl (2007a), a cognitive model was defined as a “simplified description of human problem solving on standardized educational tasks, which helps to characterize the knowledge and skills students at different levels of learning have acquired and to facilitate the explanation and prediction of students' performance” (p. 6). In Chapter 1, we indicated that large-scale educational tests, redesigned and redeveloped from cognitive models in the learning sciences, may offer enhanced information (test-based inferences) about student problem solving and thinking. This enhanced information may help remediate the relatively low test performance of many students, including U.S. students, who are struggling to learn and demonstrate knowledge in core domains. In Chapter 1, we also presented accepted knowledge and principles from the learning sciences about the nature of thinking, learning, and performance to set the stage for what may be required for redesigning and redeveloping large-scale assessments. Illustrative empirical studies in the field of educational measurement were described to demonstrate attempts at redesigning and redeveloping educational assessments based on the learning sciences. Chapters 2, 3, 4, and 5 presented our criteria for evaluating cognitive models and also offered examples of diagrammatic cognitive models in reading, science, and mathematics that have garnered substantial empirical support.

Research focused on the application of cognitive principles to assessment practices is thriving in educational measurement, particularly in the area of cognitively based statistical methods. Since the publication of key articles, chapters, and books two decades ago (e.g. Frederiksen, Glaser, Lesgold, & Shafto, 1990; Nichols, 1994; Nichols, Chipman, & Brennan, 1995; Ronning, Glover, Conoley, & Witt, 1990; Snow & Lohman, 1989), there has been clear recognition that inferences about examinees' knowledge and skills require detailed information on the organization, representation, and production of attributes from a cognitive model. The form that these models may take, their learning scientific foundations, and their applicability to the design and development of large-scale educational assessment are topics in this book. The desire to integrate cognition with assessment has also spawned a range of ambitious research activities designed to identify and evaluate examinees' knowledge and skills using new statistical methods. Many recent examples can be cited to document these focused research efforts. For example, an American Educational Research Association (AERA) special interest group, Cognition and Assessment, was formed in 2007 to provide a platform for scholars presenting cutting-edge research that combines the fields of cognitive psychology, cognitive science, educational psychology, educational assessment, and statistics to solve complex assessment problems using a multi-disciplinary approach. A special issue of the Journal of Educational Measurement was published in 2007 devoted to cognitively based statistical methods.

Victor Hugo is credited with stating that “There is nothing more powerful than an idea whose time has come.” In educational achievement testing, a multi-billion-dollar activity with profound implications for individuals, governments, and countries, the idea whose time has come, it seems, is that large-scale achievement tests must be designed according to the science of human learning. Why this idea, and why now? To begin to set a context for this idea and this question, a litany of research studies and public policy reports can be cited to make the simple point that students in the United States and abroad are performing relatively poorly in relation to expected standards and projected economic growth requirements (e.g., American Association for the Advancement of Science, 1993; Chen, Gorin, Thompson, & Tatsuoka, 2008; Grigg, Lauko, & Brockway, 2006; Hanushek, 2003, 2009; Kilpatrick, Swafford, & Findell, 2001; Kirsch, Braun, & Yamamoto, 2007; Manski & Wise, 1983; Murnane, Willet, Dulhaldeborde, & Tyler, 2000; National Commission on Excellence in Education, 1983; National Mathematics Advisory Panel, 2008; National Research Council, 2005, 2007, 2009; Newcombe et al., 2009; Phillips, 2007; Provasnik, Gonzales, & Miller, 2009). According to a 2007 article in the New York Times, Gary Phillips, chief scientist at the American Institutes for Research, was quoted as saying, “our Asian economic competitors are winning the race to prepare students in math and science.”

Cognitive diagnostic assessment (CDA) is designed to measure specific knowledge structures and processing skills in students so as to provide information about their cognitive strengths and weaknesses. CDA is still in its infancy, but its parentage is fairly well established. In 1989, two seminal chapters in Robert Linn's Educational Measurement signaled both the escalating interest in and the need for cognitive diagnostic assessment. Samuel Messick's chapter, “Validity”, and the late Richard Snow and David Lohman's chapter, “Implications of Cognitive Psychology for Educational Measurement”, helped solidify the courtship of cognitive psychology within educational measurement. The ideas expressed in these chapters attracted many young scholars to educational measurement and persuaded other, well-established scholars to consider the potential of a relatively innovative branch of psychology, namely, cognitive psychology, for informing test development.

CDA can be traced to the ideas expressed in the previously mentioned chapters and, of course, to the many other authors whose ideas, in turn, inspired Messick, Snow, and Lohman (e.g., Cronbach, 1957; Cronbach & Meehl, 1955; Embretson, 1983; Loevinger, 1957; Pellegrino & Glaser, 1979). Since 1989, other influential articles, chapters, and books have been written specifically about CDA (see Frederiksen, Glaser, Lesgold, & Shafto, 1990). Most notably, the article by Paul Nichols (1994) titled “A Framework for Developing Cognitively Diagnostic Assessments” and the book coedited by Paul Nichols, Susan Chipman, and Robert Brennan (1995) appropriately titled Cognitively Diagnostic Assessment.

INTRODUCTION

Many educational assessments are based on cognitive problem-solving tasks. Cognitive diagnostic assessments are designed to model examinees' cognitive performances on these tasks and yield specific information about their problem-solving strengths and weaknesses. Although most psychometric models are based on latent trait theories, a cognitive diagnostic assessment requires a cognitive information processing approach to model the psychology of test performance because the score inference is specifically targeted to examinees' cognitive skills. Latent trait theories posit that a small number of stable underlying characteristics or traits can be used to explain test performance. Individual differences on these traits account for variation in performance over a range of testing situations (Messick, 1989). Trait performance is often used to classify or rank examinees because these traits are specified at a large grain size and are deemed to be stable over time. Cognitive information processing theories require a much deeper understanding of trait performance, where the psychological features of how a trait can produce a performance become the focus of inquiry (cf. Anderson et al., 2004). With a cognitive approach, problem solving is assumed to require the processing of information using relevant sequences of operations. Examinees are expected to differ in the knowledge they possess and the processes they apply, thereby producing response variability in each test-taking situation. Because these knowledge structures and processing skills are specified at a small grain size and are expected to vary among examinees within any testing situation, cognitive theories and models can be used to understand and evaluate specific cognitive skills that affect test performance.

The term cognitive diagnostic assessment (CDA) is used in this chapter to refer to a specific type of student evaluation. Unlike classroom-based tests designed by teachers or large-scale assessments designed by test developers to measure how much an examinee knows about a subject domain, CDAs are designed to measure the specific knowledge structures (e.g., distributive rule in mathematics) and processing skills (e.g., applying the distributive rule in appropriate mathematical contexts) an examinee has acquired. The type of information provided by results from a CDA should answer questions such as the following: Does the examinee know the content material well? Does the examinee have any misconceptions? Does the examinee show strengths for some knowledge and skills but not others? The objective of CDAs, then, is to inform stakeholders of examinees' learning by pinpointing the location where the examinee might have specific problem-solving weaknesses that could lead to difficulties in learning. To serve this objective, CDAs are normally informed by empirical investigations of how examinees understand, conceptualize, reason, and solve problems in content domains (Frederiksen, Glaser, Lesgold, & Shafto, 1990; Nichols, 1994; Nichols, Chipman, & Brennan, 1995).

In this chapter, we focus on two methods for making sense of empirical investigations of how examinees understand, conceptualize, reason, and solve problems in content domains. As a way of introduction, we first briefly discuss the importance of CDAs for providing information about examinees' strengths and weaknesses, including the ways in which CDAs differ from traditional classroom-based tests and large-scale tests.

In the Introduction to this volume, we began by describing key concepts that underlie cognitive diagnostic assessment (CDA) and by specifying some of the early ideas and precedents that guided the merger between cognitive psychology and educational measurement. Then, three distinct sections were presented where a host of esteemed contributors described research on topics related to CDA, theory, and practice. Chapters describing the foundations of CDA, principles of test design and analysis, and psychometric procedures and applications were presented. After surveying these chapters, we acknowledge that not all issues relevant to CDA were adequately covered. Some omissions occurred not because these topics are considered unimportant, but because, in some cases, the topics are not ready for discussion and, in other cases, the most appropriate authors were unavailable. Thus, in the final section, we highlight some of the important topics that were not covered in this book and, in the process, identify areas in which future research is required.

ISSUE 1: ROLE OF COGNITIVE MODELS IN COGNITIVE DIAGNOSTIC ASSESSMENT

Every author in this volume claims that some type of cognitive model is required to make inferences about examinees' problem-solving skills. These models provide the framework necessary for guiding item development and directing psychometric analyses so test performance can be linked to specific inferences about examinees' cognitive skills. The foundation for generating diagnostic inferences, in fact, rest with cognitive theories and models. Hence, the veracity of the cognitive models and the validity of the diagnostic inferences must be evaluated.