Chapter Overview and Goals

In 2001, a report was issued by the National Research Council (NRC) entitled “Knowing What Students Know: The Science and Design of Educational Assessment” (Pellegrino, Chudowsky, & Glaser, 2001). The goal was to evaluate the state of research and theory on educational assessment and establish the scientific foundations for their design and use. As argued in that volume, many of the debates that surround educational assessment emanate from a failure to understand its fundamental nature, including the ways in which theories and models (1) of learning and knowing and (2) of measurement and statistical inference interact with and influence processes of assessment design, use, and interpretation. In this chapter we review some of the key issues regarding educational assessment raised in that report as well as examples from science, technology, engineering, and mathematics (STEM) education and educational research contexts. Our goal in explicating current understanding of the science and design of educational assessment and its applications to STEM education is to provide background knowledge that supports the effective design and use of assessment in engineering education and sharpens the focus of engineering education R&D.

In the first section we briefly introduce some key ideas critical to understanding educational assessment. This includes consideration of formal and informal uses of assessment and some conceptual issues associated with assessment design, interpretation, and use. In the second section we discuss three related conceptual frameworks about assessment that should be considered by anyone using assessment for instructional or research purposes. These include (1) assessment as a process of reasoning from evidence, (2) the use of an evidence-centered design process to develop and interpret assessments, and (3) centrality of the concept of validity in the design, use, and interpretation of any assessment. In the third section we turn to a discussion of concepts of measurement as applied to assessment and the role of statistical inference. This includes the assumptions underlying different types of psychometric models used to estimate student proficiency. The fourth section then presents applications of key ideas from the preceding sections in the form of illustrative examples of assessments used in engineering education research. In the final section we close by briefly considering the significance of a careful and thoughtful approach to assessment design, use, and interpretation in engineering education research.

INTRODUCTION

There is a long history of calls for combining cognitive science and psychometrics (Cronbach, 1975; Snow & Lohman, 1989). The U.S. standards movement, begun more than 20 years ago (McKnight et al., 1987; National Council of Teachers of Mathematics, 1989), sought to articulate public standards for learning that would define and promote successful performance by all students; establish a common base for curriculum development and instructional practice; and provide a foundation for measuring progress for students, teachers and programs. The standards movement provided the first widespread call for assessment systems that directly support learning. For success, such systems must satisfy a number of conditions having to do with cognitive-science–based design, psychometrics, and implementation. This chapter focuses on the psychometric aspects of one particular system that builds on a carefully designed test and a user-selected set of relevant skills measured by that test to assess student mastery of each of the chosen skills. This type of test-based skills level assessment is called skills diagnosis. The system that the chapter describes in detail is called the Fusion Model system.

This chapter focuses on the statistical and psychometric aspects of the Fusion Model system, with skills diagnosis researchers and practitioners in mind who may be interested in working with this system. We view the statistical and psychometric aspects as situated within a comprehensive framework for diagnostic assessment test design and implementation.