Skip to main content Accessibility help

We use cookies to distinguish you from other users and to provide you with a better experience on our websites. Close this message to accept cookies or find out how to manage your cookie settings.

Close cookie message
×
Hostname: page-component-586b7cd67f-rdxmf Total loading time: 0 Render date: 2024-12-08T18:07:34.515Z Has data issue: false hasContentIssue false

Chapter 29 - The Science and Design of Assessment in Engineering Education

Published online by Cambridge University Press:  05 February 2015

By
James W. Pellegrino ,
Louis V. DiBello  and
Sean P. Brophy
Edited by
Aditya Johri  and
Barbara M. Olds
James W. Pellegrino
Affiliation:
University of Illinois
Louis V. DiBello
Affiliation:
University of Illinois
Sean P. Brophy
Affiliation:
Purdue University
Aditya Johri
Affiliation:
Virginia Polytechnic Institute and State University
Barbara M. Olds
Affiliation:
Colorado School of Mines
Get access

Summary

Chapter Overview and Goals

In 2001, a report was issued by the National Research Council (NRC) entitled “Knowing What Students Know: The Science and Design of Educational Assessment” (Pellegrino, Chudowsky, & Glaser, 2001). The goal was to evaluate the state of research and theory on educational assessment and establish the scientific foundations for their design and use. As argued in that volume, many of the debates that surround educational assessment emanate from a failure to understand its fundamental nature, including the ways in which theories and models (1) of learning and knowing and (2) of measurement and statistical inference interact with and influence processes of assessment design, use, and interpretation. In this chapter we review some of the key issues regarding educational assessment raised in that report as well as examples from science, technology, engineering, and mathematics (STEM) education and educational research contexts. Our goal in explicating current understanding of the science and design of educational assessment and its applications to STEM education is to provide background knowledge that supports the effective design and use of assessment in engineering education and sharpens the focus of engineering education R&D.

In the first section we briefly introduce some key ideas critical to understanding educational assessment. This includes consideration of formal and informal uses of assessment and some conceptual issues associated with assessment design, interpretation, and use. In the second section we discuss three related conceptual frameworks about assessment that should be considered by anyone using assessment for instructional or research purposes. These include (1) assessment as a process of reasoning from evidence, (2) the use of an evidence-centered design process to develop and interpret assessments, and (3) centrality of the concept of validity in the design, use, and interpretation of any assessment. In the third section we turn to a discussion of concepts of measurement as applied to assessment and the role of statistical inference. This includes the assumptions underlying different types of psychometric models used to estimate student proficiency. The fourth section then presents applications of key ideas from the preceding sections in the form of illustrative examples of assessments used in engineering education research. In the final section we close by briefly considering the significance of a careful and thoughtful approach to assessment design, use, and interpretation in engineering education research.

Type
Chapter
Information
Cambridge Handbook of Engineering Education Research , pp. 571 - 598
Publisher: Cambridge University Press
Print publication year: 2014

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

American Educational Research Association, American Psychological Association, & National Council on Measurement in Education (AERA/APA/NCME). (1999). Standards for educational and psychological testing. Washington, DC: Author.Google Scholar
Atman, C. J., Kilgore, D., & McKenna, A. (2008). Characterizing design learning: A mixed-methods study of engineering designers’ use of language. Journal of Engineering Education, 97(3), 309–326.CrossRefGoogle Scholar
Baker, F. (2001). The basics of item response theory. ERIC Clearinghouse on Assessment and Evaluation, University of Maryland, College Park, MD. Retrieved from
Barr, R. W., Cone, J., Roselli, R. J., & Brophy, S. P. (2003). Initial experiences using an interactive classroom participation system (CPS) for presenting the iron cross biomechanics module. In Proceedings of the 2003 ASEE Gulf-Southwest Annual Conference, the University of Texas at Arlington. Lubbock, TX.Google Scholar
Bayes, T. P. (1763). An essay towards solving a problem in the doctrine of chances. Philosophical Transactions of the Royal Society of London, 53, 370–418.CrossRefGoogle Scholar
Beatty, I. D., & Gerace, W. J. (2009). Technology-enhanced formative assessment: A research-based pedagogy for teaching science with classroom response technology. Journal of Science Education and Technology, 18(2), 1146–1162.CrossRefGoogle Scholar
Bransford, J. D., & Stein, B. S. (1993). The IDEAL problem solver (2nd ed.). New York, NY: W. H. Freeman.Google Scholar
Brophy, S., Klein, S., Portsmore, M., & Rogers, C. (2008). Advancing engineering in P-12 classrooms. Journal of Engineering Education, 97(3), 369–387.CrossRefGoogle Scholar
Brophy, S., & Li, S. (2011). Problem definition in design by first year engineering students. In Proceedings of the American Society of Engineering Education Annual Conference, Vancouver, BC, Canada.Google Scholar
Chi, M. T. H., Glaser, R., & Rees, E. (1982). Expertise in problem solving. In Sternberg, R. (Ed.), Advances in the psychology of human intelligence (Vol. 1, pp. 7–76). Hillsdale, NJ: Lawrence Erlbaum.Google Scholar
Cordray, D. S., Harris, T. R., & Klein, S. (2009). A research synthesis of the effectiveness, replicability, and generality of the VaNTH challenge-based instructional models in bioengineering. Journal of Engineering Education, 98, 335–348.CrossRefGoogle Scholar
Crocker, L., & Algina, J. (2006). Introduction to classical and modern test theory. New York, NY: Wadsworth.Google Scholar
Crouch, C. H., & Mazur, E. (2001). Peer instruction: Ten years of experience and results. American Journal of Physics, 69, 970–977.CrossRefGoogle Scholar
DiBello, L. V., Roussos, L. A., & Stout, W. F. (2007). Review of cognitively diagnostic assessment and a summary of psychometric models. In Rao, C. R. & Sinharay, S. (Eds.), Handbook of statistics, Vol. 26: Psychometrics (pp. 979–1030). Amsterdam: Elsevier.Google Scholar
DiBello, L., & Stout, W. F. (2003). Student profile scoring methods for informative assessment. In Yanai, H., Okada, A., Shigemasu, K., & Meulmann, J. J. (Eds.), New developments in psychometrics (pp. 81–92). Tokyo: Springer.CrossRefGoogle Scholar
DiBello, L. V., & Stout, W. F. (Eds.). (2007) Special issue on IRT-based cognitive diagnostic models and related methods, Journal of Educational Measurement, 44(4), 285–292.CrossRef
Dym, C. L., Agogino, A. M., Eris, V., Frey, D., & Leifer, L. J. (2005). Engineering design, thinking, teaching and learning. Journal of Engineering Education, 94(1), 103–120.CrossRefGoogle Scholar
Haertel, E. H., & Lorie, W. A. (2004). Validating standards-based test score interpretations. Measurement: Interdisciplinary Research and Perspectives, 2(2), 61–103.Google Scholar
Hake, R. R. (1998). Interactive-engagement versus traditional methods: A six-thousand-student survey of mechanics test data for introductory physics courses. American Journal of Physics, 66(1), 64–74.CrossRefGoogle Scholar
Hambleton, R. K., Swaminathan, H., & Rogers, H. J. (1991). Fundamentals of item response theory. Measurement Methods for the Social Science. Newbury Park, CA: SAGE.Google Scholar
Hatano, G., & Inagaki, K. (1986). Two courses of expertise. In Stevenson, H., Azuma, H., & Hakuta, K. (Eds.), Child development and education in Japan. New York, NY: W. H. Freeman.Google Scholar
Hestenes, D., Wells, M., & Swackhamer, G. (1992, March). Force concept inventory. The Physics Teacher, 30, 159–166.CrossRefGoogle Scholar
Hickey, D., & Pellegrino, J. W. (2005). Theory, level, and function: Three dimensions for understanding transfer and student assessment. In Mestre, J. P. (Ed.), Transfer of learning from a modern multidisciplinary perspective (pp. 251–293). Greenwich, CT: Information Age.Google Scholar
Kane, M. T. (2006). Validation. In Brennan, R. L. (Ed.), Educational measurement (4th ed., pp. 17–64). Westport, CT: Praeger.Google Scholar
Lord, R. M., & Novick, M. R. (1968). Statistical theories of mental test scores. Reading, MA: Addison-Wesley.Google Scholar
Marion, S., & Pellegrino, J. W. (2006, Winter). A validity framework for evaluating the technical quality of alternate assessments. Educational Measurement: Issues and Practice, 47–57.
Martin, T., Petrosino, A. J., Rivale, S., & Diller, K. R. (2006). The development of adaptive expertise in biotransport. New Directions for Teaching and Learning, 108, 35–47.CrossRefGoogle Scholar
Messick, S. (1994). The interplay of evidence and consequences in the validation of performance assessments. Educational Researcher, 23(2), 13–23.CrossRefGoogle Scholar
Mislevy, R. J., & Haertel, G. (2006). Implications of evidence-centered design for educational assessment. Educational Measurement: Issues and Practice, 25, 6–20.CrossRefGoogle Scholar
Mislevy, R. J., & Riconscente, M. M. (2006). Evidence-centered assessment design: Layers, concepts, and terminology. In Downing, S. & Haladyna, T. (Eds.), Handbook of test development (pp. 61–90). Mahwah, NJ: Lawrence Erlbaum.Google Scholar
Mislevy, R. J., Steinberg, L., & Almond, R. (2003). On the structure of educational assessments. Measurement: Interdisciplinary Research and Perspectives, 1, 3–67.Google Scholar
National Research Council. (2003). Assessment in support of learning and instruction: Bridging the gap between large-scale and classroom assessment. Washington, DC: The National Academies Press.Google Scholar
Nersessian, N. J. (2009). How do engineering scientists think? Model-based simulation in biomedical engineering laboratories. Topics in Cognitive Science, 1, 730–757.CrossRefGoogle Scholar
Pellegrino, J. W., Chudowsky, N., & Glaser, R. (Eds.). (2001). Knowing what students know: The science and design of educational assessment. Washington, DC: The National Academies Press.
Pellegrino, J. W., & Hickey, D. (2006). Educational assessment: Towards better alignment between theory and practice. In Verschaffel, L., Dochy, F., Boekaerts, M., & Vosniadou, S. (Eds.), Instructional psychology: Past, present and future trends. Sixteen essays in honour of Erik De Corte (pp. 169–189). Oxford: Elsevier.Google Scholar
Polya, G. (1945). How to solve it. Princeton, NJ: Princeton University Press.Google Scholar
Prince, M. (2004). Does active learning work? A review of the research. Journal of Engineering Education, 93(3), 223–231.CrossRefGoogle Scholar
Prince, M. J., & Felder, R. M. (2006). Inductive teaching and learning methods: Definitions, comparisons, and research bases. Journal of Engineering Education, 95, 123–138.CrossRefGoogle Scholar
Roselli, R. J., & Brophy, S. P. (2003). Redesigning a biomechanics course using challenge-based instruction. IEEE Engineering in Medicine and Biology Magazine, 22(4), 66–70.CrossRefGoogle ScholarPubMed
Roselli, R. J., & Brophy, S. P. (2006). Experiences with formative assessment in engineering classrooms. Journal of Engineering Education, 95(4), 325–333.CrossRefGoogle Scholar
Roselli, R. J., Howard, L., & Brophy, S. P. (2006). A computer-based free body diagram assistant. Computer Applications in Engineering Education, 14(4), 281–290.CrossRefGoogle Scholar
Ruiz-Primo, M. A., Shavelson, R. J., Hamilton, L., & Klein, S. (2002). On the evaluation of systemic science education reform: Searching for instructional sensitivity. Journal of Research in Science Teaching, 39, 369–393.CrossRefGoogle Scholar
Santiago-Román, A. I. (2009). Fitting cognitive diagnostic assessment to the concept assessment tool for statics (CATS) (Doctoral dissertation). Purdue University, Lafayette, IN.Google Scholar
Santiago-Román, A. I., Streveler, R. A., & DiBello, L. (2010). The development of estimated cognitive attribute profiles for the Concept Assessment Tool for Statics. Paper presented at the 40th ASEE/IEEE Frontiers in Education Conference, Washington, DC.CrossRefGoogle Scholar
Santiago-Román, A. I., Streveler, R. A., Steif, P., & DiBello, L. V. (2010). The development of a Q-matrix for the Concept Assessment Tool for Statics. Paper presented at the ERM Division of the ASEE Annual Conference and Exposition, Louisville, KY.Google Scholar
Schum, D. (1987). Evidence and inference for the intelligence analyst. Lantham, MD: University Press of America.Google Scholar
Schwartz, D. L., & Arena, D. (2009). Choice-based assessments for the digital age. Stanford University, School of Education. Retrieved from Google Scholar
Schwartz, D. L., Bransford, J. D., & Sears, D. (2005). Efficiency and innovation in transfer. In Mestre, J. P. (Ed.), Transfer of learning from a modern multidisciplinary perspective (pp. 1–51). Greenwich, CT: Information Age.Google Scholar
Smith, K. A., Johnson, D. W., Johnson, R. W., & Sheppard, S. D. (2005). Pedagogies of engagement: Classroom-based practices. Journal of Engineering Education, 94(1), 87–101.CrossRefGoogle Scholar
Steif, P. S., & Dantzler, J. A. (2004). A statics concept inventory: Development and psychometric analysis. Journal of Engineering Education, 94(4), 363–371.CrossRefGoogle Scholar
Steif, P. S., & Hansen, M. A. (2006). Comparisons between performances in a statics concept inventory and course examinations, International Journal of Engineering Education, 22, 1070–1076.Google Scholar
Steif, P. S., & Hansen, M. A. (2007). New practices for administering and analyzing the results of concept inventories. Journal of Engineering Education, 96, 205–212.CrossRefGoogle Scholar
Stigler, S. M. (1982). Thomas Bayes’ Bayesian inference. Journal of the Royal Statistical Society, Series A, 145, 250–258.CrossRefGoogle Scholar
Toulmin, S. E. (2003). The uses of argument. Cambridge: Cambridge University Press.CrossRefGoogle Scholar
van der Linden, W. J., & Hambleton, R. K. (Eds.) (1996). Handbook of modern item response theory. New York, NY: Springer.
Walker, J. T., Cordray, D. S., King, P. H., & Brophy, S. P. (2006). Design scenarios as an assessment of adaptive expertise. International Journal of Engineering Education, 22(1), 1–7.Google Scholar
Wiggins, G. (1998). Educative assessment: Designing assessments to inform and improve student performance. San Francisco, CA: Jossey- Bass.Google Scholar
Woods, D. R. (1997). Developing problem-solving skills: The McMaster problem solving program. Journal of Engineering Education, 86(2), 75–92.CrossRefGoogle Scholar

Save book to Kindle

To save this book to your Kindle, first ensure no-reply@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about saving to your Kindle.

Note you can select to save to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be saved to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

Available formats
×

Save book to Dropbox

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Dropbox.

Available formats
×

Save book to Google Drive

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Google Drive.

Available formats
×