Hostname: page-component-7479d7b7d-68ccn Total loading time: 0 Render date: 2024-07-11T11:32:58.358Z Has data issue: false hasContentIssue false
  • English
  • Français

Moving the field of vocabulary assessment forward: The need for more rigorous test development and validation

Published online by Cambridge University Press:  25 September 2019

Norbert Schmitt ,
Paul Nation  and
Benjamin Kremmel
Norbert Schmitt*
Affiliation:
University of Nottingham, United Kingdom
Paul Nation
Affiliation:
Victoria University of Wellington, New Zealand
Benjamin Kremmel
Affiliation:
University of Innsbruck, Austria
*
*Corresponding author. Email: norbert.schmitt@nottingham.ac.uk
Get access Rights & Permissions [Opens in a new window]

Abstract

Recently, a large number of vocabulary tests have been made available to language teachers, testers, and researchers. Unfortunately, most of them have been launched with inadequate validation evidence. The field of language testing has become increasingly more rigorous in the area of test validation, but developers of vocabulary tests have generally not given validation sufficient attention in the past. This paper argues for more rigorous and systematic procedures for test development, starting from a more precise specification of the test's purpose, intended testees and educational context, the particular aspects of vocabulary knowledge which are being measured, and the way in which the test scores should be interpreted. It also calls for greater assessment literacy among vocabulary test developers, and greater support for the end users of the tests, for instance, with the provision of detailed users' manuals. Overall, the authors present what they feel are the minimum requirements for vocabulary test development and validation. They argue that the field should self-police itself more rigorously to ensure that these requirements are met or exceeded, and made explicit for those using vocabulary tests.

Type
First Person Singular
Information
Language Teaching , Volume 53 , Issue 1 , January 2020 , pp. 109 - 120
Copyright
Copyright © Cambridge University Press 2019

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Bachman, L. F., & Palmer, A. S. (2010). Language assessment in practice. Oxford: Oxford University Press.Google Scholar
Beglar, D. (2010). A Rasch-based validation of the Vocabulary Size Test. Language Testing, 27(1), 101–118.CrossRefGoogle Scholar
Chapelle, C. A. (2012). Validity argument for language assessment: The framework is simple…. Language Testing, 29(1), 19–27.CrossRefGoogle Scholar
Chapelle, C. A., Enright, M. K., & Jamieson, J. M. (2008). Building a validity argument for the Test of English as a Foreign Language. New York: Routledge.Google Scholar
Coxhead, A. & Nation, P. (forthcoming). Group vs. individual: How administration procedure can affect vocabulary test scores.Google Scholar
Coxhead, A., Nation, P., & Sim, D. (2015). Vocabulary size and native speaker secondary school students. New Zealand Journal of Educational Studies, 50(1), doi: 10.1007/s40841-015-0002-3CrossRefGoogle Scholar
Derrah, R & Rowe, D.E. (2015). Validating the Japanese bilingual version of the Vocabulary Size Test. International Journal of Languages, Literature and Linguistics, 1(2), 131–135.CrossRefGoogle Scholar
Dörnyei, Z., & Ryan, S. (2015). The psychology of the language learner revisited. New York: Routledge.CrossRefGoogle Scholar
Elgort, I. (2013). Effects of L1 definitions and cognate status of test items on the vocabulary size test. Language Testing, 30(2), 253–272.CrossRefGoogle Scholar
Fulcher, G., & Davidson, F. (2007). Language testing and assessment: An advanced resource book. New York: Routledge.Google Scholar
Goeman, J. J., & de Jong, N. H. (2018). How well does the sum score summarize the test? Summability as a measure of internal consistency. Educational Measurement: Issues and Practice, 37(2), 54–63.CrossRefGoogle Scholar
González Fernández, B., & Schmitt, N. (in press). Word knowledge: Exploring the relationships and order of acquisition of vocabulary knowledge components. Applied Linguistics.Google Scholar
Gyllstad, H., Vilkaitė, L., & Schmitt, N. (2015). Assessing vocabulary size through multiple-choice formats: Issues with guessing and sampling rates. ITL International Journal of Applied Linguistics, 166, 276–303.CrossRefGoogle Scholar
Harding, L., & Kremmel, B. (in press). SLA researcher assessment literacy. In Winke, P. & Brunfaut, T. (Eds.), The Routledge handbook of second language acquisition and language testing. New York: Routledge.Google Scholar
Kane, M. (2006). Validation. In Brennan, R. L. (Ed.), Educational measurement (4th ed., pp. 17–64). Westport, CT: American Council on Education/Praeger.Google Scholar
Kane, M. (2012). Validating score interpretations and uses. Language Testing, 29(1), 3–17.CrossRefGoogle Scholar
Kane, M. (2013). Validating the interpretations and uses of test scores. Journal of Educational Measurement, 50(1), 1–73.CrossRefGoogle Scholar
Karami, H. (2012). The development and validation of a bilingual version of the Vocabulary Size Test. RELC Journal, 43(1), 53–67.CrossRefGoogle Scholar
Kremmel, B. (2016). Word families and frequency bands in vocabulary tests: Challenging conventions. TESOL Quarterly, 50(4), 976–987.CrossRefGoogle Scholar
Kremmel, B. (forthcoming). Developing a computer adaptive test: Which algorithm works best?Google Scholar
Kremmel, B. (2017). Development and initial validation of a diagnostic computer-adaptive profiler of vocabulary knowledge (Unpublished Ph.D. thesis). University of Nottingham.Google Scholar
Kremmel, B., & Schmitt, N. (2016). Interpreting vocabulary test scores: What do various item formats tell us about learners’ ability to employ words? Language Assessment Quarterly, 13(4), 377–392.CrossRefGoogle Scholar
Laufer, B., Elder, C., Hill, K., & Congdon, P. (2004). Size and strength: Do we need both to measure vocabulary knowledge? Language Testing, 21(2), 202–226.CrossRefGoogle Scholar
Laufer, B., & Goldstein, Z. (2004). Testing vocabulary knowledge: Size, strength, and computer adaptiveness. Language Learning, 54(3), 399–436.CrossRefGoogle Scholar
Laufer, B., & Nation, P. (1999). A vocabulary-size test of controlled productive vocabulary. Language Testing, 16(1), 33–51.CrossRefGoogle Scholar
Levitzky-Aviad, T., Mizrahi, L., & Laufer, B. (2014). A new test of active vocabulary size. EUROSLA 24, book of abstracts, p. 48.Google Scholar
Lemhöfer, K., & Broersma, M. (2012). Introducing LexTALE: A quick and valid Lexical Test for Advanced Learners of English. Behavior Research Methods, 44(2), 325–343.CrossRefGoogle ScholarPubMed
McLean, S., & Kramer, B. (2015). The creation of a New Vocabulary Levels Test. Shiken, 19(2), 1–11.Google Scholar
McLean, S., Kramer, B., & Beglar, D. (2015). The creation and validation of a listening vocabulary levels test. Language Teaching Research, 19(6), 741–760.CrossRefGoogle Scholar
Meara, P. (1987). Vocabulary in a second language, Vol. 2. Specialised Bibliography 4. London: CILT.Google Scholar
Meara, P. (1990). Some notes on the Eurocentres vocabulary tests. Retrieved from http://www.lognostics.co.uk/vlibrary/meara1990b.pdfGoogle Scholar
Meara, P. (1992). EFL vocabulary tests. University College, Swansea: Centre for Applied Language Studies.Google Scholar
Meara, P., & Fitzpatrick, T. (2000). Lex 30: An improved method of assessing productive vocabulary in an L2. System, 28, 19–30.CrossRefGoogle Scholar
Meara, P., & Milton, J. (2003). X_Lex, The Swansea Levels Test. Newbury: Express.Google Scholar
Messick, S. (1989). Validity. In Linn, R. L. (Ed.), Educational measurement (pp. 13–104). New York: Macmillan.Google Scholar
Milton, J., & Hopkins, N. (2006). Comparing phonological and orthographic vocabulary size: Do vocabulary tests underestimate the knowledge of some learners? Canadian Modern Language Review, 63(1), 127–147.Google Scholar
Nation, I. S. P. (2007). Fundamental issues in modelling and assessing vocabulary knowledge. In Daller, H., Milton, J., & Treffers-Daller, J. (Eds.), Modelling and assessing vocabulary knowledge (pp. 35–43). Cambridge University Press.CrossRefGoogle Scholar
Nation, I. S. P., & Beglar, D. (2007). A vocabulary size test. The Language Teacher, 31(7), 9–13.Google Scholar
Nation, P. (1983). Testing and teaching vocabulary. Guidelines, 5, 12–25.Google Scholar
Nguyen, L. T. C., & Nation, P. (2011). A bilingual vocabulary size test of English for Vietnamese learners. RELC Journal, 42(1), 86–99.CrossRefGoogle Scholar
Porte, G., & McManus, K. (2019). Doing replication research in applied linguistics. New York: Routledge.Google Scholar
Read, J. (1998). Validating a test to measure depth of vocabulary knowledge. In Kunnan, A. (Ed.), Validation in language assessment (pp. 41–60). Mahwah, NJ: Lawrence Erlbaum.Google Scholar
Read, J. (2000). Assessing vocabulary. Cambridge: Cambridge University Press.CrossRefGoogle Scholar
Schmitt, N. (2010). Researching vocabulary: A vocabulary research manual. Basingstoke: Palgrave Macmillan.CrossRefGoogle Scholar
Schmitt, N. (2014). Size and depth of vocabulary knowledge: What the research shows. Language Learning, 64(4), 913–951.CrossRefGoogle Scholar
Schmitt, N., Schmitt, D., & Clapham, C. (2001). Developing and exploring the behaviour of two new versions of the Vocabulary Levels Test. Language Testing, 18(1), 55–88.CrossRefGoogle Scholar
Voss, E. (2012). A validity argument for score meaning of a computer-based ESL academic collocational ability test based on a corpus-driven approach to test design (Unpublished Ph.D. thesis). Iowa State University.Google Scholar
Webb, S. (2005). Receptive and productive vocabulary learning: The effects of reading and writing on word knowledge. Studies in Second Language Acquisition, 27, 33–52.CrossRefGoogle Scholar
Webb, S., Sasao, Y., & Ballance, O. (2017). The updated Vocabulary Levels Test: Developing and validating two new forms of the VLT. ITL – International Journal of Applied Linguistics, 168(1), 34–70.CrossRefGoogle Scholar