Skip to main content Accessibility help


  • Cynthia Berger (a1), Scott Crossley (a1) and Stephen Skalicky (a1)


A large dataset of word recognition behavior from nonnative speakers (NNS) of English was collected using an online crowdsourced lexical decision task. Lexical features were used to predict NNS lexical decision latencies and accuracies. Predictors of NNS latencies and accuracy included contextual diversity, age of acquisition, and contextual distinctiveness, while length moderated the impact of contextual diversity and neighborhood size on accuracy. Results have implications for second language word recognition and demonstrate that NNS behavioral data collected through large crowdsourcing projects can afford a rich source for SLA research.


Corresponding author

*Correspondence concerning this article should be addressed to Cynthia Berger. E-mail:


Hide All
Adelman, J. S., Brown, G. D. A., & Quesada, J. F. (2006). Contextual diversity, not word frequency, determined word-naming and lexical decision times. Psychological Science, 17, 814823. doi: 10.1111/j.1467-9280.2006.01787.x 16984300.
Andrews, S. (1992). Frequency and neighborhood effects on lexical access: Lexical similarity or orthographic redundancy? Journal of Experimental Psychology: Learning, Memory, and Cognition, 18, 234254.
Andrews, S. (1997). The effect of orthographic similarity on lexical retrieval: Resolving neighborhood conflicts. Psychonomic Bulletin and Review, 41, 439461.
Balota, D. A., & Chumbley, J. I. (1990). What are the effects of frequency in visual word recognition tasks? Right where we said they were! Journal of Experimental Psychology: General, 133, 231237. doi: 10.1037/0096-3445.119.2.231.
Balota, D. A., Cortese, M. J., Sergent-Marshall, S. D., Spieler, D. H., & Yap, M. (2004). Visual word recognition of single-syllable words. Journal of Experimental Psychology: General, 133, 283316. doi: 10.1037/0096-3445.133.2.283.
Balota, D. A., Ferraro, F. R., & Connor, L. T. (1991). On the early influence of meaning in word recognition: A review of the literature. In Schwanenflugel, P. (Ed.), The psychology of word meanings (pp. 187222). Hillsdale, NJ: Erlbaum.
Balota, D. A., Yap, M. J., Hutchison, K. A., Cortese, M. J., Kessler, B., Loftis, B., Treiman, R. (2007). The English Lexicon Project. Behavior Research Methods, 39, 445459. doi: 10.3758/bf03193014.
Barnhoorn, J. S., Haasnoot, E., Bocanegra, B. R., & van Steenbergen, H. (2015). QRTEngine: An easy solution for running online reaction time experiments using Qualtrics. Behavior Research Methods, 47(4), 918929.
Berger, C. M., Crossley, S. A., & Kyle, K. (2017). Using novel word context measures to predict human ratings of lexical proficiency. Educational Technology and Society, 20, 201212.
Bijeljac-Babic, R., Biardeau, A., & Grainger, J. (1997). Masked orthographic priming in bilingual word recognition. Memory and Cognition, 25, 447457.
British National Corpus. (2007). Version 3 (BNC XML Edition). Retrieved from
Brown, G. D. A., & Watson, F. L. (1987). First in, first out: Word learning age and spoken word-frequency as predictors of word familiarity and word naming latency. Memory and Cognition, 15, 208216.
Brysbaert, M., Lange, M., & Wijnendaele, I. V. (2000). The effects of age-of-acquisition and frequency-of-occurrence in visual word recognition: Further evidence from the Dutch language. European Journal of Cognitive Psychology, 12, 6585.
Brysbaert, M., & New, B. (2009a). Moving beyond Kučera and Francis: A critical evaluation of current word frequency norms and the introduction of a new and improved word frequency measure for American English. Behavior Research Methods, 41, 977990. doi: 10.3758/brm.41.4.977.
Brysbaert, M., & New, B. (2009b). Subtlexus: American word frequencies. Retrieved from http:/
Brysbaert, M., Warriner, A. B., & Kuperman, V. (2014). Concreteness ratings for 40 thousand generally known English word lemmas. Behavior Research Methods, 46, 904911.
Buhrmester, M., Kwang, T., & Gosling, S. D. (2011). Amazon's Mechanical Turk: A new source of inexpensive, yet high-quality, data?. Perspectives on Psychological Science, 6(1), 35.
Bultena, S., & Dijkstra, T. (2013). Lexical access in bilingual visual word recognition. In Chappele, C. A. (Ed.), The encyclopedia of applied linguistics (pp. 17). Hoboken, NJ: Blackwell Publishing.
Cann, R. (2000). Functional versus lexical: A cognitive dichotomy. In Borsely, R. D. (Ed.), The nature and function of syntactic categories (Vol. 32). London, UK: Academic Press.
Carlton, J. T. (1975). The role of semantic information in lexical decisions. Journal of Experimental Psychology: Human Perception and Performance, 1, 130136.
Coltheart, M., Davelaar, E., Jonasson, J., & Besner, D. (1977). Access to the internal lexicon. In Dornic, S. (Ed.), Attention and performance (Vol. VI). Hillsdale, NJ: Lawrence Erlbaum Associates.
Coltheart, M., Rastle, K., Perry, C., Langdon, R., & Ziegler, J. (2001). DRC: A dual route cascaded model of visual word recognition and reading aloud. Psychological Review, 108, 204256.
Cook, S. V., & Gor, K. (2015). Lexical access in NNS: Representational deficit or processing constraint? Mental Lexicon, 10, 247270.
Cook, S. V., Pandža, N. B., Lancaster, A. K., & Gor, K. (2016). Fuzzy nonnative phonolexical representations lead to fuzzy form-to-meaning mappings. Frontiers in Psychology, 7, 117.
Cortese, M. J., & Balota, D. A. (2012). Visual word recognition in skilled adult readers. In Spivey, M. J., McRae, K., & Joanisse, M. F. (Eds.), The Cambridge handbook of psycholinguistics (pp. 159185). New York, NY: Cambridge University Press.
Crossley, S. A. (2013). Assessing automatic processing of hypernymic relations in first language speakers and advanced second language learners: A semantic priming approach. The Mental Lexicon, 8, 96116. doi: 10.1075/ml.8.1.05cro.
Crossley, S. A., & McNamara, D. S. (2012). Predicting second language writing proficiency: The roles of cohesion and linguistic sophistication. Journal of Research in Reading, 35, 115135.
Crossley, S. A., Subtirelu, N., & Salsbury, T. (2013). Frequency effects or context effects in second language word learning. Studies in Second Language Acquisition, 35, 727755.
Davies, M. (2009). The 385+ million word Corpus of Contemporary American English (1990–2008+): Design, architecture, and linguistic insights. International Journal of Corpus Linguistics, 14, 159190.
De Groot, A. M., Borgwaldt, S., Bos, M., & van den Eijnden, E. (2002). Lexical decision and word naming in bilinguals: Language effects and task effects. Journal of Memory and Language, 47, 91124.
De Groot, A., Dannenburg, L., & Van Hell, J. G. (1994). Forward and backward word translation by bilinguals. Journal of Memory and Language, 33, 600.
DeKeyser, R. M. (2013). Age effects in second language learning: Stepping stones toward better understanding. Language Learning, 63, 5267.
Diependaele, K., Lemhöfer, K., & Brysbaert, M. (2013). The word frequency effect in first-and second-language word recognition: A lexical entrenchment account. The Quarterly Journal of Experimental Psychology, 66, 843863.
Dijkstra, T. (2005). Bilingual visual word recognition and lexical access. In Kroll, J. F. & De Groots, A. M. (Eds.), Handbook of bilingualism: Psycholinguistic approaches (pp. 179201). London UK: Oxford University Press.
Dijkstra, T., Miwa, K., Brummelhuis, B., Sappelli, M., & Baayen, H. (2010). How cross-language similarity and task demands affect cognate recognition. Journal of Memory and Language, 62, 284301.
Dijkstra, T., & Van Heuven, W. J. (2002). The architecture of the bilingual word recognition system: From identification to decision. Bilingualism: Language and Cognition, 5, 175197.
Duyck, W., Vanderelst, D., Desmet, T., & Hartsuiker, R. J. (2008). The frequency effect in second-language visual word recognition. Psychonomic Bulletin and Review, 15, 850855.
Ellis, A. W., & Morrison, C. M. (1998). Real age-of-acquisition effects in lexical retrieval. Journal of Experimental Psychology: Learning, Memory, and Cognition, 24, 515.
Ellis, N. C. (1997). Vocabulary acquisition: Word structure, collocation, word-class, and meaning. In Schmitt, N. & McCarthy, M. (Eds.), Vocabulary: Description, acquisition and pedagogy (pp. 122139). Cambridge, UK: Cambridge University Press.
Ellis, N. C. (2002a). Frequency effects in language processing. Studies in Second Language Acquisition, 24, 143188. doi: 10.1017/s0272263102002024.
Ellis, N. C. (2002b). Reflections on frequency effects in language processing. Second Language Acquisition, 24, 297339. doi:10.1017/s0272263102002140.
Elston-Güttler, K. E., Paulmann, S., & Kotz, S. A. (2005). Who’s in control? Proficiency and L1 influence on NNS processing. Journal of Cognitive Neuroscience, 17, 15931610.
Enochson, K., & Culbertson, J. (2015). Collecting psycholinguistic response time data using Amazon Mechanical Turk. PloS ONE, 10(3).
Fellbaum, C. (1998). WordNet: An electronic lexical database: Cambridge, MA: MIT Press.
Fender, M. (2003). English word recognition and word integration skills of native Arabic- and Japanese-speaking learners of English as a second language. Applied Psycholinguistics, 24, 289315.
Flege, J. E., Yeni-Komshian, G. H., & Liu, S. (1999). Age constraints on second-language acquisition. Journal of Memory and Language, 41, 78104.
Forster, K. I., & Chambers, S. M. (1973). Lexical access and naming time. Journal of Verbal Learning and Verbal Behavior, 12, 627635.
Grainger, J., & Jacobs, A. M. (1996). Orthographic processing in visual word recognition: A multiple read-out model. Psychological Review, 103, 518565.
Hoffman, P., Ralph, M. A. L., & Rogers, T. T. (2013). Semantic diversity: A measure of semantic ambiguity based on variability in the contextual usage of words. Behavior Research Methods, 45, 718730.
Johnson, J., & Newport, E. (1989). Critical period effects in second language learning: The influence of maturational state on the acquisition of English as a second language. Cognitive Psychology, 21, 6099.
Johnson, N. F., & Pugh, K. R. (1994). A cohort model of visual word recognition. Cognitive Psychology, 26, 240346.
Keuleers, E., Diependaele, K., & Brysbaert, M. (2010). Practice effects in large-scale visual word recognition studies: A lexical decision study on 14,000 Dutch mono-and disyllabic words and nonwords. Frontiers in Psychology, 1, 174.
Keuleers, E., Lacey, P., Rastle, K., & Brysbaert, M. (2012). The British Lexicon Project: Lexical decision data for 28,730 monosyllabic and disyllabic English words. Behavior Research Methods, 44, 287304.
Kiss, G. R., Armstrong, C., Milroy, R., & Piper, J. (1973). An associative thesaurus of English and its computer analysis. In Aitken, A. J., Bailey, R. W., & Hamilton-Smith, N. (Eds.), The computer and literary studies (pp. 153165). Edinburgh, UK: Edinburgh University Press.
Koda, K. (2005). Insights into second language reading: A cross-linguistic approach. Cambridge, UK: Cambridge University Press.
Kučera, H., & Francis, N. (1967). Computational analysis of present-day American English. Providence, RI: Brown University Press.
Kuperman, V., Stadthagen-Gonzalez, H., & Brysbaert, M. (2012). Age-of-acquisition ratings for 30,000 English words. Behavior Research Methods, 44(4), 978990.
Kyle, K., & Crossley, S. A. (2015). Automatically assessing lexical sophistication: Indices, tools, findings, and application. TESOL Quarterly, 49, 757786. doi: 10.1002/tesq.194.
Kyle, K., Crossley, S., & Berger, C. M. (2017). The tool for the automatic analysis of lexical sophistication (TAALES): Version 2.0. Behavior Research Methods . Advance Online Publication.
Lakoff, G. (1987). What categories reveal about the mind. Chicago, IL: University of Chicago.
Lemhöfer, K., Dijkstra, T., Schriefers, H., Baayen, R. H., Grainger, J., & Zwitserlood, P. (2008). Native language influences on word recognition in a second language: A megastudy. Journal of Experimental Psychology: Learning, Memory, and Cognition, 34, 12.
Litman, L., Robinson, J., & Abberbock, T. (2017). A versatile crowdsourcing data acquisition platform for the behavioral sciences. Behavior Research Methods, 49(2), 433442.
Marian, V., & Blumenfeld, H. K. (2006). Phonological neighborhood density guides: Lexical access in native and non-native language production. Journal of Social and Ecological Boundaries, 2, 335.
McClelland, J. L., & Rumelhart, D. E. (1981). An interactive activation model of context effects in letter perception: An account of basic findings. Psychological Review, 88, 375407.
McDonald, S. A., & Shillcock, R. C. (2001). Rethinking the word frequency effect: The neglected role of distributional information in lexical processing. Language and Speech, 44, 295323.
Meyer, D. E., & Schvaneveldt, R. W. (1971). Facilitation in recognizing pairs of words: Evidence of a dependence between retrieval operations. Journal of Experimental Psychology, 90(2), 227234.
Miller, G. A., Beckwith, R., Fellbaum, C., Gross, D., & Miller, K. J. (1990). Introduction to WordNet: An on-line lexical database. International Journal of Lexicography, 3, 235244.
Morrison, C. M., & Ellis, A. W. (1995). Roles of word frequency and age of acquisition in word naming and lexical decision. Journal of Experimental Psychology: Learning, Memory, and Cognition, 21, 116153.
Nelson, D. L., McEvoy, C., & Schreiber, T. (1998). The University of South Florida word association, rhyme, and word fragment norms. Retrieved from
New, B., Ferrand, L., Pallier, C., & Brysbaert, M. (2006). Reexamining the word length effect in visual word recognition: New evidence from the English Lexicon Project. Psychonomic Bulletin and Review, 13, 4552.
Ortega, L. (2016). Multicompetence in second language acquisition: Inroads into the mainstream? In Cook, V. & Wei, L. (Eds.), The Cambridge handbook of linguistic multicompetence (pp. 5076). Cambridge, UK: Cambridge University Press.
Paivio, A. (1991). Images in mind: The evolution of a theory. Hertfordshire, UK: Harvester Wheatsheaf.
Perfetti, C. A. (2007). Reading ability: Lexical quality to comprehension. Scientific Studies of Reading, 11, 357383.
Pexman, P. M., Lupker, S. J., & Hino, Y. (2002). The impact of feedback semantics in visual word recognition: Number-of-features effects in lexical decision and naming tasks. Psychonomic Bulletin and Review, 9, 542549.
Portin, M., Lehtonen, M., & Laine, M. (2007). Processing of inflected nouns in late bilinguals. Applied Psycholinguistics, 28, 135156.
Rastle, K. (2009). Visual word recognition. In Gaskell, G. (Ed.), Oxford handbook of psycholinguistics (pp. 7187). Oxford. UK: Oxford University Press.
Saito, K. (2015). The role of age of acquisition in late second language oral proficiency attainment. Studies in Second Language Acquisition, 37, 713743.
Salsbury, T., Crossley, S. A., & McNamara, D. S. (2011). Psycholinguistic word information in second language oral discourse. Second Language Research, 27(3), 343360.
Schmitt, N., & Hemchua, S. (2006). An analysis of lexical errors in the English composition of Thai learners. Prospect, 21(3), 325.
Schwanenflugel, P., & Stowe, R. W. (1991). Why are abstract concepts hard to understand? In Schwanenflugel, P. (Ed.), The psychology of word meanings. Hillsdale, NJ: Erlbaum.
Schwanenflugel, P. J., & Shoben, E. J. (1983). Differential context effects in the comprehension of abstract and concrete verbal materials. Journal of Experimental Psychology: Learning, Memory, and Cognition, 9, 82102. doi: 10.1037/0278-7393.9.1.82.
Segalowitz, N., Poulsen, C., & Komoda, M. (1991). Lower level components of reading skill in higher level bilinguals: Implications for reading instruction. AILA review, 8, 1530.
Segalowitz, N. S., & Segalowitz, S. J. (1993). Skilled performance, practice, and the differentiation of speed-up from automatization effects: Evidence from second language word recognition. Applied Psycholinguistics, 14, 369385. doi: 10.1017/S0142716400010845.
Segalowitz, S. J., Segalowitz, N. S., & Wood, A. G. (1998). Assessing the development of automaticity in second language word recognition. Applied Psycholinguistics, 19, 5367. doi: 10.1017/s0142716400010572.
Spieler, D. H., & Balota, D. A. (1997). Bringing computational models of word naming down to the item level. Psychological Science, 8, 411416.
Stanovich, K. (1991). Word recognition: Changing perspectives. In Barr, R., Kamil, M., Mosenthal, P., & Pearson, P. (Eds.), Handbook of reading research (Vol. 2, pp. 418452). Mahwah, NJ: Lawrence Erlbaum.
Steyvers, M., & Tenenbaum, J. B. (2005). The large-scale structures of semantic networks: Statistical analyses and a model of semantic growth. Cognitive Science, 29, 4178.
Sunderman, G., & Campbell, A. (2013). Psycholinguistic approaches to vocabulary. In Chapelle, C. A. (Ed.), The encyclopedia of applied linguistics. Hoboken, NJ: Blackwell Publishing.
Van Heuven, W. J., Dijkstra, T., & Grainger, J. (1998). Orthographic neighborhood effects in bilingual word recognition. Journal of Memory and Language, 39, 458483.
Wang, M., & Koda, K. (2005). Commonalities and differences in word identification skills among learners of English as a second language. Language Learning, 55, 7198.
Weekes, B. S. (1997). Differential effects of number of letters on word and nonword naming latency. Quarterly Journal of Experimental Psychology: Human Experimental Psychology, 50A, 439456.
Whaley, C. (1978). Word—nonword classification time. Journal of Verbal Learning and Verbal Behavior, 17, 143154.
Yap, M. J., & Balota, D. A. (2009). Visual word recognition of multisyllabic words. Journal of Memory and Language, 60(4), 502529.
Yarkoni, T., Balota, D., & Yap, M. (2008). Moving beyond Coltheart’s N: A new measure of orthographic similarity. Psychonomic Bulletin and Review, 15, 971979.
Yates, M., Locker, L., & Simpson, G. B. (2004). The influence of phonological neighborhood on visual word perception. Psychonomic Bulletin and Review, 11, 452457.
Type Description Title
Supplementary materials

Berger et al. supplementary material
Berger et al. supplementary material 1

 Word (28 KB)
28 KB
Supplementary materials

Berger et al. supplementary material
Berger et al. supplementary material 2

 Unknown (516 KB)
516 KB


  • Cynthia Berger (a1), Scott Crossley (a1) and Stephen Skalicky (a1)


Altmetric attention score

Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed.