Skip to main content Accessibility help
×
Home

METHODOLOGICAL CHOICES IN RATING SPEECH SAMPLES

  • Mary Grantham O’Brien (a1)

Abstract

Much pronunciation research critically relies upon listeners’ judgments of speech samples, but researchers have rarely examined the impact of methodological choices. In the current study, 30 German native listeners and 42 German L2 learners (L1 English) rated speech samples produced by English-German L2 learners along three continua: accentedness, fluency, and comprehensibility. The goal was to determine whether rating condition, that is, (a) whether each speech sample is rated along all three continua after it is heard once or (b) whether all speech samples are rated along one continuum before being rated along the next continuum, and continuum order (e.g., whether participants rate speech samples for accentedness before comprehensibility or fluency) have an effect on listeners’ ratings. Results indicate no significant overall effects of rating condition or continuum order, but there is evidence of rating condition effects by listener group. The results have implications for laboratory and classroom assessments of L2 speech.

Copyright

References

Hide All
Abrahamsson, N., & Hyltenstam, K. (2008). The Robustness of aptitude effects in near-native second language acquisition. Studies in Second Language Acquisition, 30, 481509. http://dx.doi.org/10.1017/S027226310808073X.
American Council on the Teaching of Foreign Languages. (2012). ACTFL Proficiency Guidelines. http://actflproficiencyguidelines2012.org/speaking.
Anderson-Hsieh, J., Johnson, J., & Koehler, K. (1992). The relationship between native speaker judgments of nonnative pronunciation and deviance in segmentals, prosody and syllable structure. Language Learning, 42, 529555. http://dx.doi.org/10.1111/j.1467-1770.1992.tb01043.x.
Bongaerts, T., Mennen, S., & van der Slik, F. (2000). Authenticity of pronunciation in naturalistic second language acquisition: The case of very advanced learners of Dutch as a second language. Studia Linguistica, 54, 298308. http://dx.doi.org/10.1111/1467-9582.00069.
Bortfeld, H., Leon, S. D., Bloom, J. E., Schober, M. F., & Brennan, S. E. (2001). Disfluency rates in conversation: Effects of age, relationship, topic, role, and gender. Language and Speech, 44, 123147.
Brand, C., & Götz, S. (2011). Fluency versus accuracy in advanced spoken learner language: A multi-method approach. International Journal of Corpus Linguistics, 16, 255275. http://dx.doi.org/10.1075/ijcl.16.2.05bra.
Braun, B., Lemhöfer, K., & Mani, N. (2011). Perceiving unstressed vowels in foreign-accented English. The Journal of the Acoustical Society of America, 129, 376387. http://dx.doi.org/10.1121/1.3500688.
Caspers, J. (2010). The influence of erroneous stress position and segmental errors on intelligibility, comprehensibility and foreign accent in Dutch as a second language. Linguistics in the Netherlands, 27, 1729. http://dx.doi.org/10.1075/avt.27.03cas.
Chakraborty, R., & Goffman, L. (2011). Production of lexical stress in non-native speakers of American English: Kinematic correlates of stress and transfer. Journal of Speech, Language, and Hearing Research, 54, 821835. http://dx.doi.org/10.1044/1092-4388(2010/09-0018.
Chen, H. (2010). Second language timing patterns and their effects on native listeners’ perceptions. Concentric: Studies in Linguistics, 36, 183212.
Clark, H. H., & Fox Tree, J. E. (2002). Using uh and um in spontaneous speaking. Cognition, 84, 73111.
Council of Europe. (2001). Common European framework of reference for languages: Learning, teaching, assessment. Cambridge: Cambridge University Press. http://www.coe.int/t/dg4/linguistic/source/framework_en.pdf.
Cucchiarini, C., Strik, H., & Boves, L. (2002). Quantitative assessment of second language learners’ fluency: Comparisons between read and spontaneous speech. Journal of the Acoustical Society of America, 111, 28622873. http://dx.doi.org/10.1121/1.1471894.
de Jong, N. H., Steinel, M. P., Florijn, A., Schoonen, R., & Hulstijn, J. H. (2013). Linguistic skills and speaking fluency in a second language. Applied Psycholinguistics, 34, 893916. http://dx.doi.org/10.1017/S0142716412000069.
Derwing, T. M., & Munro, M. J. (1997). Accent, intelligibility, and comprehensibility: Evidence from four L1s. Studies in Second Language Acquisition, 19, 116.
Derwing, T. M., & Munro, M. J. (2009a). Putting accent in its place: Rethinking obstacles to communication. Language Teaching, 42, 476490. http://dx.doi.org/10.1017/S026144480800551X.
Derwing, T. M., & Munro, M. J. (2009b). Comprehensibility as a factor in listener interaction preferences: Implications for the workplace. The Canadian Modern Language Review/La Revue Canadienne Des Langues Vivantes, 66, 181202. http://dx.doi.org/10.3138/cmlr.66.2.181.
Derwing, T. M., & Munro, M. J. (2013). The development of L2 oral language skills in two L1 groups: A 7-year study. Language Learning, 63, 163185. http://dx.doi.org/10.1111/lang.12000.
Derwing, T. M., Munro, M. J., & Thomson, R. I. (2008). A longitudinal study of ESL learners’ fluency and comprehensibility development. Applied Linguistics, 29, 359380. http://dx.doi.org/10.1093/applin/amm041.
Derwing, T. M., Rossiter, M. J., & Ehrensberger-Dow, M. (2002). “They speaked and wrote real good”: Judgements of non-native and native grammar. Language Awareness, 11, 8499. http://dx.doi.org/10.1080/09658410208667048.
Derwing, T. M., Rossiter, M. J., Munro, M. J., & Thomson, R. I. (2004). Second language fluency: Judgments on different tasks. Language Learning, 54, 655679. http://dx.doi.org/10.1111/j.1467-9922.2004.00282.x.
Flege, J. E. (1984). The detection of French accent by American listeners. Journal of the Acoustical Society of America, 76, 692707. http://dx.doi.org/10.1121/1.391256.
Flege, J. E. (1988). Factors affecting degree of perceived foreign accent in English sentences. Journal of the Acoustical Society of America, 84, 7079. http://dx.doi.org/10.1121/1.396876.
Freed, B. F. (1995). What makes us think that students who study abroad become fluent? In Freed, B. F. (Ed.), Second language acquisition in a study-abroad context (pp. 123148). Amsterdam: John Benjamins.
Goethe Institut. (2004). Einstufungstest [Placement test]. http://www.goethe.de/cgi-bin/einstufungstest/einstufungstest.pl.
González-Bueno, M. (1997). Voice-onset-time in the perception of foreign accent by native listeners of Spanish. International Review of Applied Linguistics, 35, 251267. http://dx.doi.org/10.1515/iral.1997.35.4.251.
Gui, M. (2012). Exploring differences between Chinese and American EFL teachers’ evaluations of speech performance. Language Assessment Quarterly, 9, 186203. http://dx.doi.org/10.1080/15434303.2011.614030.
Gut, U. (2009). Non-native speech: A corpus-based analysis of phonological and phonetic properties of L2 English and German. Frankfurt: Peter Lang.
Hahn, L. D. (2004). Primary stress and intelligibility: Research to motivate the teaching of suprasegmentals. TESOL Quarterly, 38, 201223. http://dx.doi.org/10.2307/3588378.
Hanna, P. L., & Allen, A. (2012). Educator assessment: Accent as a measure of fluency in Arizona. Educational Policy, 27, 711738. http://dx.doi.org/10.1177/0895904811429293.
Hayes-Harb, R., & Hacking, J. F. (2015). Beyond rating data: What do listeners believe underlies their accentedness judgments? Journal of Second Language Pronunciation, 1, 4364. http://dx.doi.org/10.1075/jslp.1.1.02hay.
Hughes, A., & Lascaratou, C. (1982). Competing criteria for error gravity. ELT Journal, 36, 175182. http://dx.doi.org/10.1093/elt/36.3.175.
Isaacs, T., & Thomson, R. I. (2013). Rater experience, rating scale length, and judgments of L2 pronunciation: Revisiting research conventions. Language Assessment Quarterly, 10, 135159. http://dx.doi.org/10.1080/15434303.2013.769545.
Isaacs, T., & Trofimovich, P. (2011). Phonological memory, attention control, and musical ability: Effects of individual differences on rater judgments of second language speech. Applied Psycholinguistics, 32, 113140. http://dx.doi.org/10.1017/S0142716410000317.
Isaacs, T., Trofimovich, P., Yu, G., & Chereau, B. M. (2015). Examining the linguistic aspects of speech that most efficiently discriminate between upper levels of the revised IELTS Pronunciation scale. IELTS Research Report Series, 4. https://www.ielts.org/pdf/Isaacs%20et%20al%20final.pdf.
Kang, O. (2010). Relative salience of suprasegmental features on judgments of L2 comprehensibility and accentedness. System, 38, 301315. http://dx.doi.org/10.1016/j.system.2010.01.005.
Kang, O. (2012). Impact of rater characteristics and prosodic features of speaker accentedness on ratings of international teaching assistants’ oral performance. Language Assessment Quarterly, 9, 249269. http://dx.doi.org/10.1080/15434303.2011.642631.
Kang, O., Rubin, D., & Pickering, L. (2010). Suprasegmental measures of accentedness and judgments of language learner proficiency in oral English. The Modern Language Journal, 94, 554566. http://dx.doi.org/10.1111/j.1540-4781.2010.01091.x.
Kennedy, S., & Trofimovich, P. (2008). Intelligibility, comprehensibility and accentedness of L2 speech: The role of listener experience and semantic context. The Canadian Modern Language Review, 64, 459489. http://dx.doi.org/10.3138/cmlr.64.3.459.
Kim, Y.-H. (2009). An investigation into native and non-native teachers’ judgments of oral English performance: A mixed methods approach. Language Testing, 26, 187217. http://dx.doi.org/10.1177/0265532208101010.
Koponen, M., & Riggenbach, H. (2000). Overview: Varying perspectives on fluency. In Riggenbach, H. (Ed.), Perspectives on fluency (pp. 524). Ann Arbor: University of Michigan Press.
Kormos, J., & Dénes, M. (2004). Exploring measures and perceptions of fluency in the speech of second language learners. System, 32, 145164. http://dx.doi.org/10.1016/j.system.2004.01.001.
Leikin, M., Ibrahim, R., Eviatar, Z., & Sapir, S. (2009). Listening with an accent: Speech perception in a second language by late bilinguals. Journal of Psycholinguistic Research, 38, 447457. http://dx.doi.org/10.1007/s10936-009-9099-1.
Lennon, P. (1990). Investigating fluency in EFL: A quantitative approach. Language Learning, 40, 387417. http://dx.doi.org/10.1111/j.1467-1770.1990.tb00669.x.
Long, M. H. (1990). Maturational constraints on language development. Studies in Second Language Acquisition, 12, 251285. http://dx.doi.org/10.1017/S0272263100009165.
Mackay, I. R. A., Flege, J. E., & Imai, S. (2006). Evaluating the effects of chronological age and sentence duration on degree of perceived foreign accent. Applied Psycholinguistics, 27, 157183. http://dx.doi.org/10.1017/S0142716406060231.
Major, R. C. (2007). Identifying a foreign accent in an unfamiliar language. Studies in Second Language Acquisition, 29, 539556. http://dx.doi.org/10.1017/S0272263107070428.
Molnar, H. (2010). The influence of the age factor on pronunciation competence in a second language: Results of a pilot study with learners of German as a second language [Der Einfluss des Faktors Alter auf die Aussprachekompetenz in der L2: ErgebnisseeinerPilotstudiemitDaZ-Lernern]. Zeitschriftfür Interkulturellen Fremdsprachenunterricht, 15, 4260. http://zif.spz.tu-darmstadt.de/jg-15-1/beitrag/Molnar1.htm.
Moyer, A. (1999). Ultimate attainment in L2 phonology. Studies in Second Language Acquisition, 21, 81108.
Moyer, A. (2011). An investigation of experience in L2 phonology: Does quality matter more than quantity? The Canadian Modern Language Review/La Revue Canadienne Des Langues Vivantes, 67, 191216. http://dx.doi.org/10.3138/cmlr.67.2.191.
Munro, M. (1995). Nonsegmental factors in foreign accent: Rating of foreign speech. Studies in Second Language Acquisition, 17, 1733. http://dx.doi.org/10.1017/S0272263100013735.
Munro, M. J., & Derwing, T. M. (1994). Evaluations of foreign accent in extemporaneous and read material. Language Testing, 11, 253266. http://dx.doi.org/10.1177/026553229401100302.
Munro, M., & Derwing, T. M. (1995). Foreign accent, comprehensibility and intelligibility in the speech of second language learners. Language Learning, 45, 7397.
Munro, M. J., & Derwing, T. M. (2001). Modeling perceptions and the accentedness and comprehensibility of L2 speech. Studies in Second Language Acquisition, 23, 451468.
Munro, M. J., Derwing, T. M., & Burgess, C. S. (2010). Detection of nonnative speaker status from content-masked speech. Speech Communication, 52, 626637. http://dx.doi.org/10.1016/j.specom.2010.02.013.
Munro, M. J., Derwing, T. M., & Morton, S. L. (2006). The mutual intelligibility of L2 speech. Studies in Second Language Acquisition, 28, 111131. http://dx.doi.org/10.1017/S0272263106060049.
Neuendorf, J. A. (2010). The intelligibility and comprehensibility of learner speech in Russian: A study in the salience of pronunciation, lexicon, grammar and syntax (Unpublished doctoral dissertation). Bryn Mawr College.
O’Brien, M. G. (2004). Pronunciation matters. Die Unterrichtspraxis: Teaching German, 37, 19. http://dx.doi.org/10.1111/j.1756-1221.2004.tb00068.x.
O’Brien, M. G. (2014). L2 learners’ assessments of accentedness, fluency and comprehensibility of native and nonnative German speech. Language Learning, 64, 715748. http://dx.doi.org/10.1111/lang.12082.
Pinget, A-F., Bosker, H. R., Quené, H., & de Jong, N. H. (2014). Native speakers’ perceptions of fluency and accent in L2 speech. Language Testing, 31, 349365. http://dx.doi.org/10.1177/0265532214526177.
Préfontaine, Y. (2013). Perceptions of French fluency in second language speech production. Canadian Modern Language Review, 69, 324348. http://dx.doi.org/10.3138/cmlr.1748.
Reinisch, E., & Weber, A. (2012). Adapting to suprasegmental lexical stress errors in foreign-accented speech. Journal of the Acoustical Society of America, 132, 11651176. http://dx.doi.org/10.1121/1.4730884.
Riazantseva, A. (2001). Second language proficiency and pausing: A study of Russian speakers of English. Studies in Second Language Acquisition, 23, 497526.
Riggenbach, H. (1991). Towards an understanding of fluency: A microanalysis of nonnative speaker conversations. Discourse Processes, 14, 423441.
Riney, T. J., Takada, M., & Ota, M. (2000). Segmentals and global foreign accent: The Japanese flap in EFL. TESOL Quarterly, 34, 711737. http://dx.doi.org/10.2307/3587782.
Riney, T. J., Takagi, N., & Inutsuka, K. (2005). Phonetic parameters and perceptual judgments of accent in English by American and Japanese listeners. TESOL Quarterly, 39, 441466. http://dx.doi.org/10.2307/3588489.
Rossiter, M. J. (2009). Perceptions of L2 fluency by native and non-native speakers of English. The Canadian Modern Language Review, 65, 395412. http://dx.doi.org/10.3138/cmlr.65.3.395.
Saito, K., Trofimovich, P., & Isaacs, T. (forthcoming). Second language speech production: Investigating linguistic correlates of comprehensibility and accentedness for learners at different ability levels. Applied Psycholinguistics. http://dx.doi.org/10.1017/S0142716414000502.
Saito, K., Webb, S., Trofimovich, P., & Isaacs, T. (forthcoming). Lexical correlates of comprehensibility versus accentedness in second language speech. Bilingualism: Language and Cognition. http://dx.doi.org/10.1017/S1366728915000255.
Segalowitz, N. (2007). Access fluidity, attention control, and the acquisition of fluency in a second language. TESOL Quarterly, 41, 181186. http://dx.doi.org/10.1002/j.1545-7249.2007.tb00047.x.
Towell, R., Hawkins, R., & Bazergui, N. (1996). The development of fluency in advanced learners of French. Applied Linguistics, 17, 84119. http://dx.doi.org/10.1093/applin/17.1.84.
Trofimovich, P., & Baker, W. (2006). Learning second language suprasegmentals: Effect of L2 experience on prosody and fluency characteristics of L2 speech. Studies in Second Language Acquisition, 28, 130. http://dx.doi.org/10.1017/S0272263106060013.
Trofimovich, P., & Isaacs, T. (2012). Disentangling accent from comprehensibility. Bilingualism: Language and Cognition, 15, 905916. http://dx.doi.org/10.1017/S1366728912000168.
Trofimovich, R., Gatbonton, E., & Segalowitz, N. (2007). A dynamic look at L2 phonological learning: Seeking processing explanations for implicational phenomena. Studies in Second Language Acquisition, 29, 407448. http://dx.doi.org/10.1017/S027226310707026X.
Wilkerson, M. (2010). Identifying accent in German: A comparison of native and non-native listeners. Die Unterrichtspraxis, 43, 144153. http://dx.doi.org/10.1111/j.1756-1221.2010.00076.x.
Winke, P., & Gass, S. (2013). The influence of second language experience and accent familiarity on oral proficiency rating: A qualitative investigation. TESOL Quarterly, 47, 762789. http://dx.doi.org/10.1002/tesq.73.

Related content

Powered by UNSILO

METHODOLOGICAL CHOICES IN RATING SPEECH SAMPLES

  • Mary Grantham O’Brien (a1)

Metrics

Altmetric attention score

Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed.