Skip to main content Accessibility help
Hostname: page-component-55597f9d44-5zjcf Total loading time: 0.349 Render date: 2022-08-16T08:34:31.514Z Has data issue: true Feature Flags: { "shouldUseShareProductTool": true, "shouldUseHypothesis": true, "isUnsiloEnabled": true, "useRatesEcommerce": false, "useNewApi": true } hasContentIssue true

The New Fruit Fly for Applied Psychological Research

Published online by Cambridge University Press:  28 July 2015

Scott Highhouse*
Bowling Green State University
Don Zhang
Bowling Green State University
Correspondence concerning this article should be addressed to Scott Highhouse, Department of Psychology, Bowling Green State University, Bowling Green, OH 43403. E-mail:


Landers and Behrend (2015) present yet another attempt to limit reviewer and editor reliance on surface characteristics when evaluating the generalizability of study results (see also Campbell, 1986; Dipboye & Flanagan, 1979; Greenberg, 1987; Highhouse, 2009; Highhouse & Gillespie, 2009). Most of the earlier treatments of sample generalizability, however, have focused on the use of college students in (mostly) laboratory studies. Many industrial–organizational (I-O) scholars have experienced the hostility with which studies using students as participants receives. For instance, Jen Gillespie and I observed, “Reviewers and editors commonly assert that students should not be used to study workplace phenomena as though such a declaration requires no further explanation” (Highhouse & Gillespie, 2009, p. 247). The difference this time, however, is that Landers and Behrend (2015) are reacting to dismissals of research using Mechanical Turk (MTurk) workers to make inferences about behavior in organizations. Landers and Behrend (2015) make the important point that any research population is likely to be atypical on some dimensions and that all samples are samples of convenience (see also Oakes, 1972). We agree. Furthermore, we make two observations about MTurk: (a) We believe that it should be met with less resistance than student samples have historically faced, and (b) we suggest that it provides a unique opportunity to bring back randomized experimentation in I-O psychology.

Copyright © Society for Industrial and Organizational Psychology 2015 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)


Aguinis, H., & Bradley, K. J. (2014). Best practice recommendations for designing and implementing experimental vignette methodology studies. Organizational Research Methods.CrossRefGoogle Scholar
Brinberg, D., & McGrath, J. E. (1985). Validity and the research process. Newbury Park, CA: Sage.Google Scholar
Buhrmester, M., Kwang, T., & Gosling, S. D. (2011). Amazon's Mechanical Turk: A new source of inexpensive, yet high-quality data? Perspectives on Psychological Science, 6, 35.CrossRefGoogle ScholarPubMed
Calder, B. J., Phillips, L. W., & Tybout, A. M. (1981). Designing research for application. Journal of Consumer Research, 8, 197207.CrossRefGoogle Scholar
Campbell, J. P. (1986). Labs, fields, and straw issues. In Locke, E. A. (Ed.), Generalizing from laboratory to field settings (pp. 269279). Lexington, MA: Heath.Google Scholar
Colquitt, J. A. (2008). From the editors: Publishing laboratory research in AMJ: A question of when, not if. Academy of Management Journal, 51 (4), 616620.Google Scholar
Dipboye, R. L., & Flanagan, M. F. (1979). Research settings in industrial and organizational psychology: Are findings in the field more generalizable than in the laboratory? American Psychologist, 34, 141150.CrossRefGoogle Scholar
Farber, M. L. (1952). The college student as laboratory animal. American Psychologist, 7, 102.CrossRefGoogle Scholar
Fromkin, H. L., & Streufert, S. (1976). Laboratory experimentation. In Dunnette, M. D. (Ed.), Handbook of industrial and organizational psychology (pp. 415465). Chicago, IL: Rand McNally.Google Scholar
Gilovich, T., Griffin, D., & Kahneman, D. (2002). Heuristics and biases: The psychology of intuitive judgment. Cambridge, England: Cambridge University Press.CrossRefGoogle Scholar
Greenberg, J. (1987). The college sophomore as guinea pig: Setting the record straight. Academy of Management Review, 12, 157159.CrossRefGoogle Scholar
Greenberg, J., & Tomlinson, E. C. (2004). Situated experiments in organizations: Transplanting the lab to the field. Journal of Management, 30 (5), 703724.CrossRefGoogle Scholar
Highhouse, S. (2009). Designing experiments that generalize. Organizational Research Methods, 12, 554566.CrossRefGoogle Scholar
Highhouse, S., & Gillespie, J. Z. (2009). Do samples really matter that much? In Lance, C. E. & Vandenberg, R. J. (Eds.), Statistical and methodological myths and urban legends: Doctrine, verity and fable in the organizational and social sciences (pp. 249268). New York, NY: Routledge.Google Scholar
Kardes, F. R. (1996). In defense of experimental consumer psychology. Journal of Consumer Psychology, 5, 279296.CrossRefGoogle Scholar
Landers, R. N., & Behrend, T. S. (2015). An inconvenient truth: Arbitrary distinctions between organizations, Mechanical Turk, and other convenience samples. Industrial and Organizational Psychology: Perspectives on Science and Practice.Google Scholar
Locke, E. A. (1986). Generalizing from laboratory to field: Ecological validity or abstraction of essential elements? In Locke, E. A. (Ed.), Generalizing from laboratory to field settings (pp. 257267). Lexington, MA: Heath.Google Scholar
Mitchell, G. (2012). Revisiting truth or triviality the external validity of research in the psychological laboratory. Perspectives on Psychological Science, 7 (2), 109117.CrossRefGoogle ScholarPubMed
Mook, D. (1983). In defense of external invalidity. American Psychologist, 38, 379387.CrossRefGoogle Scholar
Oakes, W. (1972). External validity and the use of real people as subjects. American Psychologist, 27, 959962.CrossRefGoogle Scholar
Paolacci, G., & Chandler, J. (2014). Inside the Turk: Understanding Mechanical Turk as a participant pool. Current Directions in Psychological Science, 23, 184188.CrossRefGoogle Scholar
Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010). Running experiments on Amazon Mechanical Turk. Judgment and Decision Making, 5, 411419.Google Scholar
Sears, D. O. (1986). College sophomores in the laboratory: Influences of a narrow data base on social psychology's view of human nature. Journal of Personality and Social Psychology, 51 (3), 515.CrossRefGoogle Scholar
Shen, W., Kiger, T. B., Davies, S. E., Rasch, R. L., Simon, K. M., & Ones, D. S. (2011). Samples in applied psychology: Over a decade of research in review. Journal of Applied Psychology, 96 (5), 10551064.CrossRefGoogle Scholar
Stone-Romero, E. F. (2002). The relative validity and usefulness of various empirical research designs. In Rogelberg, S. G. (Ed.), Handbook of research methods in industrial and organizational psychology (pp. 7798). Malden, MA: Blackwell.Google Scholar
Cited by

Save article to Kindle

To save this article to your Kindle, first ensure is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about saving to your Kindle.

Note you can select to save to either the or variations. ‘’ emails are free but can only be saved to your device when it is connected to wi-fi. ‘’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

The New Fruit Fly for Applied Psychological Research
Available formats

Save article to Dropbox

To save this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you used this feature, you will be asked to authorise Cambridge Core to connect with your Dropbox account. Find out more about saving content to Dropbox.

The New Fruit Fly for Applied Psychological Research
Available formats

Save article to Google Drive

To save this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you used this feature, you will be asked to authorise Cambridge Core to connect with your Google Drive account. Find out more about saving content to Google Drive.

The New Fruit Fly for Applied Psychological Research
Available formats

Reply to: Submit a response

Please enter your response.

Your details

Please enter a valid email address.

Conflicting interests

Do you have any conflicting interests? *