Skip to main content Accessibility help
×
Hostname: page-component-7c8c6479df-94d59 Total loading time: 0 Render date: 2024-03-29T14:00:05.959Z Has data issue: false hasContentIssue false

28 - Lost In The Crowd

Crowdsourcing as a Research Method

from Part VI - Technology in Statistics and Research Methods

Published online by Cambridge University Press:  18 February 2019

Richard N. Landers
Affiliation:
University of Minnesota
Get access

Summary

Image of the first page of this content. For PDF version, please use the ‘Save PDF’ preceeding this image.'
Type
Chapter
Information
Publisher: Cambridge University Press
Print publication year: 2019

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Alonso, O. & Lease, M. (February, 2011). Crowdsourcing 101: Putting the WSDM of crowds to work for you. In WSDM (pp. 12). New York, NY: ACM.Google Scholar
Alkhatib, A., Bernstein, M. S., & Levi, M. (2017). Examining Crowd Work and Gig Work Through The Historical Lens of Piecework. Paper presented at the Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, Denver, Colorado, USA.CrossRefGoogle Scholar
Amazon.com. (2017). Amazon Mechanical Turk Getting Started Guide, 2017 Retrieved from http://docs.aws.amazon.com/es_es/AWSMechTurk/latest/AWSMechanicalTurkGettingStartedGuide/amt-gsg.pdf.Google Scholar
Ashford, S. J., George, E., & Blatt, R. (2007). 2 old assumptions, new work: The opportunities and challenges of research on nonstandard employmentThe Academy of Management Annals1(1), 65117.CrossRefGoogle Scholar
Behrend, T. S., Sharek, D. J., Meade, A. W., & Wiebe, E. N. (2011). The viability of crowdsourcing for survey research. Behavior Research Methods, 43(3), 800.CrossRefGoogle ScholarPubMed
Baobao Zhang, . (August 19, 2014). How to launch a survey on Amazon Mechanical Turk (MTurk) [Video file]. Retrieved from www.youtube.com/watch?v=qPcVe4SOmz8.Google Scholar
Berinsky, A., Huber, G., & Lenz, G. (2012). Using Mechanical Turk as a subject recruitment tool for experimental research. Political Analysis, 20, 351368.CrossRefGoogle Scholar
Buhrmester, M. D., Kwang, T., & Gosling, S. D. (2011). Amazon’s Mechanical Turk: A new source of inexpensive, yet high-quality, data? Perspectives on Psychological Science, 6, 35.CrossRefGoogle Scholar
Casey, L. S., Chandler, J., Levine, A. S., Proctor, A., & Strolovitch, D. Z. (2017). Intertemporal differences among MTurk Workers: Time-based sample variations and implications for online data collection. SAGE Open, 7(2), 2158244017712774.CrossRefGoogle Scholar
Casler, K., Bickel, L., & Hackett, E. (2013). Separate but equal? A comparison of participants and data gathered via Amazon’s MTurk, social media, and face-to-face behavioral testing. Computers in Human Behavior, 29(6), 21562160.CrossRefGoogle Scholar
Chambers, S., Nimon, K., & Anthony-McMann, P. (2016). A primer for conducting survey research using MTurk: Tips for the field. International Journal of Adult Vocational Education and Technology (IJAVET), 7(2), 5473.CrossRefGoogle Scholar
Chandler, J., Mueller, P., &Paolacci, G. (2014).Nonnaiveté among Amazon Mechanical Turk Workers: Consequences and solutions for behavioral researchers. Behavioral Research, 46, 112130.CrossRefGoogle ScholarPubMed
Chandler, J. J. & Paolacci, G. (2017). Lie for a dime: When most prescreening responses are honest but most study participants are impostors. Social Psychological and Personality Science, 8, 500508. doi: 10.1177/1948550617698203.CrossRefGoogle Scholar
Chandler, J., Paolacci, G., Peer, E., Mueller, P., & Ratliff, K. A. (2015). Using nonnaive participants can reduce effect sizes. Psychological Science, 26, 11311139. doi:10.1177/0956797615585115CrossRefGoogle ScholarPubMed
Chandler, J. & Shapiro, D. (2016). Conducting clinical research using crowdsourced convenience samples. Annual Review of Clinical Psychology, 12. 5381. doi: 10.1146/annurev-clinpsy-021815-093623.CrossRefGoogle ScholarPubMed
Cheung, J. H., Burns, D. K., Sinclair, R. R., & Sliter, M. (2017). Amazon Mechanical Turk in organizational psychology: An evaluation and practical recommendations. Journal of Business and Psychology, 32(4), 347361.CrossRefGoogle Scholar
Cotter v. Lyft, Inc., 60 F. Supp. 3d 1067, 1081–82 (ND Cal. 2015).Google Scholar
Devine, E. G., Waters, M. E., Putnam, M., Surprise, C., O’Malley, K., Richambault, C., … & Streeter, C. (2013). Concealment and fabrication by experienced research subjectsClinical Trials10(6), 935948.CrossRefGoogle ScholarPubMed
Deneme. (December 21, 2009). Deneme: A blog of experiments on Amazon Mechanical Turk. http://groups.csail.mit.edu/uid/deneme/?p=523.Google Scholar
Feitosa, J., Joseph, D., & Newman, D. (2015). Crowdsourcing and personality measurement equivalence: A warning about countries whose primary language is not English. Personality and Individual Differences, 75, 4752.CrossRefGoogle Scholar
George, E., & Ng, C. K. (2010). APA Handbook of Industrial and Organizational Psychology. APA.Google Scholar
Harms, P. D., & DeSimone, J. A. (2015). Caution! MTurk workers ahead—Fines doubledIndustrial and Organizational Psychology8(2), 183190.CrossRefGoogle Scholar
Hitlin, P. (2016). Research in the crowdsourcing age, a case study. Retrieved from http://assets. pewresearch.org/wpcontent/uploads/sites/14/2016/07/PI_2016.07.11_Mechanical-Turk_ FINAL.pdf.Google Scholar
Howe, , J. (2006). Crowdsourcing: a definition. Wired Blog Network: Crowdsourcing. Retrieved September 2018, from http://crowdsourcing.typepad.com/cs/2006/06/crowdsourcing_a.htmlGoogle Scholar
Huff, , C., & Tingley, , D. (2015). “Who are these people?” Evaluating the demographic characteristics and political preferences of MTurk survey respondentsResearch & Politics2(3), 2053168015604648.CrossRefGoogle Scholar
Ipeirotis, P. (2010). Demographics of Mechanical Turk [Working Article]. New York University. Retrieved from www.archive.nyu.edu/bitstream/2451/29585/2/CeDER-10-01.pdf.Google Scholar
Keith, M. G. & Harms, P. D. (2016). Is Mechanical Turk the answer to our sampling woes?. Industrial and Organizational Psychology, 9(1), 162167.CrossRefGoogle Scholar
Keith, M. G., Tay, L., & Harms, P. D. (2017). Systems perspective of Amazon Mechanical Turk for organizational research: Review and recommendations. Frontiers in Psychology, 8, 1359.CrossRefGoogle ScholarPubMed
Kittur, A., Nickerson, J. V., Bernstein, M., Gerber, E., Shaw, A., Zimmerman, J.,… Horton, J. (2013). The future of crowd work. Paper presented at the Proceedings of the 2013 conference on Computer supported cooperative work.CrossRefGoogle Scholar
Kleemann, F., Voß, G. G., & Rieder, K. (2008). Un(der)paid innovators: The commercial utilization of consumer work through crowdsourcing. Science, Technology & Innovation Studies, 4(1), 526.Google Scholar
Kuek, S. C., Paradi-Guilford, C., Fayomi, T., Imaizumi, S., Ipeirotis, P., Pina, P., & Singh, M. (2015). The global opportunity in online outsourcing. Washington, DC: The World Bank. Retrieved from http://documents.worldbank.org/curated/en/138371468000900555/The-global-opportunity-in-online-outsourcing.Google Scholar
Landers, R. & Behrend, T. (2015). An inconvenient truth: Arbitrary distinctions between organizational, Mechanical Turk, and other convenience samples. Industrial and Organizational Psychology: Perspectives on Science and Practice, 8(2), 142164.CrossRefGoogle Scholar
Levay, K. E., Freese, J., & Druckman, J. N. (2016). The demographic and political composition of Mechanical Turk samples. Sage Open, 6(1),117.CrossRefGoogle Scholar
Litman, L., Robinson, J., & Abberbock, T. (2017). Behavior Research Methods49(2), 433442.CrossRefGoogle Scholar
Mason, W. & Suri, S. (2011). Conducting behavioral research on Amazon’s Mechanical Turk. Behavioral Research, 44, 123.CrossRefGoogle Scholar
How, I., Meade, A. W., Behrend, T. S., & Lance, C. E. (2009). Dr. StrangeLOVE, or: How I learned to stop worrying and love omitted variables. In Lance, C. E. & Vandenberg, R. J. (Eds.), Statistical and methodological myths and urban legends: Doctrine, verity and fable in the organizational and social sciences (pp. 89106). New York, NY: Routledge/Taylor & Francis Group.Google Scholar
Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010). Running experiments on Amazon Mechanical Turk. Judgment and Decision Making, 5, 411419.CrossRefGoogle Scholar
Peer, E., Samat, S., Brandimarte, L.,& Acquisti, A. (2016). Beyond the Turk: An Empirical Comparison of Alternative Platforms for Crowdsourcing Online Behavioral Research (May 1, 2016). Retrieved from SSRN. https://ssrn.com/abstract=2594183 or http://dx.doi.org/10.2139/ssrn.2594183.CrossRefGoogle Scholar
Peer, E., Brandimarte, L., Samat, S., & Acquisti, A. (2017). Beyond the Turk: Alternative platforms for crowdsourcing behavioral research. Journal of Experimental Social Psychology, 70, 153163.CrossRefGoogle Scholar
Prassl, J. & Risak, M. (2015). Uber, Taskrabbit, and Co.: Platforms as ezmployers –Rethinking the legal analysis of crowdwork. Comparative Labor Law & Policy, 37, 619.Google Scholar
Rand, D. (2012). The promise of Mechanical Turk: How online labor markets can help theorists run behavioral experiments. Journal of Theoretical Biology, 299, 172179.CrossRefGoogle ScholarPubMed
Retelny, D., Robaszkiewicz, S., To, A., Lasecki, W. S., Patel, J., Rahmati, N.,… Bernstein, M. S. (2014). Expert crowdsourcing with flash teams. Paper presented at the Proceedings of the 27th annual ACM symposium on User interface software and technology.CrossRefGoogle Scholar
Roulin, N. (2015). Don’t throw the baby out with the bathwater: Comparing data quality of crowdsourcing, online panels, and student samples. Industrial and Organizational Psychology, 8(2), 190196.CrossRefGoogle Scholar
Shapiro, D., Chandler, J., &Mueller, P. (2013). Using Mechanical Turk to study clinical populations. Clinical Psychological Science, 1, 213220.CrossRefGoogle Scholar
Sheehan, K. & Pittman, M. (2016). Amazon’s Mechanical Turk for academics: The HIT handbook for social science research. Irvine, CA: Melvin & Leigh.Google Scholar
Stewart, N., Ungemach, C., Harris, A. J., Bartels, D. M., Newell, B. R., Paolacci, G., & Chandler, J. (2015). The average laboratory samples a population of 7,300 Amazon Mechanical Turk Workers. Judgment and Decision making, 10(5), 479.CrossRefGoogle Scholar
Williamson, V. (2016). On the ethics of crowdsourced research. PS: Political Science & Politics, 49(1), 7781Google Scholar

Save book to Kindle

To save this book to your Kindle, first ensure coreplatform@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about saving to your Kindle.

Note you can select to save to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be saved to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

Available formats
×

Save book to Dropbox

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Dropbox.

Available formats
×

Save book to Google Drive

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Google Drive.

Available formats
×