IDEAS home Printed from https://ideas.repec.org/a/plo/pone00/0121595.html
   My bibliography  Save this article

Comparing the Similarity of Responses Received from Studies in Amazon’s Mechanical Turk to Studies Conducted Online and with Direct Recruitment

Author

Listed:
  • Christoph Bartneck
  • Andreas Duenser
  • Elena Moltchanova
  • Karolina Zawieska

Abstract

Computer and internet based questionnaires have become a standard tool in Human-Computer Interaction research and other related fields, such as psychology and sociology. Amazon’s Mechanical Turk (AMT) service is a new method of recruiting participants and conducting certain types of experiments. This study compares whether participants recruited through AMT give different responses than participants recruited through an online forum or recruited directly on a university campus. Moreover, we compare whether a study conducted within AMT results in different responses compared to a study for which participants are recruited through AMT but which is conducted using an external online questionnaire service. The results of this study show that there is a statistical difference between results obtained from participants recruited through AMT compared to the results from the participant recruited on campus or through online forums. We do, however, argue that this difference is so small that it has no practical consequence. There was no significant difference between running the study within AMT compared to running it with an online questionnaire service. There was no significant difference between results obtained directly from within AMT compared to results obtained in the campus and online forum condition. This may suggest that AMT is a viable and economical option for recruiting participants and for conducting studies as setting up and running a study with AMT generally requires less effort and time compared to other frequently used methods. We discuss our findings as well as limitations of using AMT for empirical studies.

Suggested Citation

  • Christoph Bartneck & Andreas Duenser & Elena Moltchanova & Karolina Zawieska, 2015. "Comparing the Similarity of Responses Received from Studies in Amazon’s Mechanical Turk to Studies Conducted Online and with Direct Recruitment," PLOS ONE, Public Library of Science, vol. 10(4), pages 1-23, April.
    • Handle: RePEc:plo:pone00:0121595
    DOI: 10.1371/journal.pone.0121595
    as

    Download full text from publisher

    File URL: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0121595
    Download Restriction: no
    File URL: https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0121595&type=printable
    Download Restriction: no
    File URL: https://libkey.io/10.1371/journal.pone.0121595?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Rajagopal, 2014. "The Human Factors," Palgrave Macmillan Books, in: Architecting Enterprise, chapter 9, pages 225-249, Palgrave Macmillan.
    2. John Horton & David Rand & Richard Zeckhauser, 2011. "The online laboratory: conducting experiments in a real labor market," Experimental Economics, Springer;Economic Science Association, vol. 14(3), pages 399-425, September.
    3. David J. Spiegelhalter & Nicola G. Best & Bradley P. Carlin & Angelika Van Der Linde, 2002. "Bayesian measures of model complexity and fit," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 64(4), pages 583-639, October.
    4. Alan Agresti & David B. Hitchcock, 2005. "Bayesian inference for categorical data analysis," Statistical Methods & Applications, Springer;Società Italiana di Statistica, vol. 14(3), pages 297-330, December.
    5. repec:cup:judgdm:v:5:y:2010:i:5:p:411-419 is not listed on IDEAS
    6. Paolacci, Gabriele & Chandler, Jesse & Ipeirotis, Panagiotis G., 2010. "Running experiments on Amazon Mechanical Turk," Judgment and Decision Making, Cambridge University Press, vol. 5(5), pages 411-419, August.
    7. Berinsky, Adam J. & Huber, Gregory A. & Lenz, Gabriel S., 2012. "Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk," Political Analysis, Cambridge University Press, vol. 20(3), pages 351-368, July.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Hongsha Wang & Qihui Chen & Bhagyashree Katare, 2023. "Nudging Chinese consumers to embrace sustainable milk consumption: How should information be provided?," Agribusiness, John Wiley & Sons, Ltd., vol. 39(S1), pages 1512-1534, December.
    2. Scott, Carol F. & Bay-Cheng, Laina Y. & Nochajski, Thomas H. & Lorraine Collins, R., 2024. "Emerging adults’ social media engagement & alcohol misuse: A multidimensional, person-centered analysis of risk," Children and Youth Services Review, Elsevier, vol. 159(C).
    3. Oliver Hahl, 2016. "Turning Back the Clock in Baseball: The Increased Prominence of Extrinsic Rewards and Demand for Authenticity," Organization Science, INFORMS, vol. 27(4), pages 929-953, August.
    4. Kamryn S. Morris & Eleanor K. Seaton & Masumi Iida & Sarah Lindstrom Johnson, 2020. "Racial Discrimination Stress, School Belonging, and School Racial Composition on Academic Attitudes and Beliefs among Black Youth," Social Sciences, MDPI, vol. 9(11), pages 1-20, October.
    5. Casella, Alessandra & Sanchez, Luis, 2019. "Storable Votes and Quadratic Voting. An Experiment on Four California Propositions," CEPR Discussion Papers 13479, C.E.P.R. Discussion Papers.
    6. Oliver Hahl & Jaekyung Ha, 2020. "Committed Diversification: Why Authenticity Insulates Against Penalties for Diversification," Organization Science, INFORMS, vol. 31(1), pages 1-22, January.
    7. repec:cup:judgdm:v:10:y:2015:i:6:p:538-548 is not listed on IDEAS
    8. Laura Biziou-van-Pol & Jana Haenen & Arianna Novaro & Andrés Occhipinti & Valerio Capraro, 2015. "Does telling white lies signal pro-social preferences?," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 10(6), pages 538-548, November.
    9. Collin Weigel & Laura A. Paul & Paul J. Ferraro & Kent D. Messer, 2021. "Challenges in Recruiting U.S. Farmers for Policy‐Relevant Economic Field Experiments," Applied Economic Perspectives and Policy, John Wiley & Sons, vol. 43(2), pages 556-572, June.
    10. Young Kim & Hyojung Park, 2017. "Is there Still a PR Problem Online? Exploring the Effects of Different Sources and Crisis Response Strategies in Online Crisis Communication Via Social Media," Corporate Reputation Review, Palgrave Macmillan, vol. 20(1), pages 76-104, February.
    11. Ksenia Koroleva & Jasminko Novak, 2020. "How to Engage with Sustainability Issues We Rarely Experience? A Gamification Model for Collective Awareness Platforms in Water-Related Sustainability," Sustainability, MDPI, vol. 12(2), pages 1-24, January.
    12. Tasci, Asli D.A., 2018. "Testing the cross-brand and cross-market validity of a consumer-based brand equity (CBBE) model for destination brands," Tourism Management, Elsevier, vol. 65(C), pages 143-159.
    13. Feenstra, Sanne & Jordan, Jennifer & Walter, Frank & Stoker, Janka I., 2020. "Antecedents of leaders' power sharing: The roles of power instability and distrust," Organizational Behavior and Human Decision Processes, Elsevier, vol. 157(C), pages 115-128.
    14. Debra A. Hoffmann & Jenna M. Marx & Jacob M. Burmeister & Dara R. Musher-Eizenman, 2018. "Friday Night Is Pizza Night: A Comparison of Children’s Dietary Intake and Maternal Perceptions and Feeding Goals on Weekdays and Weekends," IJERPH, MDPI, vol. 15(4), pages 1-12, April.
    15. Paul Delfabbro & Neophytos Georgiou & Catia Malvaso & Daniel King, 2020. "Is Self-Reported Propensity for Everyday Illusions of Control Higher in Gamblers and Is It Associated With Gambling-Specific Erroneous Beliefs?," SAGE Open, , vol. 10(1), pages 21582440198, January.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Gandullia, Luca & Lezzi, Emanuela & Parciasepe, Paolo, 2020. "Replication with MTurk of the experimental design by Gangadharan, Grossman, Jones & Leister (2018): Charitable giving across donor types," Journal of Economic Psychology, Elsevier, vol. 78(C).
    2. Haas, Nicholas & Hassan, Mazen & Mansour, Sarah & Morton, Rebecca B., 2021. "Polarizing information and support for reform," Journal of Economic Behavior & Organization, Elsevier, vol. 185(C), pages 883-901.
    3. Cantarella, Michele & Strozzi, Chiara, 2019. "Workers in the Crowd: The Labour Market Impact of the Online Platform Economy," IZA Discussion Papers 12327, Institute of Labor Economics (IZA).
    4. Atalay, Kadir & Bakhtiar, Fayzan & Cheung, Stephen & Slonim, Robert, 2014. "Savings and prize-linked savings accounts," Journal of Economic Behavior & Organization, Elsevier, vol. 107(PA), pages 86-106.
    5. Azzam, Tarek & Harman, Elena, 2016. "Crowdsourcing for quantifying transcripts: An exploratory study," Evaluation and Program Planning, Elsevier, vol. 54(C), pages 63-73.
    6. Wladislaw Mill & Cornelius Schneider, 2023. "The Bright Side of Tax Evasion," CESifo Working Paper Series 10615, CESifo.
    7. Prissé, Benjamin & Jorrat, Diego, 2022. "Lab vs online experiments: No differences," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 100(C).
    8. Valerio Capraro & Hélène Barcelo, 2021. "Punishing defectors and rewarding cooperators: Do people discriminate between genders?," Journal of the Economic Science Association, Springer;Economic Science Association, vol. 7(1), pages 19-32, September.
    9. Lefgren, Lars J. & Sims, David P. & Stoddard, Olga B., 2016. "Effort, luck, and voting for redistribution," Journal of Public Economics, Elsevier, vol. 143(C), pages 89-97.
    10. Tim Straub & Henner Gimpel & Florian Teschner & Christof Weinhardt, 2015. "How (not) to Incent Crowd Workers," Business & Information Systems Engineering: The International Journal of WIRTSCHAFTSINFORMATIK, Springer;Gesellschaft für Informatik e.V. (GI), vol. 57(3), pages 167-179, June.
    11. Nicolas Jacquemet & Alexander G James & Stéphane Luchini & James J Murphy & Jason F Shogren, 2021. "Do truth-telling oaths improve honesty in crowd-working?," PLOS ONE, Public Library of Science, vol. 16(1), pages 1-18, January.
    12. Chandler, Dana & Kapelner, Adam, 2013. "Breaking monotony with meaning: Motivation in crowdsourcing markets," Journal of Economic Behavior & Organization, Elsevier, vol. 90(C), pages 123-133.
    13. Harman, Elena & Azzam, Tarek, 2018. "Incorporating public values into evaluative criteria: Using crowdsourcing to identify criteria and standards," Evaluation and Program Planning, Elsevier, vol. 71(C), pages 68-82.
    14. Christ, Margaret H. & Vance, Thomas W., 2018. "Cascading controls: The effects of managers’ incentives on subordinate effort to help or harm," Accounting, Organizations and Society, Elsevier, vol. 65(C), pages 20-32.
    15. Brañas-Garza, Pablo & Capraro, Valerio & Rascón-Ramírez, Ericka, 2018. "Gender differences in altruism on Mechanical Turk: Expectations and actual behaviour," Economics Letters, Elsevier, vol. 170(C), pages 19-23.
    16. Michele Cantarella & Chiara Strozzi, 2018. "Labour market effects of crowdwork in US and EU: an empirical investigation," Department of Economics 0139, University of Modena and Reggio E., Faculty of Economics "Marco Biagi".
    17. David Johnson & John Barry Ryan, 2020. "Amazon Mechanical Turk workers can provide consistent and economically meaningful data," Southern Economic Journal, John Wiley & Sons, vol. 87(1), pages 369-385, July.
    18. Binder, Carola Conces, 2022. "Time-of-day and day-of-week variations in Amazon Mechanical Turk survey responses," Journal of Macroeconomics, Elsevier, vol. 71(C).
    19. Antonio A. Arechar & Gordon T. Kraft-Todd & David G. Rand, 2017. "Turking overtime: how participant characteristics and behavior vary over time and day on Amazon Mechanical Turk," Journal of the Economic Science Association, Springer;Economic Science Association, vol. 3(1), pages 1-11, July.
    20. Anthony C. Bucaro & Kevin E. Jackson & Jeremy B. Lill, 2020. "The Influence of Corporate Social Responsibility Measures on Investors' Judgments When Integrated in a Financial Report Versus Presented in a Separate Report," Contemporary Accounting Research, John Wiley & Sons, vol. 37(2), pages 665-695, June.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pone00:0121595. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: plosone (email available below). General contact details of provider: https://journals.plos.org/plosone/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.