IDEAS home Printed from https://ideas.repec.org/a/plo/pone00/0157732.html
   My bibliography  Save this article

Measuring the Prevalence of Problematic Respondent Behaviors among MTurk, Campus, and Community Participants

Author

Listed:
  • Elizabeth A Necka
  • Stephanie Cacioppo
  • Greg J Norman
  • John T Cacioppo

Abstract

The reliance on small samples and underpowered studies may undermine the replicability of scientific findings. Large sample sizes may be necessary to achieve adequate statistical power. Crowdsourcing sites such as Amazon’s Mechanical Turk (MTurk) have been regarded as an economical means for achieving larger samples. Because MTurk participants may engage in behaviors which adversely affect data quality, much recent research has focused on assessing the quality of data obtained from MTurk samples. However, participants from traditional campus- and community-based samples may also engage in behaviors which adversely affect the quality of the data that they provide. We compare an MTurk, campus, and community sample to measure how frequently participants report engaging in problematic respondent behaviors. We report evidence that suggests that participants from all samples engage in problematic respondent behaviors with comparable rates. Because statistical power is influenced by factors beyond sample size, including data integrity, methodological controls must be refined to better identify and diminish the frequency of participant engagement in problematic respondent behaviors.

Suggested Citation

  • Elizabeth A Necka & Stephanie Cacioppo & Greg J Norman & John T Cacioppo, 2016. "Measuring the Prevalence of Problematic Respondent Behaviors among MTurk, Campus, and Community Participants," PLOS ONE, Public Library of Science, vol. 11(6), pages 1-19, June.
    • Handle: RePEc:plo:pone00:0157732
    DOI: 10.1371/journal.pone.0157732
    as

    Download full text from publisher

    File URL: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0157732
    Download Restriction: no
    File URL: https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0157732&type=printable
    Download Restriction: no
    File URL: https://libkey.io/10.1371/journal.pone.0157732?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Konstantinos K Tsilidis & Orestis A Panagiotou & Emily S Sena & Eleni Aretouli & Evangelos Evangelou & David W Howells & Rustam Al-Shahi Salman & Malcolm R Macleod & John P A Ioannidis, 2013. "Evaluation of Excess Significance Bias in Animal Studies of Neurological Diseases," PLOS Biology, Public Library of Science, vol. 11(7), pages 1-10, July.
    2. Paolacci, Gabriele & Chandler, Jesse & Ipeirotis, Panagiotis G., 2010. "Running experiments on Amazon Mechanical Turk," Judgment and Decision Making, Cambridge University Press, vol. 5(5), pages 411-419, August.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Rebecca R Carter & Analisa DiFeo & Kath Bogie & Guo-Qiang Zhang & Jiayang Sun, 2014. "Crowdsourcing Awareness: Exploration of the Ovarian Cancer Knowledge Gap through Amazon Mechanical Turk," PLOS ONE, Public Library of Science, vol. 9(1), pages 1-10, January.
    2. Yamada, Katsunori & Sato, Masayuki, 2013. "Another avenue for anatomy of income comparisons: Evidence from hypothetical choice experiments," Journal of Economic Behavior & Organization, Elsevier, vol. 89(C), pages 35-57.
    3. Sweldens, Steven & Puntoni, Stefano & Paolacci, Gabriele & Vissers, Maarten, 2014. "The bias in the bias: Comparative optimism as a function of event social undesirability," Organizational Behavior and Human Decision Processes, Elsevier, vol. 124(2), pages 229-244.
    4. S. Venus Jin & Aziz Muqaddam, 2019. "Product placement 2.0: “Do Brands Need Influencers, or Do Influencers Need Brands?”," Journal of Brand Management, Palgrave Macmillan, vol. 26(5), pages 522-537, September.
    5. Hsu, Dan K. & Burmeister-Lamp, Katrin & Simmons, Sharon A. & Foo, Maw-Der & Hong, Michelle C. & Pipes, Jesse D., 2019. "“I know I can, but I don't fit”: Perceived fit, self-efficacy, and entrepreneurial intention," Journal of Business Venturing, Elsevier, vol. 34(2), pages 311-326.
    6. Lutz, Christoph & Newlands, Gemma, 2018. "Consumer segmentation within the sharing economy: The case of Airbnb," Journal of Business Research, Elsevier, vol. 88(C), pages 187-196.
    7. Mariconda, Simone & Lurati, Francesco, 2015. "Does familiarity breed stability? The role of familiarity in moderating the effects of new information on reputation judgments," Journal of Business Research, Elsevier, vol. 68(5), pages 957-964.
    8. Gandullia, Luca & Lezzi, Emanuela, 2018. "The price elasticity of charitable giving: New experimental evidence," Economics Letters, Elsevier, vol. 173(C), pages 88-91.
    9. Tobias Schlager & Ashley V. Whillans, 2022. "People underestimate the probability of contracting the coronavirus from friends," Palgrave Communications, Palgrave Macmillan, vol. 9(1), pages 1-11, December.
    10. Charness, Gary & Gneezy, Uri & Kuhn, Michael A., 2013. "Experimental methods: Extra-laboratory experiments-extending the reach of experimental economics," Journal of Economic Behavior & Organization, Elsevier, vol. 91(C), pages 93-100.
    11. Gerhard, Patrick & Hoffmann, Arvid O.I. & Post, Thomas, 2017. "Past performance framing and investors’ belief updating: Is seeing long-term returns always associated with smaller belief updates?," Journal of Behavioral and Experimental Finance, Elsevier, vol. 15(C), pages 38-51.
    12. Orazi, Davide C. & Pizzetti, Marta, 2015. "Revisiting fear appeals: A structural re-inquiry of the protection motivation model," International Journal of Research in Marketing, Elsevier, vol. 32(2), pages 223-225.
    13. Haas, Nicholas & Hassan, Mazen & Mansour, Sarah & Morton, Rebecca B., 2021. "Polarizing information and support for reform," Journal of Economic Behavior & Organization, Elsevier, vol. 185(C), pages 883-901.
    14. Cantarella, Michele & Strozzi, Chiara, 2019. "Workers in the Crowd: The Labour Market Impact of the Online Platform Economy," IZA Discussion Papers 12327, Institute of Labor Economics (IZA).
    15. Armenak Antinyan & Luca Corazzini & Filippo Pavesi, 2018. "What Matters for Whistleblowing on Tax Evaders? Survey and Experimental Evidence," Working Papers 07/2018, University of Verona, Department of Economics.
    16. Atalay, Kadir & Bakhtiar, Fayzan & Cheung, Stephen & Slonim, Robert, 2014. "Savings and prize-linked savings accounts," Journal of Economic Behavior & Organization, Elsevier, vol. 107(PA), pages 86-106.
    17. Hindsley, Paul & McEvoy, David M. & Morgan, O. Ashton, 2020. "Consumer Demand for Ethical Products and the Role of Cultural Worldviews: The Case of Direct-Trade Coffee," Ecological Economics, Elsevier, vol. 177(C).
    18. Gökçe Esenduran & James A. Hill & In Joon Noh, 2020. "Understanding the Choice of Online Resale Channel for Used Electronics," Production and Operations Management, Production and Operations Management Society, vol. 29(5), pages 1188-1211, May.
    19. Capraro, Valerio & Rodriguez-Lara, Ismael & Ruiz-Martos, Maria J., 2020. "Preferences for efficiency, rather than preferences for morality, drive cooperation in the one-shot Stag-Hunt game," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 86(C).
    20. Azzam, Tarek & Harman, Elena, 2016. "Crowdsourcing for quantifying transcripts: An exploratory study," Evaluation and Program Planning, Elsevier, vol. 54(C), pages 63-73.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pone00:0157732. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: plosone (email available below). General contact details of provider: https://journals.plos.org/plosone/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.