IDEAS home Printed from https://ideas.repec.org/a/sae/sagope/v6y2016i1p2158244016636433.html
   My bibliography  Save this article

The Demographic and Political Composition of Mechanical Turk Samples

Author

Listed:
  • Kevin E. Levay
  • Jeremy Freese
  • James N. Druckman

Abstract

One of the most notable recent developments in survey research is the increased usage of online convenience samples drawn from Amazon’s Mechanical Turk (MTurk). While scholars have noted various social and political differences (e.g., age, partisanship) between MTurk and population-based samples, the breadth and depth of these variations remain unclear. We investigate the extent to which MTurk samples differ from population samples, and the underlying nature of these differences. We do so by replicating items from the population-based American National Election Studies (ANES) 2012 Time Series Study in a survey administered to a sample of MTurk respondents. With few exceptions, we not only find that MTurk respondents differ significantly from respondents completing the 2012 ANES via the Web but also that most differences are reduced considerably when controlling for easily measurable sample features. Thus, MTurk respondents do not appear to differ fundamentally from population-based respondents in unmeasurable ways. This suggests that MTurk data can be used to advance research programs, particularly if researchers measure and account for a range of political and demographic variables as needed.

Suggested Citation

  • Kevin E. Levay & Jeremy Freese & James N. Druckman, 2016. "The Demographic and Political Composition of Mechanical Turk Samples," SAGE Open, , vol. 6(1), pages 21582440166, March.
    • Handle: RePEc:sae:sagope:v:6:y:2016:i:1:p:2158244016636433
    DOI: 10.1177/2158244016636433
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.1177/2158244016636433
    Download Restriction: no
    File URL: https://libkey.io/10.1177/2158244016636433?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Huber, Gregory A. & Hill, Seth J. & Lenz, Gabriel S., 2012. "Sources of Bias in Retrospective Decision Making: Experimental Evidence on Voters’ Limitations in Controlling Incumbents," American Political Science Review, Cambridge University Press, vol. 106(4), pages 720-741, November.
    2. Gabriele Paolacci & Jesse Chandler & Panagiotis G. Ipeirotis, 2010. "Running experiments on Amazon Mechanical Turk," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 5(5), pages 411-419, August.
    3. Krupnikov, Yanna & Levine, Adam Seth, 2014. "Cross-Sample Comparisons and External Validity," Journal of Experimental Political Science, Cambridge University Press, vol. 1(1), pages 59-80, April.
    4. repec:cup:judgdm:v:5:y:2010:i:5:p:411-419 is not listed on IDEAS
    5. Mullinix, Kevin J. & Leeper, Thomas J. & Druckman, James N. & Freese, Jeremy, 2015. "The Generalizability of Survey Experiments," Journal of Experimental Political Science, Cambridge University Press, vol. 2(2), pages 109-138, January.
    6. Kevin Arceneaux, 2012. "Cognitive Biases and the Strength of Political Arguments," American Journal of Political Science, John Wiley & Sons, vol. 56(2), pages 271-285, April.
    7. Berinsky, Adam J. & Huber, Gregory A. & Lenz, Gabriel S., 2012. "Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk," Political Analysis, Cambridge University Press, vol. 20(3), pages 351-368, July.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Shari De Baets & Dilek Önkal & Wasim Ahmed, 2022. "Do Risky Scenarios Affect Forecasts of Savings and Expenses?," Forecasting, MDPI, vol. 4(1), pages 1-28, February.
    2. Antonio A. Arechar & Simon Gächter & Lucas Molleman, 2018. "Conducting interactive experiments online," Experimental Economics, Springer;Economic Science Association, vol. 21(1), pages 99-131, March.
    3. Logan S. Casey & Jesse Chandler & Adam Seth Levine & Andrew Proctor & Dara Z. Strolovitch, 2017. "Intertemporal Differences Among MTurk Workers: Time-Based Sample Variations and Implications for Online Data Collection," SAGE Open, , vol. 7(2), pages 21582440177, June.
    4. Blaine G. Robbins, 2017. "Status, identity, and ability in the formation of trust," Rationality and Society, , vol. 29(4), pages 408-448, November.
    5. Scott Simon Boddery & Damon Cann & Laura Moyer & Jeff Yates, 2023. "The role of cable news hosts in public support for Supreme Court decisions," Journal of Empirical Legal Studies, John Wiley & Sons, vol. 20(4), pages 1045-1069, December.
    6. Lamberova, Natalia, 2021. "The puzzling politics of R&D: Signaling competence through risky projects," Journal of Comparative Economics, Elsevier, vol. 49(3), pages 801-818.
    7. Eugene Y. Chan & Jack Lin, 2022. "Political ideology and psychological reactance: how serious should climate change be?," Climatic Change, Springer, vol. 172(1), pages 1-22, May.
    8. Stone, Daniel & sood, Gaurav & Garz, Marcel & Wallace, Justin, 2018. "The supply of media slant across outlets and demand for slant within-outlets: Evidence from US presidential campaign news," SocArXiv fy2we_v1, Center for Open Science.
    9. Sönke Ehret & Sara M. Constantino & Elke U. Weber & Charles Efferson & Sonja Vogt, 2022. "Group Identities Make Fragile Tipping Points," CESifo Working Paper Series 9737, CESifo.
    10. Austin M Strange & Ryan D Enos & Mark Hill & Amy Lakeman, 2019. "Online volunteer laboratories for human subjects research," PLOS ONE, Public Library of Science, vol. 14(8), pages 1-13, August.
    11. Soojong Kim, 2019. "Directionality of information flow and echoes without chambers," PLOS ONE, Public Library of Science, vol. 14(5), pages 1-22, May.
    12. Schwaiger, Rene & Hueber, Laura, 2021. "Do MTurkers exhibit myopic loss aversion?," Economics Letters, Elsevier, vol. 209(C).
    13. David Johnson & John Barry Ryan, 2020. "Amazon Mechanical Turk workers can provide consistent and economically meaningful data," Southern Economic Journal, John Wiley & Sons, vol. 87(1), pages 369-385, July.
    14. Adam Seth Levine & Reuben Kline, 2017. "A new approach for evaluating climate change communication," Climatic Change, Springer, vol. 142(1), pages 301-309, May.
    15. Garz, Marcel & Sood, Gaurav & Stone, Daniel F. & Wallace, Justin, 2020. "The supply of media slant across outlets and demand for slant within outlets: Evidence from US presidential campaign news," European Journal of Political Economy, Elsevier, vol. 63(C).
    16. Luke Fowler & Stephen Utych, 2021. "Are people better employees than machines? Dehumanizing language and employee performance appraisals," Social Science Quarterly, Southwestern Social Science Association, vol. 102(4), pages 2006-2019, July.
    17. Lala Muradova & Ross James Gildea, 2021. "Oil wealth and US public support for war," Conflict Management and Peace Science, Peace Science Society (International), vol. 38(1), pages 3-19, January.
    18. Kobayashi, Yoshiharu & Howell, Christopher & Heinrich, Tobias, 2021. "Vaccine hesitancy, state bias, and Covid-19: Evidence from a survey experiment using Phase-3 results announcement by BioNTech and Pfizer," Social Science & Medicine, Elsevier, vol. 282(C).
    19. Danny Hayes & Jennifer L. Lawless & Gail Baitinger, 2014. "Who Cares What They Wear? Media, Gender, and the Influence of Candidate Appearance," Social Science Quarterly, Southwestern Social Science Association, vol. 95(5), pages 1194-1212, December.
    20. Allison Carnegie & Lindsay R. Dolan, 2021. "The effects of rejecting aid on recipients’ reputations: Evidence from natural disaster responses," The Review of International Organizations, Springer, vol. 16(3), pages 495-519, July.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:sagope:v:6:y:2016:i:1:p:2158244016636433. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.