IDEAS home Printed from https://ideas.repec.org/a/sae/sagope/v6y2016i1p2158244016636433.html
   My bibliography  Save this article

The Demographic and Political Composition of Mechanical Turk Samples

Author

Listed:
  • Kevin E. Levay
  • Jeremy Freese
  • James N. Druckman

Abstract

One of the most notable recent developments in survey research is the increased usage of online convenience samples drawn from Amazon’s Mechanical Turk (MTurk). While scholars have noted various social and political differences (e.g., age, partisanship) between MTurk and population-based samples, the breadth and depth of these variations remain unclear. We investigate the extent to which MTurk samples differ from population samples, and the underlying nature of these differences. We do so by replicating items from the population-based American National Election Studies (ANES) 2012 Time Series Study in a survey administered to a sample of MTurk respondents. With few exceptions, we not only find that MTurk respondents differ significantly from respondents completing the 2012 ANES via the Web but also that most differences are reduced considerably when controlling for easily measurable sample features. Thus, MTurk respondents do not appear to differ fundamentally from population-based respondents in unmeasurable ways. This suggests that MTurk data can be used to advance research programs, particularly if researchers measure and account for a range of political and demographic variables as needed.

Suggested Citation

  • Kevin E. Levay & Jeremy Freese & James N. Druckman, 2016. "The Demographic and Political Composition of Mechanical Turk Samples," SAGE Open, , vol. 6(1), pages 21582440166, March.
    • Handle: RePEc:sae:sagope:v:6:y:2016:i:1:p:2158244016636433
    DOI: 10.1177/2158244016636433
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.1177/2158244016636433
    Download Restriction: no
    File URL: https://libkey.io/10.1177/2158244016636433?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Huber, Gregory A. & Hill, Seth J. & Lenz, Gabriel S., 2012. "Sources of Bias in Retrospective Decision Making: Experimental Evidence on Voters’ Limitations in Controlling Incumbents," American Political Science Review, Cambridge University Press, vol. 106(4), pages 720-741, November.
    2. Krupnikov, Yanna & Levine, Adam Seth, 2014. "Cross-Sample Comparisons and External Validity," Journal of Experimental Political Science, Cambridge University Press, vol. 1(1), pages 59-80, April.
    3. repec:cup:judgdm:v:5:y:2010:i:5:p:411-419 is not listed on IDEAS
    4. Mullinix, Kevin J. & Leeper, Thomas J. & Druckman, James N. & Freese, Jeremy, 2015. "The Generalizability of Survey Experiments," Journal of Experimental Political Science, Cambridge University Press, vol. 2(2), pages 109-138, January.
    5. Kevin Arceneaux, 2012. "Cognitive Biases and the Strength of Political Arguments," American Journal of Political Science, John Wiley & Sons, vol. 56(2), pages 271-285, April.
    6. Berinsky, Adam J. & Huber, Gregory A. & Lenz, Gabriel S., 2012. "Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk," Political Analysis, Cambridge University Press, vol. 20(3), pages 351-368, July.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Logan S. Casey & Jesse Chandler & Adam Seth Levine & Andrew Proctor & Dara Z. Strolovitch, 2017. "Intertemporal Differences Among MTurk Workers: Time-Based Sample Variations and Implications for Online Data Collection," SAGE Open, , vol. 7(2), pages 21582440177, June.
    2. Antonio A. Arechar & Simon Gächter & Lucas Molleman, 2018. "Conducting interactive experiments online," Experimental Economics, Springer;Economic Science Association, vol. 21(1), pages 99-131, March.
    3. Blaine G. Robbins, 2017. "Status, identity, and ability in the formation of trust," Rationality and Society, , vol. 29(4), pages 408-448, November.
    4. Shari De Baets & Dilek Önkal & Wasim Ahmed, 2022. "Do Risky Scenarios Affect Forecasts of Savings and Expenses?," Forecasting, MDPI, vol. 4(1), pages 1-28, February.
    5. Scott Simon Boddery & Damon Cann & Laura Moyer & Jeff Yates, 2023. "The role of cable news hosts in public support for Supreme Court decisions," Journal of Empirical Legal Studies, John Wiley & Sons, vol. 20(4), pages 1045-1069, December.
    6. Eugene Y. Chan & Jack Lin, 2022. "Political ideology and psychological reactance: how serious should climate change be?," Climatic Change, Springer, vol. 172(1), pages 1-22, May.
    7. Sönke Ehret & Sara M. Constantino & Elke U. Weber & Charles Efferson & Sonja Vogt, 2022. "Group Identities Make Fragile Tipping Points," CESifo Working Paper Series 9737, CESifo.
    8. Soojong Kim, 2019. "Directionality of information flow and echoes without chambers," PLOS ONE, Public Library of Science, vol. 14(5), pages 1-22, May.
    9. Schwaiger, Rene & Hueber, Laura, 2021. "Do MTurkers exhibit myopic loss aversion?," Economics Letters, Elsevier, vol. 209(C).
    10. Adam Seth Levine & Reuben Kline, 2017. "A new approach for evaluating climate change communication," Climatic Change, Springer, vol. 142(1), pages 301-309, May.
    11. Luke Fowler & Stephen Utych, 2021. "Are people better employees than machines? Dehumanizing language and employee performance appraisals," Social Science Quarterly, Southwestern Social Science Association, vol. 102(4), pages 2006-2019, July.
    12. Lala Muradova & Ross James Gildea, 2021. "Oil wealth and US public support for war," Conflict Management and Peace Science, Peace Science Society (International), vol. 38(1), pages 3-19, January.
    13. Kobayashi, Yoshiharu & Howell, Christopher & Heinrich, Tobias, 2021. "Vaccine hesitancy, state bias, and Covid-19: Evidence from a survey experiment using Phase-3 results announcement by BioNTech and Pfizer," Social Science & Medicine, Elsevier, vol. 282(C).
    14. David Johnson & John Barry Ryan, 2020. "Amazon Mechanical Turk workers can provide consistent and economically meaningful data," Southern Economic Journal, John Wiley & Sons, vol. 87(1), pages 369-385, July.
    15. Nicholas Haas & Rebecca B. Morton, 2018. "Saying versus doing: a new donation method for measuring ideal points," Public Choice, Springer, vol. 176(1), pages 79-106, July.
    16. Salil D. Benegal & Lyle A. Scruggs, 2018. "Correcting misinformation about climate change: the impact of partisanship in an experimental setting," Climatic Change, Springer, vol. 148(1), pages 61-80, May.
    17. Gordon Pennycook & Adam Bear & Evan T. Collins & David G. Rand, 2020. "The Implied Truth Effect: Attaching Warnings to a Subset of Fake News Headlines Increases Perceived Accuracy of Headlines Without Warnings," Management Science, INFORMS, vol. 66(11), pages 4944-4957, November.
    18. Adam S. Chilton, 2015. "The Laws of War and Public Opinion: An Experimental Study," Journal of Institutional and Theoretical Economics (JITE), Mohr Siebeck, Tübingen, vol. 171(1), pages 181-201, March.
    19. Morten Hjortskov, 2020. "Interpreting expectations: Normative and predictive expectations from the citizens’ viewpoint," Journal of Behavioral Public Administration, Center for Experimental and Behavioral Public Administration, vol. 3(1).
    20. Lamberova, Natalia, 2021. "The puzzling politics of R&D: Signaling competence through risky projects," Journal of Comparative Economics, Elsevier, vol. 49(3), pages 801-818.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:sagope:v:6:y:2016:i:1:p:2158244016636433. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.