IDEAS home Printed from https://ideas.repec.org/p/zbw/esprep/249771.html
   My bibliography  Save this paper

Interactive Experiments in Toloka

Author

Listed:
  • Chapkovski, Philipp

Abstract

The popularity of online behavioral experiments grew steadily even before the COVID-19 pandemic. With the start of lockdowns, online studies were often the only available option for the behavioral economists, sociologists and political scientists. The usage of most well-known platforms such as mTurk was so intensive that it harmed the quality of data. But even before the pandemics-induced quality crisis, online studies were limited in scope, since real-time interactions between participants were hard to achieve due to the large proportion of drop-outs and issues with creating stable groups. Using the crowdsourcing platform Toloka, we successfully ran several multi-round interactive experiments. Toloka’s large online audience, relatively low exposure of participants to sociological surveys and behavioral studies, and a convenient application programming interface makes it a perfect tool to run behavioral studies that require real-time interactions of participants.

Suggested Citation

  • Chapkovski, Philipp, 2022. "Interactive Experiments in Toloka," EconStor Preprints 249771, ZBW - Leibniz Information Centre for Economics.
    • Handle: RePEc:zbw:esprep:249771
    DOI: 10.2139/ssrn.3727227
    as

    Download full text from publisher

    File URL: https://www.econstor.eu/bitstream/10419/249771/1/SSRN-id3727227.pdf
    Download Restriction: no
    File URL: https://libkey.io/10.2139/ssrn.3727227?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Fischbacher, Urs & Gächter, Simon & Quercia, Simone, 2012. "The behavioral validity of the strategy method in public good experiments," Journal of Economic Psychology, Elsevier, vol. 33(4), pages 897-913.
    2. Johannes Abeler & Armin Falk & Lorenz Goette & David Huffman, 2011. "Reference Points and Effort Provision," American Economic Review, American Economic Association, vol. 101(2), pages 470-492, April.
    3. Charness, Gary & Gneezy, Uri & Halladay, Brianna, 2016. "Experimental methods: Pay one or pay all," Journal of Economic Behavior & Organization, Elsevier, vol. 131(PA), pages 141-150.
    4. Antonio A. Arechar & Simon Gächter & Lucas Molleman, 2018. "Conducting interactive experiments online," Experimental Economics, Springer;Economic Science Association, vol. 21(1), pages 99-131, March.
    5. Heinz, Matthias & Juranek, Steffen & Rau, Holger A., 2012. "Do women behave more reciprocally than men? Gender differences in real effort dictator games," Journal of Economic Behavior & Organization, Elsevier, vol. 83(1), pages 105-110.
    6. R. Mark Isaac & James M. Walker, 1988. "Group Size Effects in Public Goods Provision: The Voluntary Contributions Mechanism," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 103(1), pages 179-199.
    7. Erin L. Krupka & Roberto A. Weber, 2013. "Identifying Social Norms Using Coordination Games: Why Does Dictator Game Sharing Vary?," Journal of the European Economic Association, European Economic Association, vol. 11(3), pages 495-524, June.
    8. David J. Cooper & Hanming Fang, 2008. "Understanding Overbidding in Second Price Auctions: An Experimental Study," Economic Journal, Royal Economic Society, vol. 118(532), pages 1572-1595, October.
    9. Todd L. Cherry & Peter Frykblom & Jason F. Shogren, 2002. "Hardnose the Dictator," American Economic Review, American Economic Association, vol. 92(4), pages 1218-1221, September.
    10. Christoph Engel, 2011. "Dictator games: a meta study," Experimental Economics, Springer;Economic Science Association, vol. 14(4), pages 583-610, November.
    11. Simon Gachter & Ernst Fehr, 2000. "Cooperation and Punishment in Public Goods Experiments," American Economic Review, American Economic Association, vol. 90(4), pages 980-994, September.
    12. Volker Benndorf & Claudia Moellers & Hans-Theo Normann, 2017. "Experienced vs. inexperienced participants in the lab: do they behave differently?," Journal of the Economic Science Association, Springer;Economic Science Association, vol. 3(1), pages 12-25, July.
    13. Erik Snowberg & Leeat Yariv, 2021. "Testing the Waters: Behavior across Participant Pools," American Economic Review, American Economic Association, vol. 111(2), pages 687-719, February.
    14. Muriel Niederle & Lise Vesterlund, 2007. "Do Women Shy Away From Competition? Do Men Compete Too Much?," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 122(3), pages 1067-1101.
    15. Nagel, Rosemarie, 1995. "Unraveling in Guessing Games: An Experimental Study," American Economic Review, American Economic Association, vol. 85(5), pages 1313-1326, December.
    16. Benndorf, Volker & Moellers, Claudia & Normann, Hans-Theo, 2017. "Experienced vs. inexperienced participants in the lab: Do they behave differently?," DICE Discussion Papers 251, Heinrich Heine University Düsseldorf, Düsseldorf Institute for Competition Economics (DICE).
    17. Andreoni, James, 1988. "Why free ride? : Strategies and learning in public goods experiments," Journal of Public Economics, Elsevier, vol. 37(3), pages 291-304, December.
    18. Abeler, Johannes & Becker, Anke & Falk, Armin, 2014. "Representative evidence on lying costs," Journal of Public Economics, Elsevier, vol. 113(C), pages 96-104.
    19. Yun Shin Lee & Yong Won Seo & Enno Siemsen, 2018. "Running Behavioral Operations Experiments Using Amazon's Mechanical Turk," Production and Operations Management, Production and Operations Management Society, vol. 27(5), pages 973-989, May.
    20. Hugh-Jones, David, 2016. "Honesty, beliefs about honesty, and economic growth in 15 countries," Journal of Economic Behavior & Organization, Elsevier, vol. 127(C), pages 99-114.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Chapkovski, Philipp & Zakharov, Alexei, 2023. "Does voluntary disclosure of polarizing information make polarization deeper? An online experiment on Russo-Ukrainian War," MPRA Paper 116305, University Library of Munich, Germany.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Chapkovski, Philipp, 2023. "Conducting interactive experiments on Toloka," Journal of Behavioral and Experimental Finance, Elsevier, vol. 37(C).
    2. Chapkovski, Philipp, 2022. "Interactive experiments in Toloka," MPRA Paper 111980, University Library of Munich, Germany.
    3. Benndorf, Volker & Rau, Holger A. & Sölch, Christian, 2019. "Minimizing learning in repeated real-effort tasks," Journal of Behavioral and Experimental Finance, Elsevier, vol. 22(C), pages 239-248.
    4. Thiemann, Petra & Schulz, Jonathan & Sunde, Uwe & Thöni, Christian, 2022. "Selection into experiments: New evidence on the role of preferences, cognition, and recruitment protocols," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 98(C).
    5. Roggenkamp, Hauke C., 2024. "Revisiting ‘Growth and Inequality in Public Good Provision’—Reproducing and Generalizing Through Inconvenient Online Experimentation," OSF Preprints 6rn97, Center for Open Science.
    6. Eugen Dimant & Tobias Gesche, 2021. "Nudging Enforcers: How Norm Perceptions and Motives for Lying Shape Sanctions," CESifo Working Paper Series 9385, CESifo.
    7. Dohmen, Thomas, 2014. "Behavioral labor economics: Advances and future directions," Labour Economics, Elsevier, vol. 30(C), pages 71-85.
    8. Breitmoser, Yves & Vorjohann, Pauline, 2018. "Welfare-Based Altruism," Rationality and Competition Discussion Paper Series 89, CRC TRR 190 Rationality and Competition.
    9. Daniel Jones & Sera Linardi, 2014. "Wallflowers: Experimental Evidence of an Aversion to Standing Out," Management Science, INFORMS, vol. 60(7), pages 1757-1771, July.
    10. Riehm, Tobias & Fugger, Nicolas & Gillen, Philippe & Gretschko, Vitali & Werner, Peter, 2022. "Social norms, sanctions, and conditional entry in markets with externalities: Evidence from an artefactual field experiment," Journal of Public Economics, Elsevier, vol. 212(C).
    11. Cecchi, Francesco & Melesse, Mequanint Biset, 2016. "Formal law and customary change: A lab-in-field experiment in Ethiopia," Journal of Economic Behavior & Organization, Elsevier, vol. 125(C), pages 67-85.
    12. Bryan C. McCannon & Paul Walker, 2020. "Individual Competence and Committee Decision Making: Experimental Evidence," Southern Economic Journal, John Wiley & Sons, vol. 86(4), pages 1531-1558, April.
    13. Erik O. Kimbrough & Alexander Vostroknutov, 2016. "Norms Make Preferences Social," Journal of the European Economic Association, European Economic Association, vol. 14(3), pages 608-638, June.
    14. Dengler-Roscher, Kathrin & Montinari, Natalia & Panganiban, Marian & Ploner, Matteo & Werner, Benedikt, 2018. "On the malleability of fairness ideals: Spillover effects in partial and impartial allocation tasks," Journal of Economic Psychology, Elsevier, vol. 65(C), pages 60-74.
    15. Chaudhuri, Ananish & Paichayontvijit, Tirnud & Smith, Alexander, 2017. "Belief heterogeneity and contributions decay among conditional cooperators in public goods games," Journal of Economic Psychology, Elsevier, vol. 58(C), pages 15-30.
    16. Lilia Zhurakhovska, 2014. "Strategic Trustworthiness via Unstrategic Third-party Reward – An Experiment," Discussion Paper Series of the Max Planck Institute for Research on Collective Goods 2014_06, Max Planck Institute for Research on Collective Goods, revised Jan 2017.
    17. Lorenzo Lotti & Shanali Pethiyagoda, 2022. "Generosity during COVID-19: investigating socioeconomic shocks and game framing," Palgrave Communications, Palgrave Macmillan, vol. 9(1), pages 1-10, December.
    18. Arechar, Antonio A. & Rand, David G., 2022. "Learning to be selfish? A large-scale longitudinal analysis of Dictator games played on Amazon Mechanical Turk," Journal of Economic Psychology, Elsevier, vol. 90(C).
    19. Kovacs, Roxanne & Dunaiski, Maurice & Galizzi, Matteo M. & Grimalda, Gianluca & Hortala-Vallve, Rafael & Murtin, Fabrice & Putterman, Louis, 2024. "The determinants of trust: findings from large, representative samples in six OECD countries," LSE Research Online Documents on Economics 124608, London School of Economics and Political Science, LSE Library.
    20. Amasino, Dianna R. & Pace, Davide Domenico & van der Weele, Joël, 2023. "Self-serving bias in redistribution choices: Accounting for beliefs and norms," Journal of Economic Psychology, Elsevier, vol. 98(C).

    More about this item

    Keywords

    Crowdsourcing; survey research; MTurk; online research;
    All these keywords.

    JEL classification:

    • C90 - Mathematical and Quantitative Methods - - Design of Experiments - - - General
    • C92 - Mathematical and Quantitative Methods - - Design of Experiments - - - Laboratory, Group Behavior
    • C81 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - Methodology for Collecting, Estimating, and Organizing Microeconomic Data; Data Access
    • C88 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - Other Computer Software
    • B41 - Schools of Economic Thought and Methodology - - Economic Methodology - - - Economic Methodology

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:zbw:esprep:249771. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: ZBW - Leibniz Information Centre for Economics (email available below). General contact details of provider: https://edirc.repec.org/data/zbwkide.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.