IDEAS home Printed from https://ideas.repec.org/a/eee/joepsy/v74y2019ics0167487018307153.html
   My bibliography  Save this article

The reliability of questionnaires in laboratory experiments: What can we do?

Author

Listed:
  • Wolff, Irenaeus

Abstract

Questionnaires eliciting personality traits and other characteristics of a person are important tools for many experimental economists. While a lot is known about how to run experiments and about how to construct and run field surveys, much less is known about how to administer such surveys in a post-experimental context. A short survey among experimental economists documents substantial heterogeneity in the procedures used, and in expectations about the effects of procedural details. I run an experiment on five aspects that are specific to the laboratory context. I find that (i) paying participants as soon as they finish the questionnaire yields a lower answer quality than waiting for all or an intermediate procedure; (ii) having participants enter their names for receipt preparation does not reduce answer quality (and does not increase the social desirability of answers); (iii) a higher overall payment increases answer quality, while (iv) framing the fixed part of participants’ payment as being “for completing the questionnaire” as well as (v) progress feedback do not affect answer quality.

Suggested Citation

  • Wolff, Irenaeus, 2019. "The reliability of questionnaires in laboratory experiments: What can we do?," Journal of Economic Psychology, Elsevier, vol. 74(C).
    • Handle: RePEc:eee:joepsy:v:74:y:2019:i:c:s0167487018307153
    DOI: 10.1016/j.joep.2019.102197
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0167487018307153
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.joep.2019.102197?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Urs Fischbacher & Franziska Föllmi-Heusi, 2013. "Lies In Disguise—An Experimental Study On Cheating," Journal of the European Economic Association, European Economic Association, vol. 11(3), pages 525-547, June.
    2. Anke Becker & Thomas Deckers & Thomas Dohmen & Armin Falk & Fabian Kosse, 2012. "The Relationship Between Economic Preferences and Psychological Personality Measures," Annual Review of Economics, Annual Reviews, vol. 4(1), pages 453-478, July.
    3. David K. Levine, 1998. "Modeling Altruism and Spitefulness in Experiment," Review of Economic Dynamics, Elsevier for the Society for Economic Dynamics, vol. 1(3), pages 593-622, July.
    4. Yaw Nyarko & Andrew Schotter, 2002. "An Experimental Study of Belief Learning Using Elicited Beliefs," Econometrica, Econometric Society, vol. 70(3), pages 971-1005, May.
    5. Anke Becker & Thomas Deckers & Thomas Dohmen & Armin Falk & Fabian Kosse, 2012. "The Relationship Between Economic Preferences and Psychological Personality Measures," Annual Review of Economics, Annual Reviews, vol. 4(1), pages 453-478, July.
    6. Martin G. Kocher & David Schindler & Stefan T. Trautmann & Yilong Xu, 2019. "Risk, time pressure, and selection effects," Experimental Economics, Springer;Economic Science Association, vol. 22(1), pages 216-246, March.
    7. Rödiger, Stefan & Friedrichsmeier, Thomas & Kapat, Prasenjit & Michalke, Meik, 2012. "RKWard: A Comprehensive Graphical User Interface and Integrated Development Environment for Statistical Analysis with R," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 49(i09).
    8. Bruttel, Lisa & Fischbacher, Urs, 2013. "Taking the initiative. What characterizes leaders?," European Economic Review, Elsevier, vol. 64(C), pages 147-168.
    9. Eugenio Proto & Aldo Rustichini & Andis Sofianos, 2019. "Intelligence, Personality, and Gains from Cooperation in Repeated Interactions," Journal of Political Economy, University of Chicago Press, vol. 127(3), pages 1351-1390.
    10. Urs Fischbacher, 2007. "z-Tree: Zurich toolbox for ready-made economic experiments," Experimental Economics, Springer;Economic Science Association, vol. 10(2), pages 171-178, June.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Pin, Paolo & Rotesi, Tiziano, 2023. "App-based experiments," Journal of Economic Psychology, Elsevier, vol. 99(C).
    2. Burdea, Valeria & Woon, Jonathan, 2022. "Online belief elicitation methods," Journal of Economic Psychology, Elsevier, vol. 90(C).
    3. Sofianos, Andis, 2022. "Self-reported & revealed trust: Experimental evidence," Journal of Economic Psychology, Elsevier, vol. 88(C).
    4. Kogler, Christoph & Olsen, Jerome & Bogaers, Rebecca I., 2020. "Enhanced anonymity in tax experiments does not affect compliance," Journal of Economic Behavior & Organization, Elsevier, vol. 177(C), pages 390-398.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Sofianos, Andis, 2022. "Self-reported & revealed trust: Experimental evidence," Journal of Economic Psychology, Elsevier, vol. 88(C).
    2. Lisa Bruttel & Irenaeus Wolff, 2018. "Incentives and Random Answers in Post-Experimental Questionnaires," TWI Research Paper Series 110, Thurgauer Wirtschaftsinstitut, Universität Konstanz.
    3. Felix Kölle & Simone Quercia & Egon Tripodi, 2023. "Social Preferences under the Shadow of the Future," Rationality and Competition Discussion Paper Series 406, CRC TRR 190 Rationality and Competition.
    4. Schmutzler, Armin & Holger, Herz & André, Volk, 2014. "Honesty and Relational Contracts," VfS Annual Conference 2014 (Hamburg): Evidence-based Economic Policy 100363, Verein für Socialpolitik / German Economic Association.
    5. Breaban, Adriana & van de Kuilen, Gijs & Noussair, Charles, 2016. "Prudence, Personality, Cognitive Ability and Emotional State," Other publications TiSEM 9a01a5ab-e03d-49eb-9cd7-4, Tilburg University, School of Economics and Management.
    6. Masaki Aoyagi & Guillaume Frechette & Sevgi Yuksel, 2021. "Beliefs in Repeated Games," ISER Discussion Paper 1119rr, Institute of Social and Economic Research, Osaka University, revised May 2022.
    7. Drouvelis, Michalis & Georgantzis, Nikolaos, 2019. "Does revealing personality data affect prosocial behaviour?," Journal of Economic Behavior & Organization, Elsevier, vol. 159(C), pages 409-420.
    8. Dato, Simon & Feess, Eberhard & Nieken, Petra, 2019. "Lying and reciprocity," Games and Economic Behavior, Elsevier, vol. 118(C), pages 193-218.
    9. Kagel, John & McGee, Peter, 2014. "Personality and cooperation in finitely repeated prisoner’s dilemma games," Economics Letters, Elsevier, vol. 124(2), pages 274-277.
    10. Gerhards, Leonie & Gravert, Christina, 2015. "Grit Trumps Talent? An experimental approach," Working Papers in Economics 629, University of Gothenburg, Department of Economics.
    11. Vranka, Marek Albert & Bahník, Štěpán, 2017. "Predictors of Bribe-Taking: The Role of Bribe Size and Personality," OSF Preprints mzhkq, Center for Open Science.
    12. repec:zbw:rwirep:0488 is not listed on IDEAS
    13. Bergheim, Ralf, 2014. "Information Acquisition and Decisions under Risk and Ambiguity," Ruhr Economic Papers 488, RWI - Leibniz-Institut für Wirtschaftsforschung, Ruhr-University Bochum, TU Dortmund University, University of Duisburg-Essen.
    14. Tremewan, James & Vanberg, Christoph, 2016. "The dynamics of coalition formation – A multilateral bargaining experiment with free timing of moves," Journal of Economic Behavior & Organization, Elsevier, vol. 130(C), pages 33-46.
    15. Robert Stüber, 2020. "The benefit of the doubt: willful ignorance and altruistic punishment," Experimental Economics, Springer;Economic Science Association, vol. 23(3), pages 848-872, September.
    16. Gerald Eisenkopf & Ruslan Gurtoviy & Verena Utikal, 2011. "Size matters - when it comes to lies," IAAEG Discussion Papers until 2011 20110517, Institute of Labour Law and Industrial Relations in the European Union (IAAEU).
    17. Friehe, Tim & Schildberg-Hörisch, Hannah, 2014. "The Individual and Joint Performance of Economic Preferences, Personality, and Self-Control in Predicting Criminal Behavior," IZA Discussion Papers 7894, Institute of Labor Economics (IZA).
    18. Kurt A. Ackermann & Ryan O. Murphy, 2019. "Explaining Cooperative Behavior in Public Goods Games: How Preferences and Beliefs Affect Contribution Levels," Games, MDPI, vol. 10(1), pages 1-34, March.
    19. Francisca Jiménez-Jiménez, 2023. "Heterogeneity, coordination and competition: the distribution of individual preferences in organisations," Economics of Governance, Springer, vol. 24(1), pages 67-107, March.
    20. Peeters, Ronald & Vorsatz, Marc & Walzl, Markus, 2015. "Beliefs and truth-telling: A laboratory experiment," Journal of Economic Behavior & Organization, Elsevier, vol. 113(C), pages 1-12.
    21. Jana Cahlíková & Lubomír Cingl, 2017. "Risk preferences under acute stress," Experimental Economics, Springer;Economic Science Association, vol. 20(1), pages 209-236, March.

    More about this item

    Keywords

    Experimental economics; Methods; Survey; Payment procedures;
    All these keywords.

    JEL classification:

    • C83 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - Survey Methods; Sampling Methods
    • C91 - Mathematical and Quantitative Methods - - Design of Experiments - - - Laboratory, Individual Behavior

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:joepsy:v:74:y:2019:i:c:s0167487018307153. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/joep .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.