IDEAS home Printed from https://ideas.repec.org/a/eee/joepsy/v74y2019ics0167487018307153.html
   My bibliography  Save this article

The reliability of questionnaires in laboratory experiments: What can we do?

Author

Listed:
  • Wolff, Irenaeus

Abstract

Questionnaires eliciting personality traits and other characteristics of a person are important tools for many experimental economists. While a lot is known about how to run experiments and about how to construct and run field surveys, much less is known about how to administer such surveys in a post-experimental context. A short survey among experimental economists documents substantial heterogeneity in the procedures used, and in expectations about the effects of procedural details. I run an experiment on five aspects that are specific to the laboratory context. I find that (i) paying participants as soon as they finish the questionnaire yields a lower answer quality than waiting for all or an intermediate procedure; (ii) having participants enter their names for receipt preparation does not reduce answer quality (and does not increase the social desirability of answers); (iii) a higher overall payment increases answer quality, while (iv) framing the fixed part of participants’ payment as being “for completing the questionnaire” as well as (v) progress feedback do not affect answer quality.

Suggested Citation

  • Wolff, Irenaeus, 2019. "The reliability of questionnaires in laboratory experiments: What can we do?," Journal of Economic Psychology, Elsevier, vol. 74(C).
    • Handle: RePEc:eee:joepsy:v:74:y:2019:i:c:s0167487018307153
    DOI: 10.1016/j.joep.2019.102197
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0167487018307153
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.joep.2019.102197?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Anke Becker & Thomas Deckers & Thomas Dohmen & Armin Falk & Fabian Kosse, 2012. "The Relationship Between Economic Preferences and Psychological Personality Measures," Annual Review of Economics, Annual Reviews, vol. 4(1), pages 453-478, July.
    2. Bruttel, Lisa & Fischbacher, Urs, 2013. "Taking the initiative. What characterizes leaders?," European Economic Review, Elsevier, vol. 64(C), pages 147-168.
    3. Urs Fischbacher, 2007. "z-Tree: Zurich toolbox for ready-made economic experiments," Experimental Economics, Springer;Economic Science Association, vol. 10(2), pages 171-178, June.
    4. Urs Fischbacher & Franziska Föllmi-Heusi, 2013. "Lies In Disguise—An Experimental Study On Cheating," Journal of the European Economic Association, European Economic Association, vol. 11(3), pages 525-547, June.
    5. Martin G. Kocher & David Schindler & Stefan T. Trautmann & Yilong Xu, 2019. "Risk, time pressure, and selection effects," Experimental Economics, Springer;Economic Science Association, vol. 22(1), pages 216-246, March.
    6. David K. Levine, 1998. "Modeling Altruism and Spitefulness in Experiment," Review of Economic Dynamics, Elsevier for the Society for Economic Dynamics, vol. 1(3), pages 593-622, July.
    7. Eugenio Proto & Aldo Rustichini & Andis Sofianos, 2019. "Intelligence, Personality, and Gains from Cooperation in Repeated Interactions," Journal of Political Economy, University of Chicago Press, vol. 127(3), pages 1351-1390.
    8. Yaw Nyarko & Andrew Schotter, 2002. "An Experimental Study of Belief Learning Using Elicited Beliefs," Econometrica, Econometric Society, vol. 70(3), pages 971-1005, May.
    9. Rödiger, Stefan & Friedrichsmeier, Thomas & Kapat, Prasenjit & Michalke, Meik, 2012. "RKWard: A Comprehensive Graphical User Interface and Integrated Development Environment for Statistical Analysis with R," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 49(i09).
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Kogler, Christoph & Olsen, Jerome & Bogaers, Rebecca I., 2020. "Enhanced anonymity in tax experiments does not affect compliance," Journal of Economic Behavior & Organization, Elsevier, vol. 177(C), pages 390-398.
    2. Burdea, Valeria & Woon, Jonathan, 2022. "Online belief elicitation methods," Journal of Economic Psychology, Elsevier, vol. 90(C).
    3. Pin, Paolo & Rotesi, Tiziano, 2023. "App-based experiments," Journal of Economic Psychology, Elsevier, vol. 99(C).
    4. Sofianos, Andis, 2022. "Self-reported & revealed trust: Experimental evidence," Journal of Economic Psychology, Elsevier, vol. 88(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Lisa Bruttel & Irenaeus Wolff, 2018. "Incentives and Random Answers in Post-Experimental Questionnaires," TWI Research Paper Series 110, Thurgauer Wirtschaftsinstitut, Universität Konstanz.
    2. Masaki Aoyagi & Guillaume Frechette & Sevgi Yuksel, 2021. "Beliefs in Repeated Games," ISER Discussion Paper 1119rr, Institute of Social and Economic Research, Osaka University, revised May 2022.
    3. Dato, Simon & Feess, Eberhard & Nieken, Petra, 2019. "Lying and reciprocity," Games and Economic Behavior, Elsevier, vol. 118(C), pages 193-218.
    4. Sofianos, Andis, 2022. "Self-reported & revealed trust: Experimental evidence," Journal of Economic Psychology, Elsevier, vol. 88(C).
    5. Robert Stüber, 2020. "The benefit of the doubt: willful ignorance and altruistic punishment," Experimental Economics, Springer;Economic Science Association, vol. 23(3), pages 848-872, September.
    6. Gerald Eisenkopf & Ruslan Gurtoviy & Verena Utikal, 2011. "Size matters - when it comes to lies," IAAEG Discussion Papers until 2011 20110517, Institute of Labour Law and Industrial Relations in the European Union (IAAEU).
    7. Kurt A. Ackermann & Ryan O. Murphy, 2019. "Explaining Cooperative Behavior in Public Goods Games: How Preferences and Beliefs Affect Contribution Levels," Games, MDPI, vol. 10(1), pages 1-34, March.
    8. Peeters, Ronald & Vorsatz, Marc & Walzl, Markus, 2015. "Beliefs and truth-telling: A laboratory experiment," Journal of Economic Behavior & Organization, Elsevier, vol. 113(C), pages 1-12.
    9. Nagore Iriberri & Pedro Rey‐Biel, 2013. "Elicited beliefs and social information in modified dictator games: What do dictators believe other dictators do?," Quantitative Economics, Econometric Society, vol. 4(3), pages 515-547, November.
    10. Bartuli, Jenny & Djawadi, Behnud Mir & Fahr, René, 2016. "Business Ethics in Organizations: An Experimental Examination of Whistleblowing and Personality," IZA Discussion Papers 10190, Institute of Labor Economics (IZA).
    11. Wolff, Irenaeus, 2022. "Predicting Voluntary Contributions by `Revealed-Preference Nash-Equilibrium'," VfS Annual Conference 2022 (Basel): Big Data in Economics 264072, Verein für Socialpolitik / German Economic Association.
    12. Felix Kölle & Simone Quercia & Egon Tripodi, 2023. "Social Preferences under the Shadow of the Future," Rationality and Competition Discussion Paper Series 406, CRC TRR 190 Rationality and Competition.
    13. Thomas Giebe & Radosveta Ivanova-Stenzel & Martin G. Kocher & Simeon Schudy, 2024. "Cross-game learning and cognitive ability in auctions," Experimental Economics, Springer;Economic Science Association, vol. 27(1), pages 80-108, March.
    14. Nick Feltovich, 2019. "The interaction between competition and unethical behaviour," Experimental Economics, Springer;Economic Science Association, vol. 22(1), pages 101-130, March.
    15. Schmutzler, Armin & Holger, Herz & André, Volk, 2014. "Honesty and Relational Contracts," VfS Annual Conference 2014 (Hamburg): Evidence-based Economic Policy 100363, Verein für Socialpolitik / German Economic Association.
    16. Sascha Behnk & Iván Barreda-Tarrazona & Aurora García-Gallego, 2017. "An experimental test of reporting systems for deception," Working Papers 2017/11, Economics Department, Universitat Jaume I, Castellón (Spain).
    17. Behnk, Sascha & Barreda-Tarrazona, Iván & García-Gallego, Aurora, 2019. "Deception and reputation – An experimental test of reporting systems," Journal of Economic Psychology, Elsevier, vol. 71(C), pages 37-58.
    18. Breaban, Adriana & van de Kuilen, Gijs & Noussair, Charles, 2016. "Prudence, Personality, Cognitive Ability and Emotional State," Other publications TiSEM 9a01a5ab-e03d-49eb-9cd7-4, Tilburg University, School of Economics and Management.
    19. Sandra Ludwig & Julia Nafziger, 2011. "Beliefs about overconfidence," Theory and Decision, Springer, vol. 70(4), pages 475-500, April.
    20. Robert S. Gibbons & Manuel Grieder & Holger Herz & Christian Zehnder, 2019. "Building an Equilibrium: Rules Versus Principles in Relational Contracts," CESifo Working Paper Series 7871, CESifo.

    More about this item

    Keywords

    Experimental economics; Methods; Survey; Payment procedures;
    All these keywords.

    JEL classification:

    • C83 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - Survey Methods; Sampling Methods
    • C91 - Mathematical and Quantitative Methods - - Design of Experiments - - - Laboratory, Individual Behavior

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:joepsy:v:74:y:2019:i:c:s0167487018307153. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/joep .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.