IDEAS home Printed from https://ideas.repec.org/p/ces/ceswps/_11703.html
   My bibliography  Save this paper

Reassessing Qualitative Self-Assessments and Experimental Validation

Author

Listed:
  • Jonathan Chapman
  • Pietro Ortoleva
  • Erik Snowberg
  • Leeat Yariv
  • Colin Camerer

Abstract

Qualitative self-assessments of economic preferences have recently gained popularity, often supported by experimental validation, a method that links them to choices in incentivized elicitations. We illustrate theoretically that experimental validation may fail to produce reliable new measures. Empirically, analyzing data from over 13,000 participants across diverse samples, we document four key findings. First, qualitative self-assessments and traditional incentivized measures exhibit weak correlations, even when accounting for response noise. Second, qualitative self-assessments sometimes correlate more strongly with theoretically distinct incentivized elicitations than those for which they are intended to proxy. Third, relationships between qualitative self-assessments and various attributes—including geographical location, demographics, and behaviors—are unrelated to variation in incentivized elicitations. Fourth, qualitative self-assessments are no simpler for participants than incentivized elicitations: these questions show a common heuristic of extreme or midpoint responses, especially by individuals with lower cognitive ability.

Suggested Citation

  • Jonathan Chapman & Pietro Ortoleva & Erik Snowberg & Leeat Yariv & Colin Camerer, 2025. "Reassessing Qualitative Self-Assessments and Experimental Validation," CESifo Working Paper Series 11703, CESifo.
  • Handle: RePEc:ces:ceswps:_11703
    as

    Download full text from publisher

    File URL: https://www.cesifo.org/DocDL/cesifo1_wp11703.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Armin Falk & Anke Becker & Thomas Dohmen & David Huffman & Uwe Sunde, 2023. "The Preference Survey Module: A Validated Instrument for Measuring Risk, Time, and Social Preferences," Management Science, INFORMS, vol. 69(4), pages 1935-1950, April.
    2. Gary Charness & Thomas Garcia & Theo Offerman & Marie Claire Villeval, 2020. "Do measures of risk attitude in the laboratory predict behavior under risk in and outside of the laboratory?," Journal of Risk and Uncertainty, Springer, vol. 60(2), pages 99-123, April.
    3. Pablo Brañas-Garza & Diego Jorrat & Antonio M. Espín & Angel Sánchez, 2023. "Paid and hypothetical time preferences are the same: lab, field and online evidence," Experimental Economics, Springer;Economic Science Association, vol. 26(2), pages 412-434, April.
    4. Daniel J. Benjamin & Kristen Cooper & Ori Heffetz & Miles S. Kimball & Jiannan Zhou, 2023. "Adjusting for Scale-Use Heterogeneity in Self-Reported Well-Being," NBER Working Papers 31728, National Bureau of Economic Research, Inc.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Kıbrıs, Arzu & Cesur, Resul & Uler, Neslihan & Yıldırım, Sadullah, 2025. "Identifying the Impact of Exposure to Armed Conflict on Individual Preferences and Field Behavior : Evidence from Turkish Draft Veterans," QAPEC Discussion Papers 27, Quantitative and Analytical Political Economy Research Centre.
    2. Carvajal, Daniel & Franco, Catalina & Isaksson, Siri, 2024. "Will Artificial Intelligence Get in the Way of Achieving Gender Equality?," Discussion Paper Series in Economics 3/2024, Norwegian School of Economics, Department of Economics, revised 20 Mar 2025.
    3. Arslan, Ruben C. & Brümmer, Martin & Dohmen, Thomas & Drewelies, Johanna & Hertwig, Ralph & Wagner, Gert G., 2020. "How people know their risk preference," EconStor Open Access Articles and Book Chapters, ZBW - Leibniz Information Centre for Economics, vol. 10.
    4. Rainer Kotschy & Uwe Sunde & Rainer Franz Kotschy, 2023. "Have Preferences Become More Similar Worldwide?," CESifo Working Paper Series 10735, CESifo.
    5. Andrea Hackethal & Michael Kirchler & Christine Laudenbach & Michael Razen & Annika Weber, 2020. "On the role of monetary incentives in risk preference elicitation experiments," Working Papers 2020-29, Faculty of Economics and Statistics, Universität Innsbruck.
    6. Marc Oliver Rieger & Mei Wang & Thorsten Hens, 2021. "Universal time preference," PLOS ONE, Public Library of Science, vol. 16(2), pages 1-15, February.
    7. Finley, Brian & Kalwij, Adriaan & Kapteyn, Arie, 2022. "Born to be wild: Second-to-fourth digit length ratio and risk preferences," Economics & Human Biology, Elsevier, vol. 47(C).
    8. Andreas Hackethal & Michael Kirchler & Christine Laudenbach & Michael Razen & Annika Weber, 2023. "On the role of monetary incentives in risk preference elicitation experiments," Journal of Risk and Uncertainty, Springer, vol. 66(2), pages 189-213, April.
    9. Lisa Josefin Norrgren, 2023. "The highly educated live longer: The role of time preference, cognitive ability, and educational plans," Health Economics, John Wiley & Sons, Ltd., vol. 32(8), pages 1767-1784, August.
    10. Rafaï, Ismaël & Blayac, Thierry & Dubois, Dimitri & Duchêne, Sébastien & Nguyen-Van, Phu & Ventelou, Bruno & Willinger, Marc, 2023. "Stated preferences outperform elicited preferences for predicting reported compliance with COVID-19 prophylactic measures," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 107(C).
    11. Hackethal, Andreas & Kirchler, Michael & Laudenbach, Christine & Razen, Michael & Weber, Annika, 2021. "On the role of monetary incentives in risk preference elicitation experiments," SAFE Working Paper Series 286, Leibniz Institute for Financial Research SAFE, revised 2021.
    12. Kalwij, Adriaan, 2023. "Risk preferences, preventive behaviour, and the probability of a loss: Empirical evidence from the COVID-19 pandemic," Social Science & Medicine, Elsevier, vol. 334(C).
    13. Yohei Mitani & Nobuyuki Hanaki, 2025. "Pay a lot to a few instead of a bit to all! Evidence from online donation experiments," ISER Discussion Paper 1273, Institute of Social and Economic Research, The University of Osaka.
    14. Daniel Horn & Hubert Kiss Janos & Sára Khayouti, 2020. "Does trust associate with political regime?," CERS-IE WORKING PAPERS 2013, Institute of Economics, Centre for Economic and Regional Studies.
    15. Falk, Armin & Abeler, Johannes & Kosse, Fabian, 2021. "Malleability of preferences for honesty," CEPR Discussion Papers 16164, C.E.P.R. Discussion Papers.
    16. Collewet, Marion & Fairley, Kim & Kessels, Roselinde & Knoef, Marike & van Vliet, Olaf, 2024. "The design of welfare: unraveling taxpayers' preferences," OSF Preprints 4am7e, Center for Open Science.
    17. Goytom Abraha Kahsay & Daniel Osberghaus, 2018. "Storm Damage and Risk Preferences: Panel Evidence from Germany," Environmental & Resource Economics, Springer;European Association of Environmental and Resource Economists, vol. 71(1), pages 301-318, September.
    18. Uwe Sunde & Thomas Dohmen & Benjamin Enke & Armin Falkbriq & David Huffman & Gerrit Meyerheim, 2022. "Patience and Comparative Development," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 89(5), pages 2806-2840.
    19. Fabian Kosse & Thomas Deckers & Pia Pinger & Hannah Schildberg-Hörisch & Armin Falk, 2020. "The Formation of Prosociality: Causal Evidence on the Role of Social Environment," Journal of Political Economy, University of Chicago Press, vol. 128(2), pages 434-467.
    20. Bartels, Lara & Kesternich, Martin, 2022. "Motivate the crowd or crowd- them out? The impact of local government spending on the voluntary provision of a green public good," ZEW Discussion Papers 22-040, ZEW - Leibniz Centre for European Economic Research.

    More about this item

    Keywords

    econographics; self-assessments; risk preferences; time preferences; social preferences; preference elicitation; experimental validation;
    All these keywords.

    JEL classification:

    • C90 - Mathematical and Quantitative Methods - - Design of Experiments - - - General
    • C91 - Mathematical and Quantitative Methods - - Design of Experiments - - - Laboratory, Individual Behavior
    • C93 - Mathematical and Quantitative Methods - - Design of Experiments - - - Field Experiments
    • D90 - Microeconomics - - Micro-Based Behavioral Economics - - - General

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:ces:ceswps:_11703. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Klaus Wohlrabe (email available below). General contact details of provider: https://edirc.repec.org/data/cesifde.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.