IDEAS home Printed from https://ideas.repec.org/p/hal/psewpa/halshs-04893009.html
   My bibliography  Save this paper

How large is "large enough" ? Large-scale experimental investigation of the reliability of confidence measures

Author

Listed:
  • Clémentine Bouleau

    (PSE - Paris School of Economics - UP1 - Université Paris 1 Panthéon-Sorbonne - ENS-PSL - École normale supérieure - Paris - PSL - Université Paris Sciences et Lettres - EHESS - École des hautes études en sciences sociales - ENPC - École nationale des ponts et chaussées - CNRS - Centre National de la Recherche Scientifique - INRAE - Institut National de Recherche pour l’Agriculture, l’Alimentation et l’Environnement, CES - Centre d'économie de la Sorbonne - UP1 - Université Paris 1 Panthéon-Sorbonne - CNRS - Centre National de la Recherche Scientifique)

  • Nicolas Jacquemet

    (PSE - Paris School of Economics - UP1 - Université Paris 1 Panthéon-Sorbonne - ENS-PSL - École normale supérieure - Paris - PSL - Université Paris Sciences et Lettres - EHESS - École des hautes études en sciences sociales - ENPC - École nationale des ponts et chaussées - CNRS - Centre National de la Recherche Scientifique - INRAE - Institut National de Recherche pour l’Agriculture, l’Alimentation et l’Environnement, CES - Centre d'économie de la Sorbonne - UP1 - Université Paris 1 Panthéon-Sorbonne - CNRS - Centre National de la Recherche Scientifique)

  • Maël Lebreton

    (PSE - Paris School of Economics - UP1 - Université Paris 1 Panthéon-Sorbonne - ENS-PSL - École normale supérieure - Paris - PSL - Université Paris Sciences et Lettres - EHESS - École des hautes études en sciences sociales - ENPC - École nationale des ponts et chaussées - CNRS - Centre National de la Recherche Scientifique - INRAE - Institut National de Recherche pour l’Agriculture, l’Alimentation et l’Environnement, PJSE - Paris Jourdan Sciences Economiques - UP1 - Université Paris 1 Panthéon-Sorbonne - ENS-PSL - École normale supérieure - Paris - PSL - Université Paris Sciences et Lettres - EHESS - École des hautes études en sciences sociales - ENPC - École nationale des ponts et chaussées - CNRS - Centre National de la Recherche Scientifique - INRAE - Institut National de Recherche pour l’Agriculture, l’Alimentation et l’Environnement, UNIGE - Université de Genève = University of Geneva)

Abstract

Whether individuals feel confident about their own actions, choices, or statements being correct, and how these confidence levels differ between individuals are two key primitives for countless behavioral theories and phenomena. In cognitive tasks, individual confidence is typically measured as the average of reports about choice accuracy, but how reliable is the resulting characterization of within-and between-individual confidence remains surprisingly undocumented. Here, we perform a large-scale resampling exercise in the Confidence Database to investigate the reliability of individual confidence estimates, and of comparisons across individuals' confidence levels. Our results show that confidence estimates are more stable than their choice-accuracy counterpart, reaching a reliability plateau after roughly 50 trials, regardless of a number of task design characteristics. While constituting a reliability upper-bound for task-based confidence measures, and thereby leaving open the question of the reliability of the construct itself, these results characterize the robustness of past and future task designs.

Suggested Citation

  • Clémentine Bouleau & Nicolas Jacquemet & Maël Lebreton, 2025. "How large is "large enough" ? Large-scale experimental investigation of the reliability of confidence measures," PSE Working Papers halshs-04893009, HAL.
  • Handle: RePEc:hal:psewpa:halshs-04893009
    Note: View the original document on HAL open archive server: https://shs.hal.science/halshs-04893009v1
    as

    Download full text from publisher

    File URL: https://shs.hal.science/halshs-04893009v1/document
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Markus M. Möbius & Muriel Niederle & Paul Niehaus & Tanya S. Rosenblat, 2022. "Managing Self-Confidence: Theory and Experimental Evidence," Management Science, INFORMS, vol. 68(11), pages 7793-7817, November.
    2. Muriel Niederle & Lise Vesterlund, 2007. "Do Women Shy Away From Competition? Do Men Compete Too Much?," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 122(3), pages 1067-1101.
    3. Joaquin Navajas & Chandni Hindocha & Hebah Foda & Mehdi Keramati & Peter E. Latham & Bahador Bahrami, 2017. "The idiosyncratic nature of confidence," Nature Human Behaviour, Nature, vol. 1(11), pages 810-818, November.
    4. Koellinger, Philipp & Minniti, Maria & Schade, Christian, 2007. ""I think I can, I think I can": Overconfidence and entrepreneurial behavior," Journal of Economic Psychology, Elsevier, vol. 28(4), pages 502-527, August.
    5. Maël Lebreton & Sophie Bavard & Jean Daunizeau & Stefano Palminteri, 2019. "Assessing inter-individual differences with task-related functional neuroimaging," Nature Human Behaviour, Nature, vol. 3(9), pages 897-905, September.
    6. Pietro Ortoleva & Erik Snowberg, 2015. "Overconfidence in Political Behavior," American Economic Review, American Economic Association, vol. 105(2), pages 504-535, February.
    7. Zoe M. Boundy-Singer & Corey M. Ziemba & Robbe L. T. Goris, 2023. "Confidence reflects a noisy decision reliability estimate," Nature Human Behaviour, Nature, vol. 7(1), pages 142-154, January.
    8. Roland Bénabou & Jean Tirole, 2005. "Self-Confidence and Personal Motivation," International Economic Association Series, in: Bina Agarwal & Alessandro Vercelli (ed.), Psychology, Rationality and Economic Behaviour, chapter 2, pages 19-57, Palgrave Macmillan.
    9. Karl Schlag & James Tremewan & Joël Weele, 2015. "A penny for your thoughts: a survey of methods for eliciting beliefs," Experimental Economics, Springer;Economic Science Association, vol. 18(3), pages 457-490, September.
    10. Kobe Desender & Luc Vermeylen & Tom Verguts, 2022. "Dynamic influences on static measures of metacognition," Nature Communications, Nature, vol. 13(1), pages 1-12, December.
    11. Smith, Vernon L & Walker, James M, 1993. "Monetary Rewards and Decision Cost in Experimental Economics," Economic Inquiry, Western Economic Association International, vol. 31(2), pages 245-261, April.
    12. Roey Schurr & Daniel Reznik & Hanna Hillman & Rahul Bhui & Samuel J. Gershman, 2024. "Dynamic computational phenotyping of human cognition," Nature Human Behaviour, Nature, vol. 8(5), pages 917-931, May.
    13. Florian Zimmermann, 2020. "The Dynamics of Motivated Beliefs," American Economic Review, American Economic Association, vol. 110(2), pages 337-361, February.
    14. Matthew J C Crump & John V McDonnell & Todd M Gureckis, 2013. "Evaluating Amazon's Mechanical Turk as a Tool for Experimental Behavioral Research," PLOS ONE, Public Library of Science, vol. 8(3), pages 1-18, March.
    15. Jose A. Scheinkman & Wei Xiong, 2003. "Overconfidence and Speculative Bubbles," Journal of Political Economy, University of Chicago Press, vol. 111(6), pages 1183-1219, December.
    16. Andreas Pedroni & Renato Frey & Adrian Bruhin & Gilles Dutilh & Ralph Hertwig & Jörg Rieskamp, 2017. "The risk elicitation puzzle," Nature Human Behaviour, Nature, vol. 1(11), pages 803-809, November.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Clémentine Bouleau & Nicolas Jacquemet & Maël Lebreton, 2025. "How large is "large enough" ? Large-scale experimental investigation of the reliability of confidence measures," Université Paris1 Panthéon-Sorbonne (Post-Print and Working Papers) halshs-04893009, HAL.
    2. Danz, David, 2020. "Never underestimate your opponent: Hindsight bias causes overplacement and overentry into competition," Games and Economic Behavior, Elsevier, vol. 124(C), pages 588-603.
    3. Gangadharan, Lata & Grossman, Philip J. & Xue, Nina, 2024. "Belief elicitation under competing motivations: Does it matter how you ask?," European Economic Review, Elsevier, vol. 169(C).
    4. Pedro Gonzalez-Fernandez, 2024. "Belief Bias Identification," Papers 2404.09297, arXiv.org, revised Nov 2024.
    5. Peter Schwardmann & Joël van der Weele, 2016. "Deception and Self-Deception," Tinbergen Institute Discussion Papers 16-012/I, Tinbergen Institute.
    6. Bobba, Matteo & Frisancho, Veronica, 2022. "Self-perceptions about academic achievement: Evidence from Mexico City," Journal of Econometrics, Elsevier, vol. 231(1), pages 58-73.
    7. Jonathan F Schulz & Christian Thöni, 2016. "Overconfidence and Career Choice," PLOS ONE, Public Library of Science, vol. 11(1), pages 1-8, January.
    8. Banerjee, Ritwik & Gupta, Nabanita Datta & Villeval, Marie Claire, 2020. "Feedback spillovers across tasks, self-confidence and competitiveness," Games and Economic Behavior, Elsevier, vol. 123(C), pages 127-170.
    9. Burro, Giovanni & Castagnetti, Alessandro, 2024. "The ego is no fool: Absence of motivated belief formation in strategic interactions," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 109(C).
    10. Zeinab Aboutalebi & Ayush Pant, 2021. "Believe ... and you are there. On Self-Confidence and Feedback," Working Papers 64, Ashoka University, Department of Economics.
    11. Falk, Armin & Kosse, Fabian & Schildberg-Hörisch, Hannah & Zimmermann, Florian, 2023. "Self-assessment: The role of the social environment," Journal of Public Economics, Elsevier, vol. 223(C).
    12. Cheng, Ing-Haw & Hsiaw, Alice, 2022. "Distrust in experts and the origins of disagreement," Journal of Economic Theory, Elsevier, vol. 200(C).
    13. Cheung, Stephen L. & Johnstone, Lachlan, 2017. "True Overconfidence, Revealed through Actions: An Experiment," IZA Discussion Papers 10545, Institute of Labor Economics (IZA).
    14. Benjamin Elsner & Ingo E Isphording & Ulf Zölitz, 2021. "Achievement Rank Affects Performance and Major Choices in College [Gender, competitiveness, and socialization at a young age: evidence from a matrilineal and a patriarchal society]," The Economic Journal, Royal Economic Society, vol. 131(640), pages 3182-3206.
    15. Fischer, Mira & Sliwka, Dirk, 2018. "Confidence in knowledge or confidence in the ability to learn: An experiment on the causal effects of beliefs on motivation," Games and Economic Behavior, Elsevier, vol. 111(C), pages 122-142.
    16. Dasgupta, Utteeyo & Gangadharan, Lata & Maitra, Pushkar & Mani, Subha & Subramanian, Samyukta, 2015. "Choosing to be trained: Do behavioral traits matter?," Journal of Economic Behavior & Organization, Elsevier, vol. 110(C), pages 145-159.
    17. Thomas M. Eisenbach & Martin C. Schmalz, 2015. "Anxiety and pro-cyclical risk taking with Bayesian agents," Staff Reports 711, Federal Reserve Bank of New York.
    18. Markus M. Möbius & Muriel Niederle & Paul Niehaus & Tanya S. Rosenblat, 2022. "Managing Self-Confidence: Theory and Experimental Evidence," Management Science, INFORMS, vol. 68(11), pages 7793-7817, November.
    19. Adrian Bruhin & Fidel Petros & Luís Santos-Pinto, 2024. "The role of self-confidence in teamwork: experimental evidence," Experimental Economics, Springer;Economic Science Association, vol. 27(3), pages 687-712, July.
    20. Fellner-Röhling, Gerlinde & Hromek, Kristijan & Kleinknecht, Janina & Ludwig, Sandra, 2023. "How to counteract biased self-assessments? An experimental investigation of reactions to social information," Journal of Economic Behavior & Organization, Elsevier, vol. 206(C), pages 1-25.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:hal:psewpa:halshs-04893009. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: CCSD (email available below). General contact details of provider: https://hal.archives-ouvertes.fr/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.