IDEAS home Printed from https://ideas.repec.org/a/nat/natcom/v14y2023i1d10.1038_s41467-023-42589-5.html
   My bibliography  Save this article

Neural and computational underpinnings of biased confidence in human reinforcement learning

Author

Listed:
  • Chih-Chung Ting

    (Universität Hamburg
    Universiteit van Amsterdam)

  • Nahuel Salem-Garcia

    (University of Geneva)

  • Stefano Palminteri

    (PSL Research University
    Laboratoire de Neurosciences Cognitives et Computationnelles, Institut National de la Santé et de la Recherche Médicale)

  • Jan B. Engelmann

    (Universiteit van Amsterdam
    The Tinbergen Institute)

  • Maël Lebreton

    (University of Geneva
    Economics of Human Behavior group, Paris-Jourdan Sciences Économiques UMR8545, Paris School of Economics)

Abstract

While navigating a fundamentally uncertain world, humans and animals constantly evaluate the probability of their decisions, actions or statements being correct. When explicitly elicited, these confidence estimates typically correlates positively with neural activity in a ventromedial-prefrontal (VMPFC) network and negatively in a dorsolateral and dorsomedial prefrontal network. Here, combining fMRI with a reinforcement-learning paradigm, we leverage the fact that humans are more confident in their choices when seeking gains than avoiding losses to reveal a functional dissociation: whereas the dorsal prefrontal network correlates negatively with a condition-specific confidence signal, the VMPFC network positively encodes task-wide confidence signal incorporating the valence-induced bias. Challenging dominant neuro-computational models, we found that decision-related VMPFC activity better correlates with confidence than with option-values inferred from reinforcement-learning models. Altogether, these results identify the VMPFC as a key node in the neuro-computational architecture that builds global feeling-of-confidence signals from latent decision variables and contextual biases during reinforcement-learning.

Suggested Citation

  • Chih-Chung Ting & Nahuel Salem-Garcia & Stefano Palminteri & Jan B. Engelmann & Maël Lebreton, 2023. "Neural and computational underpinnings of biased confidence in human reinforcement learning," Nature Communications, Nature, vol. 14(1), pages 1-18, December.
  • Handle: RePEc:nat:natcom:v:14:y:2023:i:1:d:10.1038_s41467-023-42589-5
    DOI: 10.1038/s41467-023-42589-5
    as

    Download full text from publisher

    File URL: https://www.nature.com/articles/s41467-023-42589-5
    File Function: Abstract
    Download Restriction: no

    File URL: https://libkey.io/10.1038/s41467-023-42589-5?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Marine Hainguerlot & Jean-Christophe Vergnaud & Vincent de Gardelle, 2018. "Metacognitive ability predicts learning cue-stimulus associations in the absence of external feedback," Université Paris1 Panthéon-Sorbonne (Post-Print and Working Papers) hal-01761531, HAL.
    2. Karl Schlag & James Tremewan & Joël Weele, 2015. "A penny for your thoughts: a survey of methods for eliciting beliefs," Experimental Economics, Springer;Economic Science Association, vol. 18(3), pages 457-490, September.
    3. Stefano Palminteri & Mehdi Khamassi & Mateus Joffily & Giorgio Coricelli, 2015. "Contextual modulation of value signals in reward and punishment learning," Nature Communications, Nature, vol. 6(1), pages 1-14, November.
    4. Sophie Bavard & Maël Lebreton & Mehdi Khamassi & Giorgio Coricelli & Stefano Palminteri, 2018. "Reference-point centering and range-adaptation enhance human reinforcement learning at the cost of irrational preferences," Nature Communications, Nature, vol. 9(1), pages 1-12, December.
    5. Guillaume Hollard & Sébastien Massoni & Jean-Christophe Vergnaud, 2016. "In search of good probability assessors: an experimental comparison of elicitation rules for confidence judgments," Theory and Decision, Springer, vol. 80(3), pages 363-387, March.
    6. Guillaume Hollard & Sébastien Massoni & Jean-Christophe Vergnaud, 2016. "In search of good probability assessors: an experimental comparison of elicitation rules for confidence judgments," Université Paris1 Panthéon-Sorbonne (Post-Print and Working Papers) hal-01306258, HAL.
    7. Maël Lebreton & Sophie Bavard & Jean Daunizeau & Stefano Palminteri, 2019. "Assessing inter-individual differences with task-related functional neuroimaging," Nature Human Behaviour, Nature, vol. 3(9), pages 897-905, September.
    8. Germain Lefebvre & Maël Lebreton & Florent Meyniel & Sacha Bourgeois-Gironde & Stefano Palminteri, 2017. "Behavioural and neural characterization of optimistic reinforcement learning," Nature Human Behaviour, Nature, vol. 1(4), pages 1-9, April.
    9. Jean Daunizeau & Vincent Adam & Lionel Rigoux, 2014. "VBA: A Probabilistic Treatment of Nonlinear Models for Neurobiological and Behavioural Data," PLOS Computational Biology, Public Library of Science, vol. 10(1), pages 1-16, January.
    10. Nadescha Trudel & Jacqueline Scholl & Miriam C. Klein-Flügge & Elsa Fouragnan & Lev Tankelevitch & Marco K. Wittmann & Matthew F. S. Rushworth, 2021. "Polarity of uncertainty representation during exploration and exploitation in ventromedial prefrontal cortex," Nature Human Behaviour, Nature, vol. 5(1), pages 83-98, January.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Maël Lebreton & Karin Bacily & Stefano Palminteri & Jan B Engelmann, 2019. "Contextual influence on confidence judgments in human reinforcement learning," PLOS Computational Biology, Public Library of Science, vol. 15(4), pages 1-27, April.
    2. Juan Dubra & Jean-Pierre Benoît & Giorgia Romagnoli, 2019. "Belief elicitation when more than money matters," Documentos de Trabajo/Working Papers 1901, Facultad de Ciencias Empresariales y Economia. Universidad de Montevideo..
    3. Jean-Pierre Benoît & Juan Dubra & Giorgia Romagnoli, 2022. "Belief Elicitation When More than Money Matters: Controlling for "Control"," American Economic Journal: Microeconomics, American Economic Association, vol. 14(3), pages 837-888, August.
    4. Dominik Bauer & Irenaeus Wolff, 2018. "Biases in Beliefs: Experimental Evidence," TWI Research Paper Series 109, Thurgauer Wirtschaftsinstitut, Universität Konstanz.
    5. Rafkin, Charlie & Shreekumar, Advik & Vautrey, Pierre-Luc, 2021. "When guidance changes: Government stances and public beliefs," Journal of Public Economics, Elsevier, vol. 196(C).
    6. Folli, Dominik & Wolff, Irenaeus, 2022. "Biases in belief reports," Journal of Economic Psychology, Elsevier, vol. 88(C).
    7. Bauer, Dominik & Wolff, Irenaeus, 2019. "Biases in Beliefs," VfS Annual Conference 2019 (Leipzig): 30 Years after the Fall of the Berlin Wall - Democracy and Market Economy 203601, Verein für Socialpolitik / German Economic Association.
    8. Stefano Palminteri & Germain Lefebvre & Emma J Kilford & Sarah-Jayne Blakemore, 2017. "Confirmation bias in human reinforcement learning: Evidence from counterfactual feedback processing," PLOS Computational Biology, Public Library of Science, vol. 13(8), pages 1-22, August.
    9. Jan B. Engelmann & Maël Lebreton & Nahuel A. Salem-Garcia & Peter Schwardmann & Joël J. van der Weele, 2024. "Anticipatory Anxiety and Wishful Thinking," American Economic Review, American Economic Association, vol. 114(4), pages 926-960, April.
    10. Johann Lussange & Stefano Vrizzi & Stefano Palminteri & Boris Gutkin, 2024. "Modelling crypto markets by multi-agent reinforcement learning," Papers 2402.10803, arXiv.org.
    11. Antoine Collomb-Clerc & Maëlle C. M. Gueguen & Lorella Minotti & Philippe Kahane & Vincent Navarro & Fabrice Bartolomei & Romain Carron & Jean Regis & Stephan Chabardès & Stefano Palminteri & Julien B, 2023. "Human thalamic low-frequency oscillations correlate with expected value and outcomes during reinforcement learning," Nature Communications, Nature, vol. 14(1), pages 1-10, December.
    12. Johann Lussange & Stefano Vrizzi & Sacha Bourgeois-Gironde & Stefano Palminteri & Boris Gutkin, 2023. "Stock Price Formation: Precepts from a Multi-Agent Reinforcement Learning Model," Computational Economics, Springer;Society for Computational Economics, vol. 61(4), pages 1523-1544, April.
    13. Gneezy, Uri & Saccardo, Silvia & Serra-Garcia, Marta & van Veldhuizen, Roel, 2020. "Bribing the Self," EconStor Open Access Articles and Book Chapters, ZBW - Leibniz Information Centre for Economics, vol. 120, pages 311-324.
    14. Johann Lussange & Ivan Lazarevich & Sacha Bourgeois-Gironde & Stefano Palminteri & Boris Gutkin, 2021. "Modelling Stock Markets by Multi-agent Reinforcement Learning," Computational Economics, Springer;Society for Computational Economics, vol. 57(1), pages 113-147, January.
    15. Ambroise Descamps & Changxia Ke & Lionel Page, 2022. "How success breeds success," Quantitative Economics, Econometric Society, vol. 13(1), pages 355-385, January.
    16. Thomas Garcia & Sébastien Massoni, 2017. "Aiming to choose correctly or to choose wisely ? The optimality-accuracy trade-off in decisions under uncertainty," Working Papers 1714, Groupe d'Analyse et de Théorie Economique Lyon St-Étienne (GATE Lyon St-Étienne), Université de Lyon.
    17. repec:cup:judgdm:v:16:y:2021:i:4:p:932-949 is not listed on IDEAS
    18. Dorin, Camille & Hainguerlot, Marine & Huber-Yahi, Hélène & Vergnaud, Jean-Christophe & de Gardelle, Vincent, 2021. "How economic success shapes redistribution: The role of self-serving beliefs, in-group bias and justice principles," Judgment and Decision Making, Cambridge University Press, vol. 16(4), pages 932-949, July.
    19. Dmitri Vinogradov & Yousef Makhlouf, 2017. "Signaling Probabilities in Ambiguity: on the impact of vague news," Working Papers 2017_12, Business School - Economics, University of Glasgow.
    20. Chen, Si & Schildberg-Hörisch, Hannah, 2018. "Looking at the bright side: The motivation value of overconfidence," DICE Discussion Papers 291, Heinrich Heine University Düsseldorf, Düsseldorf Institute for Competition Economics (DICE).
    21. Nura Sidarus & Stefano Palminteri & Valérian Chambon, 2019. "Cost-benefit trade-offs in decision-making and learning," PLOS Computational Biology, Public Library of Science, vol. 15(9), pages 1-28, September.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:nat:natcom:v:14:y:2023:i:1:d:10.1038_s41467-023-42589-5. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.nature.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.