IDEAS home Printed from https://ideas.repec.org/a/plo/pcbi00/1006973.html
   My bibliography  Save this article

Contextual influence on confidence judgments in human reinforcement learning

Author

Listed:
  • Maël Lebreton
  • Karin Bacily
  • Stefano Palminteri
  • Jan B Engelmann

Abstract

The ability to correctly estimate the probability of one’s choices being correct is fundamental to optimally re-evaluate previous choices or to arbitrate between different decision strategies. Experimental evidence nonetheless suggests that this metacognitive process—confidence judgment- is susceptible to numerous biases. Here, we investigate the effect of outcome valence (gains or losses) on confidence while participants learned stimulus-outcome associations by trial-and-error. In two experiments, participants were more confident in their choices when learning to seek gains compared to avoiding losses, despite equal difficulty and performance between those two contexts. Computational modelling revealed that this bias is driven by the context-value, a dynamically updated estimate of the average expected-value of choice options, necessary to explain equal performance in the gain and loss domain. The biasing effect of context-value on confidence, revealed here for the first time in a reinforcement-learning context, is therefore domain-general, with likely important functional consequences. We show that one such consequence emerges in volatile environments, where the (in)flexibility of individuals’ learning strategies differs when outcomes are framed as gains or losses. Despite apparent similar behavior- profound asymmetries might therefore exist between learning to avoid losses and learning to seek gains.Author summary: In order to arbitrate between different decision strategies, as well as to inform future choices, a decision maker needs to estimate the probability of her choices being correct as precisely as possible. Surprisingly, this metacognitive operation, known as confidence judgment, has not been systematically investigated in the context of simple instrumental-learning tasks. Here, we assessed how confident individuals are in their choices when learning stimulus-outcome associations by trial-and-errors to maximize gains or to minimize losses. In two experiments, we show that individuals are more confident in their choices when learning to seek gains compared to avoiding losses, despite equal difficulty and performance between those two contexts. To simultaneously account for this pattern of choices and confidence judgments, we propose that individuals learn context-values, which approximate the average expected-value of choice options. We finally show that, in volatile environments, the biasing effect of context-value on confidence induces difference in learning flexibility when outcomes are framed as gains or losses.

Suggested Citation

  • Maël Lebreton & Karin Bacily & Stefano Palminteri & Jan B Engelmann, 2019. "Contextual influence on confidence judgments in human reinforcement learning," PLOS Computational Biology, Public Library of Science, vol. 15(4), pages 1-27, April.
  • Handle: RePEc:plo:pcbi00:1006973
    DOI: 10.1371/journal.pcbi.1006973
    as

    Download full text from publisher

    File URL: https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1006973
    Download Restriction: no

    File URL: https://journals.plos.org/ploscompbiol/article/file?id=10.1371/journal.pcbi.1006973&type=printable
    Download Restriction: no

    File URL: https://libkey.io/10.1371/journal.pcbi.1006973?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Erev, Ido & Roth, Alvin E, 1998. "Predicting How People Play Games: Reinforcement Learning in Experimental Games with Unique, Mixed Strategy Equilibria," American Economic Review, American Economic Association, vol. 88(4), pages 848-881, September.
    2. Stefano Palminteri & Mehdi Khamassi & Mateus Joffily & Giorgio Coricelli, 2015. "Contextual modulation of value signals in reward and punishment learning," Nature Communications, Nature, vol. 6(1), pages 1-14, November.
    3. Sophie Bavard & Maël Lebreton & Mehdi Khamassi & Giorgio Coricelli & Stefano Palminteri, 2018. "Reference-point centering and range-adaptation enhance human reinforcement learning at the cost of irrational preferences," Nature Communications, Nature, vol. 9(1), pages 1-12, December.
    4. Edi Karni, 2009. "A Mechanism for Eliciting Probabilities," Econometrica, Econometric Society, vol. 77(2), pages 603-606, March.
    5. Guillaume Hollard & Sébastien Massoni & Jean-Christophe Vergnaud, 2016. "In search of good probability assessors: an experimental comparison of elicitation rules for confidence judgments," Université Paris1 Panthéon-Sorbonne (Post-Print and Working Papers) hal-01306258, HAL.
    6. Stefano Palminteri & Mehdi Khamassi & Mateus Joffily & Giorgio Coricelli, 2015. "Contextual modulation of value signals in reward and punishment learning," Post-Print halshs-01236045, HAL.
    7. Florent Meyniel & Daniel Schlunegger & Stanislas Dehaene, 2015. "The Sense of Confidence during Probabilistic Learning: A Normative Account," PLOS Computational Biology, Public Library of Science, vol. 11(6), pages 1-25, June.
    8. Karl Schlag & James Tremewan & Joël Weele, 2015. "A penny for your thoughts: a survey of methods for eliciting beliefs," Experimental Economics, Springer;Economic Science Association, vol. 18(3), pages 457-490, September.
    9. Stefano Palminteri & Emma J Kilford & Giorgio Coricelli & Sarah-Jayne Blakemore, 2016. "The Computational Development of Reinforcement Learning during Adolescence," PLOS Computational Biology, Public Library of Science, vol. 12(6), pages 1-25, June.
    10. Nathaniel D. Daw & John P. O'Doherty & Peter Dayan & Ben Seymour & Raymond J. Dolan, 2006. "Cortical substrates for exploratory decisions in humans," Nature, Nature, vol. 441(7095), pages 876-879, June.
    11. Mathias Pessiglione & Ben Seymour & Guillaume Flandin & Raymond J. Dolan & Chris D. Frith, 2006. "Dopamine-dependent prediction errors underpin reward-seeking behaviour in humans," Nature, Nature, vol. 442(7106), pages 1042-1045, August.
    12. Karl Schlag & James Tremewan & Joël Weele, 2015. "A penny for your thoughts: a survey of methods for eliciting beliefs," Experimental Economics, Springer;Economic Science Association, vol. 18(3), pages 457-490, September.
    13. Philipp Koellinger & Theresa Treffers, 2015. "Joy Leads to Overconfidence, and a Simple Countermeasure," PLOS ONE, Public Library of Science, vol. 10(12), pages 1-22, December.
    14. Vincent de Gardelle & François Le Corre & Pascal Mamassian, 2016. "Confidence as a Common Currency between Vision and Audition," PLOS ONE, Public Library of Science, vol. 11(1), pages 1-11, January.
    15. Jean Daunizeau & Vincent Adam & Lionel Rigoux, 2014. "VBA: A Probabilistic Treatment of Nonlinear Models for Neurobiological and Behavioural Data," PLOS Computational Biology, Public Library of Science, vol. 10(1), pages 1-16, January.
    16. repec:cup:judgdm:v:5:y:2010:i:6:p:437-449 is not listed on IDEAS
    17. Tilmann A. Klein & Markus Ullsperger & Gerhard Jocham, 2017. "Learning relative values in the striatum induces violations of normative decision making," Nature Communications, Nature, vol. 8(1), pages 1-12, December.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Minhua Shao & Congcong Xie & Lijun Sun & Lijuan Jiang, 2019. "Optimal Layout of Static Guidance Information in Comprehensive Transportation Hubs Based on Passenger Pathfinding Behavior," Sustainability, MDPI, vol. 11(13), pages 1-21, July.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Chih-Chung Ting & Nahuel Salem-Garcia & Stefano Palminteri & Jan B. Engelmann & Maël Lebreton, 2023. "Neural and computational underpinnings of biased confidence in human reinforcement learning," Nature Communications, Nature, vol. 14(1), pages 1-18, December.
    2. Antoine Collomb-Clerc & Maëlle C. M. Gueguen & Lorella Minotti & Philippe Kahane & Vincent Navarro & Fabrice Bartolomei & Romain Carron & Jean Regis & Stephan Chabardès & Stefano Palminteri & Julien B, 2023. "Human thalamic low-frequency oscillations correlate with expected value and outcomes during reinforcement learning," Nature Communications, Nature, vol. 14(1), pages 1-10, December.
    3. Stefano Palminteri & Germain Lefebvre & Emma J Kilford & Sarah-Jayne Blakemore, 2017. "Confirmation bias in human reinforcement learning: Evidence from counterfactual feedback processing," PLOS Computational Biology, Public Library of Science, vol. 13(8), pages 1-22, August.
    4. Lefebvre, Germain & Nioche, Aurélien & Bourgeois-Gironde, Sacha & Palminteri, Stefano, 2018. "An Empirical Investigation of the Emergence of Money: Contrasting Temporal Difference and Opportunity Cost Reinforcement Learning," MPRA Paper 85586, University Library of Munich, Germany.
    5. Stefano Palminteri & Emma J Kilford & Giorgio Coricelli & Sarah-Jayne Blakemore, 2016. "The Computational Development of Reinforcement Learning during Adolescence," PLOS Computational Biology, Public Library of Science, vol. 12(6), pages 1-25, June.
    6. Juan Dubra & Jean-Pierre Benoît & Giorgia Romagnoli, 2019. "Belief elicitation when more than money matters," Documentos de Trabajo/Working Papers 1901, Facultad de Ciencias Empresariales y Economia. Universidad de Montevideo..
    7. Dominik Bauer & Irenaeus Wolff, 2018. "Biases in Beliefs: Experimental Evidence," TWI Research Paper Series 109, Thurgauer Wirtschaftsinstitut, Universität Konstanz.
    8. Folli, Dominik & Wolff, Irenaeus, 2022. "Biases in belief reports," Journal of Economic Psychology, Elsevier, vol. 88(C).
    9. Bauer, Dominik & Wolff, Irenaeus, 2019. "Biases in Beliefs," VfS Annual Conference 2019 (Leipzig): 30 Years after the Fall of the Berlin Wall - Democracy and Market Economy 203601, Verein für Socialpolitik / German Economic Association.
    10. Tal Neiman & Yonatan Loewenstein, 2011. "Reinforcement learning in professional basketball players," Discussion Paper Series dp593, The Federmann Center for the Study of Rationality, the Hebrew University, Jerusalem.
    11. Alexander Coutts, 2019. "Good news and bad news are still news: experimental evidence on belief updating," Experimental Economics, Springer;Economic Science Association, vol. 22(2), pages 369-395, June.
    12. Randolph Sloof & Ferdinand A. Siemens, 2017. "Illusion of control and the pursuit of authority," Experimental Economics, Springer;Economic Science Association, vol. 20(3), pages 556-573, September.
    13. Jan B. Engelmann & Maël Lebreton & Nahuel A. Salem-Garcia & Peter Schwardmann & Joël J. van der Weele, 2024. "Anticipatory Anxiety and Wishful Thinking," American Economic Review, American Economic Association, vol. 114(4), pages 926-960, April.
    14. Alexander Coutts, 2017. "Good news and bad news are still news: Experimental evidence on belief updating," FEUNL Working Paper Series novaf:wp1703, Universidade Nova de Lisboa, Faculdade de Economia.
    15. Theo Offerman & Asa B. Palley, 2016. "Lossed in translation: an off-the-shelf method to recover probabilistic beliefs from loss-averse agents," Experimental Economics, Springer;Economic Science Association, vol. 19(1), pages 1-30, March.
    16. van Veldhuizen, Roel, 2022. "Gender Differences in Tournament Choices: Risk Preferences, Overconfidence or Competitiveness?," EconStor Open Access Articles and Book Chapters, ZBW - Leibniz Information Centre for Economics, vol. 20(4), pages 1595-1618.
    17. Bose, Subir & Daripa, Arup, 2023. "Eliciting second-order beliefs," Journal of Mathematical Economics, Elsevier, vol. 107(C).
    18. Crosetto, Paolo & Filippin, Antonio & Katuščák, Peter & Smith, John, 2020. "Central tendency bias in belief elicitation," Journal of Economic Psychology, Elsevier, vol. 78(C).
    19. Johann Lussange & Stefano Vrizzi & Sacha Bourgeois-Gironde & Stefano Palminteri & Boris Gutkin, 2023. "Stock Price Formation: Precepts from a Multi-Agent Reinforcement Learning Model," Computational Economics, Springer;Society for Computational Economics, vol. 61(4), pages 1523-1544, April.
    20. de Haan, Thomas, 2020. "Eliciting belief distributions using a random two-level partitioning of the state space," Working Papers in Economics 1/20, University of Bergen, Department of Economics.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pcbi00:1006973. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: ploscompbiol (email available below). General contact details of provider: https://journals.plos.org/ploscompbiol/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.