IDEAS home Printed from https://ideas.repec.org/p/hal/journl/halshs-01236045.html
   My bibliography  Save this paper

Contextual modulation of value signals in reward and punishment learning

Author

Listed:
  • Stefano Palminteri

    (Institute of Cognitive Neuroscience - UCL - University College of London [London], LNC2 - Laboratoire de Neurosciences Cognitives & Computationnelles - DEC - Département d'Etudes Cognitives - ENS-PSL - ENS-PSL - École normale supérieure - Paris - PSL - Université Paris Sciences et Lettres - INSERM - Institut National de la Santé et de la Recherche Médicale)

  • Mehdi Khamassi

    (AMAC - ISIR - Institut des Systèmes Intelligents et de Robotique - UPMC - Université Pierre et Marie Curie - Paris 6 - CNRS - Centre National de la Recherche Scientifique, UNITN - Università degli Studi di Trento = University of Trento)

  • Mateus Joffily

    (GATE Lyon Saint-Étienne - Groupe d'Analyse et de Théorie Economique Lyon - Saint-Etienne - ENS de Lyon - École normale supérieure de Lyon - Université de Lyon - UL2 - Université Lumière - Lyon 2 - UCBL - Université Claude Bernard Lyon 1 - Université de Lyon - UJM - Université Jean Monnet - Saint-Étienne - CNRS - Centre National de la Recherche Scientifique, UNITN - Università degli Studi di Trento = University of Trento)

  • Giorgio Coricelli

    (LNC2 - Laboratoire de Neurosciences Cognitives & Computationnelles - DEC - Département d'Etudes Cognitives - ENS-PSL - ENS-PSL - École normale supérieure - Paris - PSL - Université Paris Sciences et Lettres - INSERM - Institut National de la Santé et de la Recherche Médicale)

Abstract

Compared with reward seeking, punishment avoidance learning is less clearly understood at both the computational and neurobiological levels. Here we demonstrate, using computational modelling and fMRI in humans, that learning option values in a relative—context-dependent—scale offers a simple computational solution for avoidance learning. The context (or state) value sets the reference point to which an outcome should be compared before updating the option value. Consequently, in contexts with an overall negative expected value, successful punishment avoidance acquires a positive value, thus reinforcing the response. As revealed by post-learning assessment of options values, contextual influences are enhanced when subjects are informed about the result of the forgone alternative (counterfactual information). This is mirrored at the neural level by a shift in negative outcome encoding from the anterior insula to the ventral striatum, suggesting that value contextualization also limits the need to mobilize an opponent punishment learning system.

Suggested Citation

  • Stefano Palminteri & Mehdi Khamassi & Mateus Joffily & Giorgio Coricelli, 2015. "Contextual modulation of value signals in reward and punishment learning," Post-Print halshs-01236045, HAL.
  • Handle: RePEc:hal:journl:halshs-01236045
    DOI: 10.1038/ncomms9096
    Note: View the original document on HAL open archive server: https://shs.hal.science/halshs-01236045v1
    as

    Download full text from publisher

    File URL: https://shs.hal.science/halshs-01236045v1/document
    Download Restriction: no

    File URL: https://libkey.io/10.1038/ncomms9096?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Loomes, Graham & Sugden, Robert, 1982. "Regret Theory: An Alternative Theory of Rational Choice under Uncertainty," Economic Journal, Royal Economic Society, vol. 92(368), pages 805-824, December.
    2. Camillo Padoa-Schioppa & Aldo Rustichini, 2014. "Rational Attention and Adaptive Coding: A Puzzle and a Solution," American Economic Review, American Economic Association, vol. 104(5), pages 507-513, May.
    3. Mathias Pessiglione & Ben Seymour & Guillaume Flandin & Raymond J. Dolan & Chris D. Frith, 2006. "Dopamine-dependent prediction errors underpin reward-seeking behaviour in humans," Nature, Nature, vol. 442(7106), pages 1042-1045, August.
    4. Jean Daunizeau & Vincent Adam & Lionel Rigoux, 2014. "VBA: A Probabilistic Treatment of Nonlinear Models for Neurobiological and Behavioural Data," PLOS Computational Biology, Public Library of Science, vol. 10(1), pages 1-16, January.
    5. Richard W. Morris & Amir Dezfouli & Kristi R. Griffiths & Bernard W. Balleine, 2014. "Action-value comparisons in the dorsolateral prefrontal cortex control choice between goal-directed actions," Nature Communications, Nature, vol. 5(1), pages 1-10, December.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Johann Lussange & Ivan Lazarevich & Sacha Bourgeois-Gironde & Stefano Palminteri & Boris Gutkin, 2021. "Modelling Stock Markets by Multi-agent Reinforcement Learning," Computational Economics, Springer;Society for Computational Economics, vol. 57(1), pages 113-147, January.
    2. repec:cup:judgdm:v:17:y:2022:i:2:p:425-448 is not listed on IDEAS
    3. Stefano Palminteri & Germain Lefebvre & Emma J Kilford & Sarah-Jayne Blakemore, 2017. "Confirmation bias in human reinforcement learning: Evidence from counterfactual feedback processing," PLOS Computational Biology, Public Library of Science, vol. 13(8), pages 1-22, August.
    4. Koen M. M. Frolichs & Gabriela Rosenblau & Christoph W. Korn, 2022. "Incorporating social knowledge structures into computational models," Nature Communications, Nature, vol. 13(1), pages 1-18, December.
    5. Johann Lussange & Stefano Vrizzi & Sacha Bourgeois-Gironde & Stefano Palminteri & Boris Gutkin, 2023. "Stock Price Formation: Precepts from a Multi-Agent Reinforcement Learning Model," Computational Economics, Springer;Society for Computational Economics, vol. 61(4), pages 1523-1544, April.
    6. Lefebvre, Germain & Nioche, Aurélien & Bourgeois-Gironde, Sacha & Palminteri, Stefano, 2018. "An Empirical Investigation of the Emergence of Money: Contrasting Temporal Difference and Opportunity Cost Reinforcement Learning," MPRA Paper 85586, University Library of Munich, Germany.
    7. M. A. Pisauro & E. F. Fouragnan & D. H. Arabadzhiyska & M. A. J. Apps & M. G. Philiastides, 2022. "Neural implementation of computational mechanisms underlying the continuous trade-off between cooperation and competition," Nature Communications, Nature, vol. 13(1), pages 1-18, December.
    8. Antoine Collomb-Clerc & Maëlle C. M. Gueguen & Lorella Minotti & Philippe Kahane & Vincent Navarro & Fabrice Bartolomei & Romain Carron & Jean Regis & Stephan Chabardès & Stefano Palminteri & Julien B, 2023. "Human thalamic low-frequency oscillations correlate with expected value and outcomes during reinforcement learning," Nature Communications, Nature, vol. 14(1), pages 1-10, December.
    9. Lou Safra & Coralie Chevallier & Stefano Palminteri, 2019. "Depressive symptoms are associated with blunted reward learning in social contexts," PLOS Computational Biology, Public Library of Science, vol. 15(7), pages 1-22, July.
    10. Wei-Hsiang Lin & Justin L Gardner & Shih-Wei Wu, 2020. "Context effects on probability estimation," PLOS Biology, Public Library of Science, vol. 18(3), pages 1-45, March.
    11. Johann Lussange & Boris Gutkin, 2023. "Order book regulatory impact on stock market quality: a multi-agent reinforcement learning perspective," Papers 2302.04184, arXiv.org.
    12. Simon Ciranka & Juan Linde-Domingo & Ivan Padezhki & Clara Wicharz & Charley M. Wu & Bernhard Spitzer, 2022. "Asymmetric reinforcement learning facilitates human inference of transitive relations," Nature Human Behaviour, Nature, vol. 6(4), pages 555-564, April.
    13. Maël Lebreton & Karin Bacily & Stefano Palminteri & Jan B Engelmann, 2019. "Contextual influence on confidence judgments in human reinforcement learning," PLOS Computational Biology, Public Library of Science, vol. 15(4), pages 1-27, April.
    14. Mikhail S. Spektor & Hannah Seidler, 2022. "Violations of economic rationality due to irrelevant information during learning in decision from experience," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 17(2), pages 425-448, March.
    15. Stefano Palminteri & Emma J Kilford & Giorgio Coricelli & Sarah-Jayne Blakemore, 2016. "The Computational Development of Reinforcement Learning during Adolescence," PLOS Computational Biology, Public Library of Science, vol. 12(6), pages 1-25, June.
    16. Sepulveda, Pradyumna & Aitsahalia, Ines & Kumar, Krishan & Atkin, Tobias & Iigaya, Kiyohito, 2024. "Addressing Altered Anticipation as a Transdiagnostic Target through Computational Psychiatry," OSF Preprints dtm3r, Center for Open Science.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Antoine Collomb-Clerc & Maëlle C. M. Gueguen & Lorella Minotti & Philippe Kahane & Vincent Navarro & Fabrice Bartolomei & Romain Carron & Jean Regis & Stephan Chabardès & Stefano Palminteri & Julien B, 2023. "Human thalamic low-frequency oscillations correlate with expected value and outcomes during reinforcement learning," Nature Communications, Nature, vol. 14(1), pages 1-10, December.
    2. He A Xu & Alireza Modirshanechi & Marco P Lehmann & Wulfram Gerstner & Michael H Herzog, 2021. "Novelty is not surprise: Human exploratory and adaptive behavior in sequential decision-making," PLOS Computational Biology, Public Library of Science, vol. 17(6), pages 1-32, June.
    3. Maël Lebreton & Karin Bacily & Stefano Palminteri & Jan B Engelmann, 2019. "Contextual influence on confidence judgments in human reinforcement learning," PLOS Computational Biology, Public Library of Science, vol. 15(4), pages 1-27, April.
    4. Romane Cecchi & Antoine Collomb-Clerc & Inès Rachidi & Lorella Minotti & Philippe Kahane & Mathias Pessiglione & Julien Bastin, 2024. "Direct stimulation of anterior insula and ventromedial prefrontal cortex disrupts economic choices," Nature Communications, Nature, vol. 15(1), pages 1-11, December.
    5. Daniel Serra, 2021. "Decision-making: from neuroscience to neuroeconomics—an overview," Theory and Decision, Springer, vol. 91(1), pages 1-80, July.
    6. Stefano Palminteri & Emma J Kilford & Giorgio Coricelli & Sarah-Jayne Blakemore, 2016. "The Computational Development of Reinforcement Learning during Adolescence," PLOS Computational Biology, Public Library of Science, vol. 12(6), pages 1-25, June.
    7. Georgia Perakis & Guillaume Roels, 2008. "Regret in the Newsvendor Model with Partial Information," Operations Research, INFORMS, vol. 56(1), pages 188-203, February.
    8. Jinyi Hu, 2023. "Linguistic Multiple-Attribute Decision Making Based on Regret Theory and Minimax-DEA," Mathematics, MDPI, vol. 11(20), pages 1-14, October.
    9. repec:cup:judgdm:v:15:y:2020:i:6:p:1044-1051 is not listed on IDEAS
    10. Sacha Bourgeois-Gironde, 2017. "How regret moves individual and collective choices towards rationality," Chapters, in: Morris Altman (ed.), Handbook of Behavioural Economics and Smart Decision-Making, chapter 11, pages 188-204, Edward Elgar Publishing.
    11. Martín Egozcue & Xu Guo & Wing-Keung Wong, 2015. "Optimal output for the regret-averse competitive firm under price uncertainty," Eurasian Economic Review, Springer;Eurasia Business and Economics Society, vol. 5(2), pages 279-295, December.
    12. Jhunjhunwala, Tanushree, 2021. "Searching to avoid regret: An experimental evidence," Journal of Economic Behavior & Organization, Elsevier, vol. 189(C), pages 298-319.
    13. Chih-Chung Ting & Nahuel Salem-Garcia & Stefano Palminteri & Jan B. Engelmann & Maël Lebreton, 2023. "Neural and computational underpinnings of biased confidence in human reinforcement learning," Nature Communications, Nature, vol. 14(1), pages 1-18, December.
    14. van Dijk, Wilco W. & van der Pligt, Joop, 1997. "The Impact of Probability and Magnitude of Outcome on Disappointment and Elation," Organizational Behavior and Human Decision Processes, Elsevier, vol. 69(3), pages 277-284, March.
    15. Ming Li & Dipjyoti Majumdar, 2010. "A Psychologically Based Model of Voter Turnout," Journal of Public Economic Theory, Association for Public Economic Theory, vol. 12(5), pages 979-1002, October.
    16. Kobana Abukari & Isaac Otchere, 2020. "Dominance of hybrid contratum strategies over momentum and contrarian strategies: half a century of evidence," Financial Markets and Portfolio Management, Springer;Swiss Society for Financial Market Research, vol. 34(4), pages 471-505, December.
    17. Peng Li & Ju Liu & Cuiping Wei, 2019. "A Dynamic Decision Making Method Based on GM(1,1) Model with Pythagorean Fuzzy Numbers for Selecting Waste Disposal Enterprises," Sustainability, MDPI, vol. 11(20), pages 1-19, October.
    18. Vaida Kaduskeviciute & Sigitas Urbonavicius, 2019. "Webrooming: A Way of Dealing with Uncertainties in Purchasing," Tržište/Market, Faculty of Economics and Business, University of Zagreb, vol. 31(2), pages 139-152.
    19. Qin, Jie, 2015. "A model of regret, investor behavior, and market turbulence," Journal of Economic Theory, Elsevier, vol. 160(C), pages 150-174.
    20. Grenet, Julien & He, YingHua & Kübler, Dorothea, 2022. "Preference Discovery in University Admissions: The Case for Dynamic Multioffer Mechanisms," EconStor Open Access Articles and Book Chapters, ZBW - Leibniz Information Centre for Economics, vol. 130(6), pages 1-1.
    21. Embrey, Matthew & Seel, Christian & Philipp Reiss, J., 2024. "Gambling in risk-taking contests: Experimental evidence," Journal of Economic Behavior & Organization, Elsevier, vol. 221(C), pages 570-585.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:hal:journl:halshs-01236045. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: CCSD (email available below). General contact details of provider: https://hal.archives-ouvertes.fr/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.