IDEAS home Printed from https://ideas.repec.org/a/nat/natcom/v15y2024i1d10.1038_s41467-024-50205-3.html
   My bibliography  Save this article

Learning to express reward prediction error-like dopaminergic activity requires plastic representations of time

Author

Listed:
  • Ian Cone

    (Imperial College London
    University of Texas Medical School at Houston
    Rice University)

  • Claudia Clopath

    (Imperial College London)

  • Harel Z. Shouval

    (University of Texas Medical School at Houston
    Rice University)

Abstract

The dominant theoretical framework to account for reinforcement learning in the brain is temporal difference learning (TD) learning, whereby certain units signal reward prediction errors (RPE). The TD algorithm has been traditionally mapped onto the dopaminergic system, as firing properties of dopamine neurons can resemble RPEs. However, certain predictions of TD learning are inconsistent with experimental results, and previous implementations of the algorithm have made unscalable assumptions regarding stimulus-specific fixed temporal bases. We propose an alternate framework to describe dopamine signaling in the brain, FLEX (Flexibly Learned Errors in Expected Reward). In FLEX, dopamine release is similar, but not identical to RPE, leading to predictions that contrast to those of TD. While FLEX itself is a general theoretical framework, we describe a specific, biophysically plausible implementation, the results of which are consistent with a preponderance of both existing and reanalyzed experimental data.

Suggested Citation

  • Ian Cone & Claudia Clopath & Harel Z. Shouval, 2024. "Learning to express reward prediction error-like dopaminergic activity requires plastic representations of time," Nature Communications, Nature, vol. 15(1), pages 1-17, December.
    • Handle: RePEc:nat:natcom:v:15:y:2024:i:1:d:10.1038_s41467-024-50205-3
    DOI: 10.1038/s41467-024-50205-3
    as

    Download full text from publisher

    File URL: https://www.nature.com/articles/s41467-024-50205-3
    File Function: Abstract
    Download Restriction: no
    File URL: https://libkey.io/10.1038/s41467-024-50205-3?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Su Z. Hong & Lukas Mesik & Cooper D. Grossman & Jeremiah Y. Cohen & Boram Lee & Daniel Severin & Hey-Kyoung Lee & Johannes W. Hell & Alfredo Kirkwood, 2022. "Norepinephrine potentiates and serotonin depresses visual cortical responses by transforming eligibility traces," Nature Communications, Nature, vol. 13(1), pages 1-12, December.
    2. Brian DePasquale & Christopher J Cueva & Kanaka Rajan & G Sean Escola & L F Abbott, 2018. "full-FORCE: A target-based method for training recurrent networks," PLOS ONE, Public Library of Science, vol. 13(2), pages 1-18, February.
    3. Pascale Waelti & Anthony Dickinson & Wolfram Schultz, 2001. "Dopamine responses comply with basic assumptions of formal learning theory," Nature, Nature, vol. 412(6842), pages 43-48, July.
    4. Ian Cone & Claudia Clopath & Harel Z. Shouval, 2024. "Learning to express reward prediction error-like dopaminergic activity requires plastic representations of time," Nature Communications, Nature, vol. 15(1), pages 1-17, December.
    5. Luke T. Coddington & Sarah E. Lindo & Joshua T. Dudman, 2023. "Mesolimbic dopamine adapts the rate of learning from action," Nature, Nature, vol. 614(7947), pages 294-302, February.
    6. Will Dabney & Zeb Kurth-Nelson & Naoshige Uchida & Clara Kwon Starkweather & Demis Hassabis & Rémi Munos & Matthew Botvinick, 2020. "A distributional code for value in dopamine-based reinforcement learning," Nature, Nature, vol. 577(7792), pages 671-675, January.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Ian Cone & Claudia Clopath & Harel Z. Shouval, 2024. "Learning to express reward prediction error-like dopaminergic activity requires plastic representations of time," Nature Communications, Nature, vol. 15(1), pages 1-17, December.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Minkyu Shin & Jin Kim & Minkyung Kim, 2020. "Measuring Human Adaptation to AI in Decision Making: Application to Evaluate Changes after AlphaGo," Papers 2012.15035, arXiv.org, revised Jan 2021.
    2. Smith, Trenton G. & Tasnadi, Attila, 2007. "A theory of natural addiction," Games and Economic Behavior, Elsevier, vol. 59(2), pages 316-344, May.
    3. Wan-Yu Shih & Hsiang-Yu Yu & Cheng-Chia Lee & Chien-Chen Chou & Chien Chen & Paul W. Glimcher & Shih-Wei Wu, 2023. "Electrophysiological population dynamics reveal context dependencies during decision making in human frontal cortex," Nature Communications, Nature, vol. 14(1), pages 1-24, December.
    4. Zhewei Zhang & Yuji K. Takahashi & Marlian Montesinos-Cartegena & Thorsten Kahnt & Angela J. Langdon & Geoffrey Schoenbaum, 2024. "Expectancy-related changes in firing of dopamine neurons depend on hippocampus," Nature Communications, Nature, vol. 15(1), pages 1-16, December.
    5. Barbara Feulner & Matthew G. Perich & Raeed H. Chowdhury & Lee E. Miller & Juan A. Gallego & Claudia Clopath, 2022. "Small, correlated changes in synaptic connectivity may facilitate rapid motor learning," Nature Communications, Nature, vol. 13(1), pages 1-14, December.
    6. Leo Chi U Seak & Simone Ferrari-Toniolo & Ritesh Jain & Kirby Nielsen & Wolfram Schultz, 2023. "Systematic comparison of risky choices in humans and monkeys," Working Papers 202316, University of Liverpool, Department of Economics.
    7. Filippo Costa & Eline V. Schaft & Geertjan Huiskamp & Erik J. Aarnoutse & Maryse A. van’t Klooster & Niklaus Krayenbühl & Georgia Ramantani & Maeike Zijlmans & Giacomo Indiveri & Johannes Sarnthein, 2024. "Robust compression and detection of epileptiform patterns in ECoG using a real-time spiking neural network hardware framework," Nature Communications, Nature, vol. 15(1), pages 1-12, December.
    8. Newman, Andrew H. & Tafkov, Ivo D. & Waddoups, Nathan J. & Xiong, Xiaomei Grazia, 2024. "The effect of reward frequency on performance under cash rewards and tangible rewards," Accounting, Organizations and Society, Elsevier, vol. 112(C).
    9. Linan Diao & Jörg Rieskamp, 2011. "Reinforcement Learning in Repeated Portfolio Decisions," Jena Economics Research Papers 2011-009, Friedrich-Schiller-University Jena.
    10. Smith, Trenton G., 2023. "Endocrine state is the physical manifestation of subjective beliefs," Journal of Economic Psychology, Elsevier, vol. 96(C).
    11. Laurel S Morris & Agnes Norbury & Derek A Smith & Neil A Harrison & Valerie Voon & James W Murrough, 2020. "Dissociating self-generated volition from externally-generated motivation," PLOS ONE, Public Library of Science, vol. 15(5), pages 1-13, May.
    12. Burkhard Pleger & Christian C Ruff & Felix Blankenburg & Stefan Klöppel & Jon Driver & Raymond J Dolan, 2009. "Influence of Dopaminergically Mediated Reward on Somatosensory Decision-Making," PLOS Biology, Public Library of Science, vol. 7(7), pages 1-10, July.
    13. Colin W. Hoy & David R. Quiroga-Martinez & Eduardo Sandoval & David King-Stephens & Kenneth D. Laxer & Peter Weber & Jack J. Lin & Robert T. Knight, 2023. "Asymmetric coding of reward prediction errors in human insula and dorsomedial prefrontal cortex," Nature Communications, Nature, vol. 14(1), pages 1-14, December.
    14. Min Jung Kim & Daniel J. Gibson & Dan Hu & Tomoko Yoshida & Emily Hueske & Ayano Matsushima & Ara Mahar & Cynthia J. Schofield & Patlapa Sompolpong & Kathy T. Tran & Lin Tian & Ann M. Graybiel, 2024. "Dopamine release plateau and outcome signals in dorsal striatum contrast with classic reinforcement learning formulations," Nature Communications, Nature, vol. 15(1), pages 1-20, December.
    15. Greg Jensen & Fabian Muñoz & Yelda Alkan & Vincent P Ferrera & Herbert S Terrace, 2015. "Implicit Value Updating Explains Transitive Inference Performance: The Betasort Model," PLOS Computational Biology, Public Library of Science, vol. 11(9), pages 1-27, September.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:nat:natcom:v:15:y:2024:i:1:d:10.1038_s41467-024-50205-3. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.nature.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.