IDEAS home Printed from https://ideas.repec.org/a/plo/pcbi00/1006370.html
   My bibliography  Save this article

Dorsal anterior cingulate-brainstem ensemble as a reinforcement meta-learner

Author

Listed:
  • Massimo Silvetti
  • Eliana Vassena
  • Elger Abrahamse
  • Tom Verguts

Abstract

Optimal decision-making is based on integrating information from several dimensions of decisional space (e.g., reward expectation, cost estimation, effort exertion). Despite considerable empirical and theoretical efforts, the computational and neural bases of such multidimensional integration have remained largely elusive. Here we propose that the current theoretical stalemate may be broken by considering the computational properties of a cortical-subcortical circuit involving the dorsal anterior cingulate cortex (dACC) and the brainstem neuromodulatory nuclei: ventral tegmental area (VTA) and locus coeruleus (LC). From this perspective, the dACC optimizes decisions about stimuli and actions, and using the same computational machinery, it also modulates cortical functions (meta-learning), via neuromodulatory control (VTA and LC). We implemented this theory in a novel neuro-computational model–the Reinforcement Meta Learner (RML). We outline how the RML captures critical empirical findings from an unprecedented range of theoretical domains, and parsimoniously integrates various previous proposals on dACC functioning.Author summary: A major challenge for all organisms is selecting optimal behaviour to obtain resources while minimizing energetic and other expenses. Evolution provided mammals with exceptional decision-making capabilities to face this challenge. Even though neuroscientists have identified a heterogeneous and distributed set of brain structures to be involved, a comprehensive theory about the biological and computational basis of such decision-making is yet to be formulated. We propose that the interaction between the medial prefrontal cortex (a part of the frontal lobes) and the subcortical nuclei releasing catecholaminergic neuromodulators will be key to such a theory. We argue that this interaction allows both the selection of optimal behaviour and, more importantly, the optimal modulation of the very brain circuits that drive such behavioral selection (i.e., meta-learning). We implemented this theory in a novel neuro-computational model, the Reinforcement Meta-Learner (RML). By means of computer simulations we showed that the RML provides a biological and computational account for a set of neuroscientific data with unprecedented scope, thereby suggesting a critical mechanism of decision-making in the mammalian brain.

Suggested Citation

  • Massimo Silvetti & Eliana Vassena & Elger Abrahamse & Tom Verguts, 2018. "Dorsal anterior cingulate-brainstem ensemble as a reinforcement meta-learner," PLOS Computational Biology, Public Library of Science, vol. 14(8), pages 1-32, August.
  • Handle: RePEc:plo:pcbi00:1006370
    DOI: 10.1371/journal.pcbi.1006370
    as

    Download full text from publisher

    File URL: https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1006370
    Download Restriction: no

    File URL: https://journals.plos.org/ploscompbiol/article/file?id=10.1371/journal.pcbi.1006370&type=printable
    Download Restriction: no

    File URL: https://libkey.io/10.1371/journal.pcbi.1006370?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Brad E. Pfeiffer & David J. Foster, 2013. "Hippocampal place-cell sequences depict future paths to remembered goals," Nature, Nature, vol. 497(7447), pages 74-79, May.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Pieter Verbeke & Tom Verguts, 2019. "Learning to synchronize: How biological agents can couple neural task modules for dealing with the stability-plasticity dilemma," PLOS Computational Biology, Public Library of Science, vol. 15(8), pages 1-25, August.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. J Matthew Mahoney & Ali S Titiz & Amanda E Hernan & Rod C Scott, 2016. "Short-Range Temporal Interactions in Sleep; Hippocampal Spike Avalanches Support a Large Milieu of Sequential Activity Including Replay," PLOS ONE, Public Library of Science, vol. 11(2), pages 1-25, February.
    2. Marta Huelin Gorriz & Masahiro Takigawa & Daniel Bendor, 2023. "The role of experience in prioritizing hippocampal replay," Nature Communications, Nature, vol. 14(1), pages 1-15, December.
    3. Nicolas Cazin & Martin Llofriu Alonso & Pablo Scleidorovich Chiodi & Tatiana Pelc & Bruce Harland & Alfredo Weitzenfeld & Jean-Marc Fellous & Peter Ford Dominey, 2019. "Reservoir computing model of prefrontal cortex creates novel combinations of previous navigation sequences from hippocampal place-cell replay with spatial reward propagation," PLOS Computational Biology, Public Library of Science, vol. 15(7), pages 1-32, July.
    4. Toon Van de Maele & Bart Dhoedt & Tim Verbelen & Giovanni Pezzulo, 2024. "A hierarchical active inference model of spatial alternation tasks and the hippocampal-prefrontal circuit," Nature Communications, Nature, vol. 15(1), pages 1-16, December.
    5. Zhewei Zhang & Yuji K. Takahashi & Marlian Montesinos-Cartegena & Thorsten Kahnt & Angela J. Langdon & Geoffrey Schoenbaum, 2024. "Expectancy-related changes in firing of dopamine neurons depend on hippocampus," Nature Communications, Nature, vol. 15(1), pages 1-16, December.
    6. Hong Yu & Xinkuan Xiang & Zongming Chen & Xu Wang & Jiaqi Dai & Xinxin Wang & Pengcheng Huang & Zheng-dong Zhao & Wei L. Shen & Haohong Li, 2021. "Periaqueductal gray neurons encode the sequential motor program in hunting behavior of mice," Nature Communications, Nature, vol. 12(1), pages 1-15, December.
    7. Anli A. Liu & Simon Henin & Saman Abbaspoor & Anatol Bragin & Elizabeth A. Buffalo & Jordan S. Farrell & David J. Foster & Loren M. Frank & Tamara Gedankien & Jean Gotman & Jennifer A. Guidera & Kari , 2022. "A consensus statement on detection of hippocampal sharp wave ripples and differentiation from other fast oscillations," Nature Communications, Nature, vol. 13(1), pages 1-14, December.
    8. Georgy Antonov & Peter Dayan, 2025. "Exploring replay," Nature Communications, Nature, vol. 16(1), pages 1-20, December.
    9. Trygve Solstad & Hosam N Yousif & Terrence J Sejnowski, 2014. "Place Cell Rate Remapping by CA3 Recurrent Collaterals," PLOS Computational Biology, Public Library of Science, vol. 10(6), pages 1-10, June.
    10. Francesco Donnarumma & Domenico Maisto & Giovanni Pezzulo, 2016. "Problem Solving as Probabilistic Inference with Subgoaling: Explaining Human Successes and Pitfalls in the Tower of Hanoi," PLOS Computational Biology, Public Library of Science, vol. 12(4), pages 1-30, April.
    11. Will D Penny & Peter Zeidman & Neil Burgess, 2013. "Forward and Backward Inference in Spatial Cognition," PLOS Computational Biology, Public Library of Science, vol. 9(12), pages 1-22, December.
    12. Lukas Grossberger & Francesco P Battaglia & Martin Vinck, 2018. "Unsupervised clustering of temporal patterns in high-dimensional neuronal ensembles using a novel dissimilarity measure," PLOS Computational Biology, Public Library of Science, vol. 14(7), pages 1-34, July.
    13. John Palmer & Adam Keane & Pulin Gong, 2017. "Learning and executing goal-directed choices by internally generated sequences in spiking neural circuits," PLOS Computational Biology, Public Library of Science, vol. 13(7), pages 1-23, July.
    14. Georgy Antonov & Christopher Gagne & Eran Eldar & Peter Dayan, 2022. "Optimism and pessimism in optimised replay," PLOS Computational Biology, Public Library of Science, vol. 18(1), pages 1-32, January.
    15. Hefei Guan & Steven J. Middleton & Takafumi Inoue & Thomas J. McHugh, 2021. "Lateralization of CA1 assemblies in the absence of CA3 input," Nature Communications, Nature, vol. 12(1), pages 1-10, December.
    16. Alexander Nitsch & Mona M. Garvert & Jacob L. S. Bellmund & Nicolas W. Schuck & Christian F. Doeller, 2024. "Grid-like entorhinal representation of an abstract value space during prospective decision making," Nature Communications, Nature, vol. 15(1), pages 1-20, December.
    17. Takamitsu Iwata & Takufumi Yanagisawa & Yuji Ikegaya & Jonathan Smallwood & Ryohei Fukuma & Satoru Oshino & Naoki Tani & Hui Ming Khoo & Haruhiko Kishima, 2024. "Hippocampal sharp-wave ripples correlate with periods of naturally occurring self-generated thoughts in humans," Nature Communications, Nature, vol. 15(1), pages 1-12, December.
    18. Guillaume Viejo & Thomas Cortier & Adrien Peyrache, 2018. "Brain-state invariant thalamo-cortical coordination revealed by non-linear encoders," PLOS Computational Biology, Public Library of Science, vol. 14(3), pages 1-25, March.
    19. Bahaaeddin Attaallah & Pierre Petitet & Rhea Zambellas & Sofia Toniolo & Maria Raquel Maio & Akke Ganse-Dumrath & Sarosh R. Irani & Sanjay G. Manohar & Masud Husain, 2024. "The role of the human hippocampus in decision-making under uncertainty," Nature Human Behaviour, Nature, vol. 8(7), pages 1366-1382, July.
    20. Siavash Ahmadi & Takuya Sasaki & Marta Sabariego & Christian Leibold & Stefan Leutgeb & Jill K. Leutgeb, 2025. "Distinct roles of dentate gyrus and medial entorhinal cortex inputs for phase precession and temporal correlations in the hippocampal CA3 area," Nature Communications, Nature, vol. 16(1), pages 1-20, December.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pcbi00:1006370. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: ploscompbiol (email available below). General contact details of provider: https://journals.plos.org/ploscompbiol/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.