IDEAS home Printed from https://ideas.repec.org/a/plo/pcbi00/1007334.html
   My bibliography  Save this article

Learning the structure of the world: The adaptive nature of state-space and action representations in multi-stage decision-making

Author

Listed:
  • Amir Dezfouli
  • Bernard W Balleine

Abstract

State-space and action representations form the building blocks of decision-making processes in the brain; states map external cues to the current situation of the agent whereas actions provide the set of motor commands from which the agent can choose to achieve specific goals. Although these factors differ across environments, it is currently unknown whether or how accurately state and action representations are acquired by the agent because previous experiments have typically provided this information a priori through instruction or pre-training. Here we studied how state and action representations adapt to reflect the structure of the world when such a priori knowledge is not available. We used a sequential decision-making task in rats in which they were required to pass through multiple states before reaching the goal, and for which the number of states and how they map onto external cues were unknown a priori. We found that, early in training, animals selected actions as if the task was not sequential and outcomes were the immediate consequence of the most proximal action. During the course of training, however, rats recovered the true structure of the environment and made decisions based on the expanded state-space, reflecting the multiple stages of the task. Similarly, we found that the set of actions expanded with training, although the emergence of new action sequences was sensitive to the experimental parameters and specifics of the training procedure. We conclude that the profile of choices shows a gradual shift from simple representations to more complex structures compatible with the structure of the world.Author summary: Everyday decision-making tasks typically require taking multiple actions and passing through multiple states before reaching desired goals. Such states constitute the state-space of the task. Here we show that, contrary to current assumptions, the state-space is not static but rather expands during training as subjects discover new states that help them efficiently solve the task. Similarly, within the same task, we show that subjects initially only consider taking simple actions, but as training progresses the set of actions can expand to include useful action sequences that reach the goal directly by passing through multiple states. These results provide evidence that state-space and action representations are not static but are acquired and then adapted to reflect the structure of the world.

Suggested Citation

  • Amir Dezfouli & Bernard W Balleine, 2019. "Learning the structure of the world: The adaptive nature of state-space and action representations in multi-stage decision-making," PLOS Computational Biology, Public Library of Science, vol. 15(9), pages 1-22, September.
  • Handle: RePEc:plo:pcbi00:1007334
    DOI: 10.1371/journal.pcbi.1007334
    as

    Download full text from publisher

    File URL: https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1007334
    Download Restriction: no

    File URL: https://journals.plos.org/ploscompbiol/article/file?id=10.1371/journal.pcbi.1007334&type=printable
    Download Restriction: no

    File URL: https://libkey.io/10.1371/journal.pcbi.1007334?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Amir Dezfouli & Bernard W Balleine, 2013. "Actions, Action Sequences and Habits: Evidence That Goal-Directed and Habitual Action Control Are Hierarchically Organized," PLOS Computational Biology, Public Library of Science, vol. 9(12), pages 1-14, December.
    2. Shiva Farashahi & Katherine Rowe & Zohra Aslami & Daeyeol Lee & Alireza Soltani, 2017. "Feature-based learning improves adaptability without compromising precision," Nature Communications, Nature, vol. 8(1), pages 1-16, December.
    3. Thomas Akam & Rui Costa & Peter Dayan, 2015. "Simple Plans or Sophisticated Habits? State, Transition and Learning Interactions in the Two-Step Task," PLOS Computational Biology, Public Library of Science, vol. 11(12), pages 1-25, December.
    4. Bates, Douglas & Mächler, Martin & Bolker, Ben & Walker, Steve, 2015. "Fitting Linear Mixed-Effects Models Using lme4," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 67(i01).
    5. Amir Dezfouli & Kristi Griffiths & Fabio Ramos & Peter Dayan & Bernard W Balleine, 2019. "Models that learn how humans learn: The case of decision-making and its disorders," PLOS Computational Biology, Public Library of Science, vol. 15(6), pages 1-33, June.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Bruno Miranda & W M Nishantha Malalasekera & Timothy E Behrens & Peter Dayan & Steven W Kennerley, 2020. "Combined model-free and model-sensitive reinforcement learning in non-human primates," PLOS Computational Biology, Public Library of Science, vol. 16(6), pages 1-25, June.
    2. Wouter Kool & Fiery A Cushman & Samuel J Gershman, 2016. "When Does Model-Based Control Pay Off?," PLOS Computational Biology, Public Library of Science, vol. 12(8), pages 1-34, August.
    3. Shiva Farashahi & Alireza Soltani, 2021. "Computational mechanisms of distributed value representations and mixed learning strategies," Nature Communications, Nature, vol. 12(1), pages 1-18, December.
    4. JANSSENS, Jochen & DE CORTE, Annelies & SÖRENSEN, Kenneth, 2016. "Water distribution network design optimisation with respect to reliability," Working Papers 2016007, University of Antwerp, Faculty of Business and Economics.
    5. Teruaki Kido & Yuko Yotsumoto & Masamichi J. Hayashi, 2025. "Hierarchical representations of relative numerical magnitudes in the human frontoparietal cortex," Nature Communications, Nature, vol. 16(1), pages 1-15, December.
    6. Raymond Hernandez & Elizabeth A. Pyatak & Cheryl L. P. Vigen & Haomiao Jin & Stefan Schneider & Donna Spruijt-Metz & Shawn C. Roll, 2021. "Understanding Worker Well-Being Relative to High-Workload and Recovery Activities across a Whole Day: Pilot Testing an Ecological Momentary Assessment Technique," IJERPH, MDPI, vol. 18(19), pages 1-17, October.
    7. Elisabeth Beckmann & Lukas Olbrich & Joseph Sakshaug, 2024. "Multivariate assessment of interviewer-related errors in a cross-national economic survey (Lukas Olbrich, Elisabeth Beckmann, Joseph W. Sakshaug)," Working Papers 253, Oesterreichische Nationalbank (Austrian Central Bank).
    8. Valentina Krenz & Arjen Alink & Tobias Sommer & Benno Roozendaal & Lars Schwabe, 2023. "Time-dependent memory transformation in hippocampus and neocortex is semantic in nature," Nature Communications, Nature, vol. 14(1), pages 1-17, December.
    9. Morán-Ordóñez, Alejandra & Ameztegui, Aitor & De Cáceres, Miquel & de-Miguel, Sergio & Lefèvre, François & Brotons, Lluís & Coll, Lluís, 2020. "Future trade-offs and synergies among ecosystem services in Mediterranean forests under global change scenarios," Ecosystem Services, Elsevier, vol. 45(C).
    10. Damian M. Herz & Manuel Bange & Gabriel Gonzalez-Escamilla & Miriam Auer & Keyoumars Ashkan & Petra Fischer & Huiling Tan & Rafal Bogacz & Muthuraman Muthuraman & Sergiu Groppa & Peter Brown, 2022. "Dynamic control of decision and movement speed in the human basal ganglia," Nature Communications, Nature, vol. 13(1), pages 1-15, December.
    11. Dongyan Liu & Chongran Zhou & John K. Keesing & Oscar Serrano & Axel Werner & Yin Fang & Yingjun Chen & Pere Masque & Janine Kinloch & Aleksey Sadekov & Yan Du, 2022. "Wildfires enhance phytoplankton production in tropical oceans," Nature Communications, Nature, vol. 13(1), pages 1-9, December.
    12. Zhaogeng Yang & Yanhui Li & Peijin Hu & Jun Ma & Yi Song, 2020. "Prevalence of Anemia and its Associated Factors among Chinese 9-, 12-, and 14-Year-Old Children: Results from 2014 Chinese National Survey on Students Constitution and Health," IJERPH, MDPI, vol. 17(5), pages 1-10, February.
    13. Marco Lopez-Cruz & Fernando M. Aguate & Jacob D. Washburn & Natalia Leon & Shawn M. Kaeppler & Dayane Cristina Lima & Ruijuan Tan & Addie Thompson & Laurence Willard Bretonne & Gustavo los Campos, 2023. "Leveraging data from the Genomes-to-Fields Initiative to investigate genotype-by-environment interactions in maize in North America," Nature Communications, Nature, vol. 14(1), pages 1-14, December.
    14. Baumann, Elias & Kern, Jana & Lessmann, Stefan, 2019. "Usage Continuance in Software-as-a-Service," IRTG 1792 Discussion Papers 2019-005, Humboldt University of Berlin, International Research Training Group 1792 "High Dimensional Nonstationary Time Series".
    15. repec:cup:judgdm:v:16:y:2021:i:1:p:201-237 is not listed on IDEAS
    16. C. Gabriel Hidalgo Pizango & Eurídice N. Honorio Coronado & Jhon del Águila-Pasquel & Gerardo Flores Llampazo & Johan de Jong & César J. Córdova Oroche & José M. Reyna Huaymacari & Steve J. Carver & D, 2022. "Sustainable palm fruit harvesting as a pathway to conserve Amazon peatland forests," Nature Sustainability, Nature, vol. 5(6), pages 479-487, June.
    17. Evans O. Mudibo & Jasper Bogaert & Caroline Tigoi & Moses M. Ngari & Benson O. Singa & Christina L. Lancioni & Abdoulaye Hama Diallo & Emmie Mbale & Ezekiel Mupere & John Mukisa & Johnstone Thitiri & , 2024. "Systemic biological mechanisms underpin poor post-discharge growth among severely wasted children with HIV," Nature Communications, Nature, vol. 15(1), pages 1-17, December.
    18. Lin-Lin Wang & Zachary Y. Huang & Wen-Fei Dai & Yong-Ping Yang & Yuan-Wen Duan, 2024. "Mixed effects of honey bees on pollination function in the Tibetan alpine grasslands," Nature Communications, Nature, vol. 15(1), pages 1-12, December.
    19. Szefer Elena & Lu Donghuan & Nathoo Farouk & Beg Mirza Faisal & Graham Jinko, 2017. "Multivariate association between single-nucleotide polymorphisms in Alzgene linkage regions and structural changes in the brain: discovery, refinement and validation," Statistical Applications in Genetics and Molecular Biology, De Gruyter, vol. 16(5-6), pages 367-386, December.
    20. Julien Collet & Samantha C Patrick & Henri Weimerskirch, 2017. "A comparative analysis of the behavioral response to fishing boats in two albatross species," Behavioral Ecology, International Society for Behavioral Ecology, vol. 28(5), pages 1337-1347.
    21. Sean Coogan & Zhixian Sui & David Raubenheimer, 2018. "Gluttony and guilt: monthly trends in internet search query data are comparable with national-level energy intake and dieting behavior," Palgrave Communications, Palgrave Macmillan, vol. 4(1), pages 1-9, December.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pcbi00:1007334. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: ploscompbiol (email available below). General contact details of provider: https://journals.plos.org/ploscompbiol/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.