IDEAS home Printed from https://ideas.repec.org/a/plo/pcbi00/1007963.html
   My bibliography  Save this article

A simple model for learning in volatile environments

Author

Listed:
  • Payam Piray
  • Nathaniel D Daw

Abstract

Sound principles of statistical inference dictate that uncertainty shapes learning. In this work, we revisit the question of learning in volatile environments, in which both the first and second-order statistics of observations dynamically evolve over time. We propose a new model, the volatile Kalman filter (VKF), which is based on a tractable state-space model of uncertainty and extends the Kalman filter algorithm to volatile environments. The proposed model is algorithmically simple and encompasses the Kalman filter as a special case. Specifically, in addition to the error-correcting rule of Kalman filter for learning observations, the VKF learns volatility according to a second error-correcting rule. These dual updates echo and contextualize classical psychological models of learning, in particular hybrid accounts of Pearce-Hall and Rescorla-Wagner. At the computational level, compared with existing models, the VKF gives up some flexibility in the generative model to enable a more faithful approximation to exact inference. When fit to empirical data, the VKF is better behaved than alternatives and better captures human choice data in two independent datasets of probabilistic learning tasks. The proposed model provides a coherent account of learning in stable or volatile environments and has implications for decision neuroscience research.Author summary: Sound principles of statistical learning dictate that uncertainty influences behavior. However, despite the success of statistically founded algorithms for learning in stable environments, in which uncertainty behaves in simple and predictable ways, it is challenging to develop a simple yet efficient algorithm for learning in volatile environments, in which uncertainty dynamically changes over time. In this article, we develop a model for learning in volatile environments. The proposed model is consistent with key concepts of classical learning theories from behavioral psychology. Furthermore, our model is algorithmically simpler, theoretically more accurate, and empirically more parsimonious than the state-of-the-art models of learning in volatile environments. The proposed model provides a coherent theory of learning under uncertainty.

Suggested Citation

  • Payam Piray & Nathaniel D Daw, 2020. "A simple model for learning in volatile environments," PLOS Computational Biology, Public Library of Science, vol. 16(7), pages 1-26, July.
  • Handle: RePEc:plo:pcbi00:1007963
    DOI: 10.1371/journal.pcbi.1007963
    as

    Download full text from publisher

    File URL: https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1007963
    Download Restriction: no

    File URL: https://journals.plos.org/ploscompbiol/article/file?id=10.1371/journal.pcbi.1007963&type=printable
    Download Restriction: no

    File URL: https://libkey.io/10.1371/journal.pcbi.1007963?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Kiyohito Iigaya & Madalena S. Fonseca & Masayoshi Murakami & Zachary F. Mainen & Peter Dayan, 2018. "An effect of serotonergic stimulation on learning rates for rewards apparent after long intertrial intervals," Nature Communications, Nature, vol. 9(1), pages 1-10, December.
    2. Payam Piray & Amir Dezfouli & Tom Heskes & Michael J Frank & Nathaniel D Daw, 2019. "Hierarchical Bayesian inference for concurrent model fitting and comparison for group studies," PLOS Computational Biology, Public Library of Science, vol. 15(6), pages 1-34, June.
    3. Elise Payzan-LeNestour & Peter Bossaerts, 2011. "Risk, Unexpected Uncertainty, and Estimation Uncertainty: Bayesian Learning in Unstable Settings," PLOS Computational Biology, Public Library of Science, vol. 7(1), pages 1-14, January.
    4. Archy O. de Berker & Robb B. Rutledge & Christoph Mathys & Louise Marshall & Gemma F. Cross & Raymond J. Dolan & Sven Bestmann, 2016. "Computations of uncertainty mediate acute stress responses in humans," Nature Communications, Nature, vol. 7(1), pages 1-11, April.
    5. Dani Gamerman & Thiago Rezende Santos & Glaura C. Franco, 2013. "A Non-Gaussian Family Of State-Space Models With Exact Marginal Likelihood," Journal of Time Series Analysis, Wiley Blackwell, vol. 34(6), pages 625-645, November.
    6. Peyman Khorsand & Alireza Soltani, 2017. "Optimal structure of metaplasticity for adaptive learning," PLOS Computational Biology, Public Library of Science, vol. 13(6), pages 1-22, June.
    7. Timothy E. J. Behrens & Laurence T. Hunt & Mark W. Woolrich & Matthew F. S. Rushworth, 2008. "Associative learning of social value," Nature, Nature, vol. 456(7219), pages 245-249, November.
    8. Anthony I. Jang & Matthew R. Nassar & Daniel G. Dillon & Michael J. Frank, 2019. "Positive reward prediction errors during decision-making strengthen memory encoding," Nature Human Behaviour, Nature, vol. 3(7), pages 719-732, July.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Payam Piray & Nathaniel D. Daw, 2021. "A model for learning based on the joint estimation of stochasticity and volatility," Nature Communications, Nature, vol. 12(1), pages 1-16, December.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Payam Piray & Nathaniel D. Daw, 2021. "A model for learning based on the joint estimation of stochasticity and volatility," Nature Communications, Nature, vol. 12(1), pages 1-16, December.
    2. Micha Heilbron & Florent Meyniel, 2019. "Confidence resets reveal hierarchical adaptive learning in humans," PLOS Computational Biology, Public Library of Science, vol. 15(4), pages 1-24, April.
    3. Daniel S Kluger & Nico Broers & Marlen A Roehe & Moritz F Wurm & Niko A Busch & Ricarda I Schubotz, 2020. "Exploitation of local and global information in predictive processing," PLOS ONE, Public Library of Science, vol. 15(4), pages 1-17, April.
    4. Caroline J. Charpentier & Qianying Wu & Seokyoung Min & Weilun Ding & Jeffrey Cockburn & John P. O’Doherty, 2024. "Heterogeneity in strategy use during arbitration between experiential and observational learning," Nature Communications, Nature, vol. 15(1), pages 1-20, December.
    5. Corgnet, Brice & Hernán-González, Roberto & Kujal, Praveen, 2020. "On booms that never bust: Ambiguity in experimental asset markets with bubbles," Journal of Economic Dynamics and Control, Elsevier, vol. 110(C).
    6. Jacqueline Scholl & Nils Kolling & Natalie Nelissen & Michael Browning & Matthew F S Rushworth & Catherine J Harmer, 2017. "Beyond negative valence: 2-week administration of a serotonergic antidepressant enhances both reward and effort learning signals," PLOS Biology, Public Library of Science, vol. 15(2), pages 1-30, February.
    7. repec:cup:judgdm:v:16:y:2021:i:6:p:1413-1438 is not listed on IDEAS
    8. Jacqueline N. Zadelaar & Joost A. Agelink van Rentergem & Jessica V. Schaaf & Tycho J. Dekkers & Nathalie de Vent & Laura M. S. Dekkers & Maria C. Olthof & Brenda R. J. Jansen & Hilde M. Huizenga, 2021. "Development of decision making based on internal and external information: A hierarchical Bayesian approach," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 16(6), pages 1413-1438, November.
    9. Holger Mohr & Katharina Zwosta & Dimitrije Markovic & Sebastian Bitzer & Uta Wolfensteller & Hannes Ruge, 2018. "Deterministic response strategies in a trial-and-error learning task," PLOS Computational Biology, Public Library of Science, vol. 14(11), pages 1-19, November.
    10. Seetha Krishnan & Chad Heer & Chery Cherian & Mark E. J. Sheffield, 2022. "Reward expectation extinction restructures and degrades CA1 spatial maps through loss of a dopaminergic reward proximity signal," Nature Communications, Nature, vol. 13(1), pages 1-19, December.
    11. Carolyn Yoon & Richard Gonzalez & Antoine Bechara & Gregory Berns & Alain Dagher & Laurette Dubé & Scott Huettel & Joseph Kable & Israel Liberzon & Hilke Plassmann & Ale Smidts & Charles Spence, 2012. "Decision neuroscience and consumer decision making," Marketing Letters, Springer, vol. 23(2), pages 473-485, June.
    12. Hu, Yingyao & Kayaba, Yutaka & Shum, Matthew, 2013. "Nonparametric learning rules from bandit experiments: The eyes have it!," Games and Economic Behavior, Elsevier, vol. 81(C), pages 215-231.
    13. Mateus Joffily & Giorgio Coricelli, 2013. "Emotional Valence and the Free-Energy Principle," Post-Print halshs-00834063, HAL.
    14. Andreea O Diaconescu & Christoph Mathys & Lilian A E Weber & Jean Daunizeau & Lars Kasper & Ekaterina I Lomakina & Ernst Fehr & Klaas E Stephan, 2014. "Inferring on the Intentions of Others by Hierarchical Bayesian Learning," PLOS Computational Biology, Public Library of Science, vol. 10(9), pages 1-19, September.
    15. Duca, Victor E.L.A. & Fonseca, Thaís C.O. & Cyrino Oliveira, Fernando Luiz, 2023. "An overview of non-Gaussian state-space models for wind speed data," Energy, Elsevier, vol. 266(C).
    16. Dimitrije Marković & Andrea M F Reiter & Stefan J Kiebel, 2019. "Predicting change: Approximate inference under explicit representation of temporal structure in changing environments," PLOS Computational Biology, Public Library of Science, vol. 15(1), pages 1-31, January.
    17. Dan Bang & Rani Moran & Nathaniel D. Daw & Stephen M. Fleming, 2022. "Neurocomputational mechanisms of confidence in self and others," Nature Communications, Nature, vol. 13(1), pages 1-14, December.
    18. Elizabeth Lomas & Julie McLeod, 2017. "Engaging with change: Information and communication technology professionals’ perspectives on change in the context of the ‘Brexit’ vote," PLOS ONE, Public Library of Science, vol. 12(11), pages 1-26, November.
    19. George I. Christopoulos & Xiao-Xiao Liu & Ying-yi Hong, 2017. "Toward an Understanding of Dynamic Moral Decision Making: Model-Free and Model-Based Learning," Journal of Business Ethics, Springer, vol. 144(4), pages 699-715, September.
    20. Damon Tomlin & Andrea Nedic & Deborah A Prentice & Philip Holmes & Jonathan D Cohen, 2013. "The Neural Substrates of Social Influence on Decision Making," PLOS ONE, Public Library of Science, vol. 8(1), pages 1-11, January.
    21. Joseph Simon IV & Erin L. Rich, 2024. "Neural populations in macaque anterior cingulate cortex encode social image identities," Nature Communications, Nature, vol. 15(1), pages 1-11, December.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pcbi00:1007963. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: ploscompbiol (email available below). General contact details of provider: https://journals.plos.org/ploscompbiol/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.