IDEAS home Printed from https://ideas.repec.org/a/plo/pone00/0007362.html
   My bibliography  Save this article

Temporal-Difference Reinforcement Learning with Distributed Representations

Author

Listed:
  • Zeb Kurth-Nelson
  • A David Redish

Abstract

Temporal-difference (TD) algorithms have been proposed as models of reinforcement learning (RL). We examine two issues of distributed representation in these TD algorithms: distributed representations of belief and distributed discounting factors. Distributed representation of belief allows the believed state of the world to distribute across sets of equivalent states. Distributed exponential discounting factors produce hyperbolic discounting in the behavior of the agent itself. We examine these issues in the context of a TD RL model in which state-belief is distributed over a set of exponentially-discounting “micro-Agents”, each of which has a separate discounting factor (γ). Each µAgent maintains an independent hypothesis about the state of the world, and a separate value-estimate of taking actions within that hypothesized state. The overall agent thus instantiates a flexible representation of an evolving world-state. As with other TD models, the value-error (δ) signal within the model matches dopamine signals recorded from animals in standard conditioning reward-paradigms. The distributed representation of belief provides an explanation for the decrease in dopamine at the conditioned stimulus seen in overtrained animals, for the differences between trace and delay conditioning, and for transient bursts of dopamine seen at movement initiation. Because each µAgent also includes its own exponential discounting factor, the overall agent shows hyperbolic discounting, consistent with behavioral experiments.

Suggested Citation

  • Zeb Kurth-Nelson & A David Redish, 2009. "Temporal-Difference Reinforcement Learning with Distributed Representations," PLOS ONE, Public Library of Science, vol. 4(10), pages 1-19, October.
  • Handle: RePEc:plo:pone00:0007362
    DOI: 10.1371/journal.pone.0007362
    as

    Download full text from publisher

    File URL: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0007362
    Download Restriction: no

    File URL: https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0007362&type=printable
    Download Restriction: no

    File URL: https://libkey.io/10.1371/journal.pone.0007362?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Paul E. M. Phillips & Garret D. Stuber & Michael L. A. V. Heien & R. Mark Wightman & Regina M. Carelli, 2003. "Subsecond dopamine release promotes cocaine seeking," Nature, Nature, vol. 422(6932), pages 614-618, April.
    2. Tapas K. Das & Abhijit Gosavi & Sridhar Mahadevan & Nicholas Marchalleck, 1999. "Solving Semi-Markov Decision Problems Using Average Reward Reinforcement Learning," Management Science, INFORMS, vol. 45(4), pages 560-574, April.
    3. Read, Daniel, 2001. "Is Time-Discounting Hyperbolic or Subadditive?," Journal of Risk and Uncertainty, Springer, vol. 23(1), pages 5-32, July.
    4. David Self, 2003. "Dopamine as chicken and egg," Nature, Nature, vol. 422(6932), pages 573-574, April.
    5. Nancy M. Petry & Warren K. Bickel, 1998. "A Behavioral Economic Analysis of Polydrug Abuse in Heroin Addicts," NBER Working Papers 6415, National Bureau of Economic Research, Inc.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Lovric, M. & Kaymak, U. & Spronk, J., 2008. "A Conceptual Model of Investor Behavior," ERIM Report Series Research in Management ERS-2008-030-F&A, Erasmus Research Institute of Management (ERIM), ERIM is the joint research institute of the Rotterdam School of Management, Erasmus University and the Erasmus School of Economics (ESE) at Erasmus University Rotterdam.
    2. Tam, Leona & Dholakia, Utpal M., 2011. "Delay and duration effects of time frames on personal savings estimates and behavior," Organizational Behavior and Human Decision Processes, Elsevier, vol. 114(2), pages 142-152, March.
    3. Yan, Yimo & Deng, Yang & Cui, Songyi & Kuo, Yong-Hong & Chow, Andy H.F. & Ying, Chengshuo, 2023. "A policy gradient approach to solving dynamic assignment problem for on-site service delivery," Transportation Research Part E: Logistics and Transportation Review, Elsevier, vol. 178(C).
    4. Stefania Albanesi & Claudia Olivetti, 2006. "Gender roles and technological progress," 2006 Meeting Papers 411, Society for Economic Dynamics.
    5. Uri Ben-Zion & Jan Pieter Krahnen & TAL SHAVIT, 2007. "Subjective Evaluation Of Delayed Risky Outcomes: An Experimental Approach," Working Papers 0709, Ben-Gurion University of the Negev, Department of Economics.
    6. Khwaja, Ahmed & Silverman, Dan & Sloan, Frank, 2007. "Time preference, time discounting, and smoking decisions," Journal of Health Economics, Elsevier, vol. 26(5), pages 927-949, September.
    7. He, Haonan & Wang, Shanyong, 2019. "Cost-benefit associations in consumer inventory problem with uncertain benefit," Journal of Retailing and Consumer Services, Elsevier, vol. 51(C), pages 271-284.
    8. Alekseev, Aleksandr & Sokolov, Mikhail V., 2021. "How to measure the average rate of change?," Mathematical Social Sciences, Elsevier, vol. 113(C), pages 43-59.
    9. Andrew Meyer, 2013. "Estimating discount factors for public and private goods and testing competing discounting hypotheses," Journal of Risk and Uncertainty, Springer, vol. 46(2), pages 133-173, April.
    10. Jofre-Bonet, Mireia & Petry, Nancy M., 2008. "Trading apples for oranges?: Results of an experiment on the effects of Heroin and Cocaine price changes on addicts' polydrug use," Journal of Economic Behavior & Organization, Elsevier, vol. 66(2), pages 281-311, May.
    11. Epper, Thomas & Fehr-Duda, Helga, 2017. "A Tale of Two Tails: On the Coexistence of Overweighting and Underweighting of Rare Extreme Events," Economics Working Paper Series 1705, University of St. Gallen, School of Economics and Political Science.
    12. repec:ubc:pmicro:halevy-04-10-29-10-08-43 is not listed on IDEAS
    13. Marzilli Ericson, K. M. & White, J. M. & Laibson, David I. & Cohen, J. D., 2015. "Money Earlier or Later? Simple Heuristics Explain Intertemporal Choices Better Than Delay Discounting Does," Scholarly Articles 30367415, Harvard University Department of Economics.
    14. Kai Ruggeri & Amma Panin & Milica Vdovic & Bojana Većkalov & Nazeer Abdul-Salaam & Jascha Achterberg & Carla Akil & Jolly Amatya & Kanchan Amatya & Thomas Lind Andersen & Sibele D. Aquino & Arjoon Aru, 2022. "The globalizability of temporal discounting," Nature Human Behaviour, Nature, vol. 6(10), pages 1386-1397, October.
    15. repec:cup:judgdm:v:8:y:2013:i:2:p:116-135 is not listed on IDEAS
    16. Ohno, Katsuhisa & Boh, Toshitaka & Nakade, Koichi & Tamura, Takayoshi, 2016. "New approximate dynamic programming algorithms for large-scale undiscounted Markov decision processes and their application to optimize a production and distribution system," European Journal of Operational Research, Elsevier, vol. 249(1), pages 22-31.
    17. Kontek, Krzysztof, 2010. "Linking Decision and Time Utilities," MPRA Paper 27541, University Library of Munich, Germany.
    18. Wang, Mei & Rieger, Marc Oliver & Hens, Thorsten, 2016. "How time preferences differ: Evidence from 53 countries," Journal of Economic Psychology, Elsevier, vol. 52(C), pages 115-135.
    19. Craig A. Gallet, 2014. "Can Price Get The Monkey Off Our Back? A Meta‐Analysis Of Illicit Drug Demand," Health Economics, John Wiley & Sons, Ltd., vol. 23(1), pages 55-68, January.
    20. Read, Daniel & Roelofsma, Peter H. M. P., 2003. "Subadditive versus hyperbolic discounting: A comparison of choice and matching," Organizational Behavior and Human Decision Processes, Elsevier, vol. 91(2), pages 140-153, July.
    21. Cruz Rambaud, Salvador & Parra Oller, Isabel María & Valls Martínez, María del Carmen, 2018. "The amount-based deformation of the q-exponential discount function: A joint analysis of delay and magnitude effects," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 508(C), pages 788-796.
    22. Destefano, Natália & Martinez, Alexandre Souto, 2011. "The additive property of the inconsistency degree in intertemporal decision making through the generalization of psychophysical laws," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 390(10), pages 1763-1772.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pone00:0007362. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: plosone (email available below). General contact details of provider: https://journals.plos.org/plosone/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.