IDEAS home Printed from https://ideas.repec.org/a/spr/mathme/v97y2023i1d10.1007_s00186-022-00806-9.html
   My bibliography  Save this article

An axiomatic approach to Markov decision processes

Author

Listed:
  • Adam Jonsson

    (Luleå University of Technology)

Abstract

This paper presents an axiomatic approach to finite Markov decision processes where the discount rate is zero. One of the principal difficulties in the no discounting case is that, even if attention is restricted to stationary policies, a strong overtaking optimal policy need not exists. We provide preference foundations for two criteria that do admit optimal policies: 0-discount optimality and average overtaking optimality. As a corollary of our results, we obtain conditions on a decision maker’s preferences which ensure that an optimal policy exists. These results have implications for disciplines where dynamic programming problems arise, including automatic control, dynamic games, and economic development.

Suggested Citation

  • Adam Jonsson, 2023. "An axiomatic approach to Markov decision processes," Mathematical Methods of Operations Research, Springer;Gesellschaft für Operations Research (GOR);Nederlands Genootschap voor Besliskunde (NGB), vol. 97(1), pages 117-133, February.
    • Handle: RePEc:spr:mathme:v:97:y:2023:i:1:d:10.1007_s00186-022-00806-9
    DOI: 10.1007/s00186-022-00806-9
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s00186-022-00806-9
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s00186-022-00806-9?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Kaushik Basu & Tapan Mitra, 2003. "Aggregating Infinite Utility Streams with InterGenerational Equity: The Impossibility of Being Paretian," Econometrica, Econometric Society, vol. 71(5), pages 1557-1563, September.
    2. Fleurbaey, Marc & Michel, Philippe, 2003. "Intertemporal equity and the extension of the Ramsey criterion," Journal of Mathematical Economics, Elsevier, vol. 39(7), pages 777-802, September.
    3. Dinah Rosenberg & Eilon Solan & Nicolas Vieille, 2000. "Blackwell Optimality in Markov Decision Processes with Partial Observation," Discussion Papers 1292, Northwestern University, Center for Mathematical Studies in Economics and Management Science.
    4. Jonsson, Adam & Voorneveld, Mark, 2018. "The limit of discounted utilitarianism," Theoretical Economics, Econometric Society, vol. 13(1), January.
    5. Asheim, Geir B. & d'Aspremont, Claude & Banerjee, Kuntal, 2010. "Generalized time-invariant overtaking," Journal of Mathematical Economics, Elsevier, vol. 46(4), pages 519-533, July.
    6. David Gale, 1967. "On Optimal Development in a Multi-Sector Economy," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 34(1), pages 1-18.
    7. Nicole Bäuerle & Ulrich Rieder, 2014. "More Risk-Sensitive Markov Decision Processes," Mathematics of Operations Research, INFORMS, vol. 39(1), pages 105-120, February.
    8. Claude D'Aspremont & Louis Gevers, 1977. "Equity and the Informational Basis of Collective Choice," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 44(2), pages 199-209.
    9. Geir Asheim & Bertil Tungodden, 2004. "Resolving distributional conflicts between generations," Economic Theory, Springer;Society for the Advancement of Economic Theory (SAET), vol. 24(1), pages 221-230, July.
    10. Adam Jonsson & Mark Voorneveld, 2015. "Utilitarianism on infinite utility streams: summable differences and finite averages," Economic Theory Bulletin, Springer;Society for the Advancement of Economic Theory (SAET), vol. 3(1), pages 19-31, April.
    11. Steven A. Lippman, 1969. "Letter to the Editor—Criterion Equivalence in Discrete Dynamic Programming," Operations Research, INFORMS, vol. 17(5), pages 920-922, October.
    12. János Flesch & Arkadi Predtetchinski & Eilon Solan, 2017. "Sporadic Overtaking Optimality in Markov Decision Problems," Dynamic Games and Applications, Springer, vol. 7(2), pages 212-228, June.
    13. W. A. Brock, 1970. "On Existence of Weakly Maximal Programmes in a Multi-Sector Economy," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 37(2), pages 275-280.
    14. Marinacci, Massimo, 1998. "An Axiomatic Approach to Complete Patience and Time Invariance," Journal of Economic Theory, Elsevier, vol. 83(1), pages 105-144, November.
    15. Amit Kothiyal & Vitalie Spinu & Peter P. Wakker, 2014. "Average Utility Maximization: A Preference Foundation," Operations Research, INFORMS, vol. 62(1), pages 207-218, February.
    16. Pivato, Marcus, 2022. "A characterization of Cesàro average utility," Journal of Economic Theory, Elsevier, vol. 201(C).
    17. Andrzej S. Nowak & Oscar Vega-Amaya, 1999. "A counterexample on overtaking optimality," Mathematical Methods of Operations Research, Springer;Gesellschaft für Operations Research (GOR);Nederlands Genootschap voor Besliskunde (NGB), vol. 49(3), pages 435-439, July.
    18. Lauwers, Luc, 2010. "Ordering infinite utility streams comes at the cost of a non-Ramsey set," Journal of Mathematical Economics, Elsevier, vol. 46(1), pages 32-37, January.
    19. Basu, Kaushik & Mitra, Tapan, 2007. "Utilitarianism for infinite utility streams: A new welfare criterion and its axiomatic characterization," Journal of Economic Theory, Elsevier, vol. 133(1), pages 350-373, March.
    20. Khan, Urmee & Stinchcombe, Maxwell B., 2018. "Planning for the long run: Programming with patient, Pareto responsive preferences," Journal of Economic Theory, Elsevier, vol. 176(C), pages 444-478.
    21. Brock, William A, 1970. "An Axiomatic Basis for the Ramsey- Weizsacker Overtaking Criterion," Econometrica, Econometric Society, vol. 38(6), pages 927-929, November.
    22. Brock, William A & Mirman, Leonard J, 1973. "Optimal Economic Growth and Uncertainty: The No Discounting Case," International Economic Review, Department of Economics, University of Pennsylvania and Osaka University Institute of Social and Economic Research Association, vol. 14(3), pages 560-573, October.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Jonsson, Adam & Voorneveld, Mark, 2018. "The limit of discounted utilitarianism," Theoretical Economics, Econometric Society, vol. 13(1), January.
    2. Adam Jonsson & Mark Voorneveld, 2015. "Utilitarianism on infinite utility streams: summable differences and finite averages," Economic Theory Bulletin, Springer;Society for the Advancement of Economic Theory (SAET), vol. 3(1), pages 19-31, April.
    3. Jonsson, Adam & Voorneveld, Mark, 2014. "Utilitarianism for infinite utility streams: summable differences and finite averages," SSE/EFI Working Paper Series in Economics and Finance 747, Stockholm School of Economics, revised 15 Apr 2014.
    4. Alain Ayong Le Kama & Thai Ha-Huy & Cuong Le Van & Katheline Schubert, 2014. "A never-decisive and anonymous criterion for optimal growth models," Economic Theory, Springer;Society for the Advancement of Economic Theory (SAET), vol. 55(2), pages 281-306, February.
    5. Geir B. Asheim & Kuntal Banerjee, 2010. "Fixed‐step anonymous overtaking and catching‐up," International Journal of Economic Theory, The International Society for Economic Theory, vol. 6(1), pages 149-165, March.
    6. Marcus Pivato, 2014. "Additive representation of separable preferences over infinite products," Theory and Decision, Springer, vol. 77(1), pages 31-83, June.
    7. Kohei Kamaga, 2016. "Infinite-horizon social evaluation with variable population size," Social Choice and Welfare, Springer;The Society for Social Choice and Welfare, vol. 47(1), pages 207-232, June.
    8. Li, Chen & Wakker, Peter P., 2024. "A simple and general axiomatization of average utility maximization for infinite streams," Journal of Economic Theory, Elsevier, vol. 216(C).
    9. Geir Asheim & Stéphane Zuber, 2013. "A complete and strongly anonymous leximin relation on infinite streams," Social Choice and Welfare, Springer;The Society for Social Choice and Welfare, vol. 41(4), pages 819-834, October.
    10. repec:ipg:wpaper:2 is not listed on IDEAS
    11. repec:hal:pseose:hal-01044568 is not listed on IDEAS
    12. repec:ipg:wpaper:2013-002 is not listed on IDEAS
    13. Asheim, Geir B. & d'Aspremont, Claude & Banerjee, Kuntal, 2010. "Generalized time-invariant overtaking," Journal of Mathematical Economics, Elsevier, vol. 46(4), pages 519-533, July.
    14. Kohei Kamaga & Takashi Kojima, 2010. "On the leximin and utilitarian overtaking criteria with extended anonymity," Social Choice and Welfare, Springer;The Society for Social Choice and Welfare, vol. 35(3), pages 377-392, September.
    15. Alvarez-Cuadrado, Francisco & Van Long, Ngo, 2009. "A mixed Bentham-Rawls criterion for intergenerational equity: Theory and implications," Journal of Environmental Economics and Management, Elsevier, vol. 58(2), pages 154-168, September.
    16. Bach Dong-Xuan & Philippe Bich, 2024. "Dynamic choices, temporal invariance and variational discounting," Papers 2408.05632, arXiv.org.
    17. Basu, Kaushik & Mitra, Tapan, 2007. "Utilitarianism for infinite utility streams: A new welfare criterion and its axiomatic characterization," Journal of Economic Theory, Elsevier, vol. 133(1), pages 350-373, March.
    18. Pivato, Marcus, 2022. "A characterization of Cesàro average utility," Journal of Economic Theory, Elsevier, vol. 201(C).
    19. Michele Lombardi & Kaname Miyagishima & Roberto Veneziani, 2016. "Liberal Egalitarianism and the Harm Principle," Economic Journal, Royal Economic Society, vol. 126(597), pages 2173-2196, November.
    20. Geir B. Asheim & Kuntal Banerjee & Tapan Mitra, 2021. "How stationarity contradicts intergenerational equity," Economic Theory, Springer;Society for the Advancement of Economic Theory (SAET), vol. 72(2), pages 423-444, September.
    21. Alain Ayong Le Kama & Cuong Le Van & Katheline Schubert, 2008. "A Non-dictatorial Criterion for Optimal Growth Models," Working Papers 14, Development and Policies Research Center (DEPOCEN), Vietnam.
    22. Claude, d’ASPREMONT, 2005. "Formal welfarism and intergenerational equity," Discussion Papers (ECON - Département des Sciences Economiques) 2005051, Université catholique de Louvain, Département des Sciences Economiques.
    23. Toyotaka Sakai, 2016. "Limit representations of intergenerational equity," Social Choice and Welfare, Springer;The Society for Social Choice and Welfare, vol. 47(2), pages 481-500, August.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:mathme:v:97:y:2023:i:1:d:10.1007_s00186-022-00806-9. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.