IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2010.07220.html
   My bibliography  Save this paper

Markov Decision Processes with Recursive Risk Measures

Author

Listed:
  • Nicole Bauerle
  • Alexander Glauner

Abstract

In this paper, we consider risk-sensitive Markov Decision Processes (MDPs) with Borel state and action spaces and unbounded cost under both finite and infinite planning horizons. Our optimality criterion is based on the recursive application of static risk measures. This is motivated by recursive utilities in the economic literature, has been studied before for the entropic risk measure and is extended here to an axiomatic characterization of suitable risk measures. We derive a Bellman equation and prove the existence of Markovian optimal policies. For an infinite planning horizon, the model is shown to be contractive and the optimal policy to be stationary. Moreover, we establish a connection to distributionally robust MDPs, which provides a global interpretation of the recursively defined objective function. Monotone models are studied in particular.

Suggested Citation

  • Nicole Bauerle & Alexander Glauner, 2020. "Markov Decision Processes with Recursive Risk Measures," Papers 2010.07220, arXiv.org.
    • Handle: RePEc:arx:papers:2010.07220
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2010.07220
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Homem-de-Mello, Tito & Pagnoncelli, Bernardo K., 2016. "Risk aversion in multistage stochastic programming: A modeling and algorithmic perspective," European Journal of Operational Research, Elsevier, vol. 249(1), pages 188-199.
    2. Riedel, Frank, 2004. "Dynamic coherent risk measures," Stochastic Processes and their Applications, Elsevier, vol. 112(2), pages 185-200, August.
    3. Epstein, Larry G. & Schneider, Martin, 2003. "Recursive multiple-priors," Journal of Economic Theory, Elsevier, vol. 113(1), pages 1-31, November.
    4. Tomasz R. Bielecki & Igor Cialenco & Marcin Pitera, 2018. "A Unified Approach to Time Consistency of Dynamic Risk Measures and Dynamic Performance Measures in Discrete Time," Mathematics of Operations Research, INFORMS, vol. 43(1), pages 204-221, February.
    5. Schur, Rouven & Gönsch, Jochen & Hassler, Michael, 2019. "Time-consistent, risk-averse dynamic pricing," European Journal of Operational Research, Elsevier, vol. 277(2), pages 587-603.
    6. Detlefsen, Kai & Scandolo, Giacomo, 2005. "Conditional and dynamic convex risk measures," SFB 649 Discussion Papers 2005-006, Humboldt University Berlin, Collaborative Research Center 649: Economic Risk.
    7. Kai Detlefsen & Giacomo Scandolo, 2005. "Conditional and dynamic convex risk measures," Finance and Stochastics, Springer, vol. 9(4), pages 539-561, October.
    8. Shapiro, Alexander, 2012. "Minimax and risk averse multistage stochastic programming," European Journal of Operational Research, Elsevier, vol. 219(3), pages 719-726.
    9. Bäuerle, Nicole & Glauner, Alexander, 2018. "Optimal risk allocation in reinsurance networks," Insurance: Mathematics and Economics, Elsevier, vol. 82(C), pages 37-47.
    10. Kreps, David M & Porteus, Evan L, 1978. "Temporal Resolution of Uncertainty and Dynamic Choice Theory," Econometrica, Econometric Society, vol. 46(1), pages 185-200, January.
    11. Pichler, Alois, 2013. "The natural Banach space for version independent risk measures," Insurance: Mathematics and Economics, Elsevier, vol. 53(2), pages 405-415.
    12. Charalambos D. Aliprantis & Kim C. Border, 2006. "Infinite Dimensional Analysis," Springer Books, Springer, edition 0, number 978-3-540-29587-7, February.
    13. Bion-Nadal, Jocelyne, 2009. "Time consistent dynamic risk processes," Stochastic Processes and their Applications, Elsevier, vol. 119(2), pages 633-654, February.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Nicole Bauerle & Alexander Glauner, 2020. "Minimizing Spectral Risk Measures Applied to Markov Decision Processes," Papers 2012.04521, arXiv.org.
    2. Alexander Glauner, 2020. "Dynamic Reinsurance in Discrete Time Minimizing the Insurer's Cost of Capital," Papers 2012.09648, arXiv.org.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Bäuerle, Nicole & Glauner, Alexander, 2022. "Markov decision processes with recursive risk measures," European Journal of Operational Research, Elsevier, vol. 296(3), pages 953-966.
    2. Dan A. Iancu & Marek Petrik & Dharmashankar Subramanian, 2015. "Tight Approximations of Dynamic Risk Measures," Mathematics of Operations Research, INFORMS, vol. 40(3), pages 655-682, March.
    3. De Lara, Michel & Leclère, Vincent, 2016. "Building up time-consistency for risk measures and dynamic optimization," European Journal of Operational Research, Elsevier, vol. 249(1), pages 177-187.
    4. Bellini, Fabio & Laeven, Roger J.A. & Rosazza Gianin, Emanuela, 2021. "Dynamic robust Orlicz premia and Haezendonck–Goovaerts risk measures," European Journal of Operational Research, Elsevier, vol. 291(2), pages 438-446.
    5. Henri Gérard & Michel Lara & Jean-Philippe Chancelier, 2020. "Equivalence between time consistency and nested formula," Annals of Operations Research, Springer, vol. 292(2), pages 627-647, September.
    6. Cheridito, Patrick & Stadje, Mitja, 2009. "Time-inconsistency of VaR and time-consistent alternatives," Finance Research Letters, Elsevier, vol. 6(1), pages 40-46, March.
    7. Ji, Ronglin & Shi, Xuejun & Wang, Shijie & Zhou, Jinming, 2019. "Dynamic risk measures for processes via backward stochastic differential equations," Insurance: Mathematics and Economics, Elsevier, vol. 86(C), pages 43-50.
    8. Zachary Feinstein & Birgit Rudloff, 2018. "Scalar multivariate risk measures with a single eligible asset," Papers 1807.10694, arXiv.org, revised Feb 2021.
    9. Elisa Mastrogiacomo & Emanuela Rosazza Gianin, 2019. "Time-consistency of risk measures: how strong is such a property?," Decisions in Economics and Finance, Springer;Associazione per la Matematica, vol. 42(1), pages 287-317, June.
    10. Chen, Zhiping & Li, Gang & Zhao, Yonggan, 2014. "Time-consistent investment policies in Markovian markets: A case of mean–variance analysis," Journal of Economic Dynamics and Control, Elsevier, vol. 40(C), pages 293-316.
    11. Acciaio, Beatrice & Föllmer, Hans & Penner, Irina, 2012. "Risk assessment for uncertain cash flows: model ambiguity, discounting ambiguity, and the role of bubbles," LSE Research Online Documents on Economics 50118, London School of Economics and Political Science, LSE Library.
    12. Zachary Feinstein & Birgit Rudloff, 2012. "Multiportfolio time consistency for set-valued convex and coherent risk measures," Papers 1212.5563, arXiv.org, revised Oct 2014.
    13. Chen, Zhi-ping & Li, Gang & Guo, Ju-e, 2013. "Optimal investment policy in the time consistent mean–variance formulation," Insurance: Mathematics and Economics, Elsevier, vol. 52(2), pages 145-156.
    14. Zachary Feinstein & Birgit Rudloff, 2018. "Time consistency for scalar multivariate risk measures," Papers 1810.04978, arXiv.org, revised Nov 2021.
    15. Zachary Feinstein & Birgit Rudloff, 2015. "Multi-portfolio time consistency for set-valued convex and coherent risk measures," Finance and Stochastics, Springer, vol. 19(1), pages 67-107, January.
    16. Zhiping Chen & Jia Liu & Gang Li & Zhe Yan, 2016. "Composite time-consistent multi-period risk measure and its application in optimal portfolio selection," TOP: An Official Journal of the Spanish Society of Statistics and Operations Research, Springer;Sociedad de Estadística e Investigación Operativa, vol. 24(3), pages 515-540, October.
    17. Andreas H Hamel, 2018. "Monetary Measures of Risk," Papers 1812.04354, arXiv.org.
    18. Xin, Linwei & Goldberg, David A., 2021. "Time (in)consistency of multistage distributionally robust inventory models with moment constraints," European Journal of Operational Research, Elsevier, vol. 289(3), pages 1127-1141.
    19. Claudia Kluppelberg & Jianing Zhang, 2015. "Time-consistency of risk measures with GARCH volatilities and their estimation," Papers 1504.04774, arXiv.org, revised Feb 2016.
    20. Zachary Feinstein & Birgit Rudloff, 2013. "Time consistency of dynamic risk measures in markets with transaction costs," Quantitative Finance, Taylor & Francis Journals, vol. 13(9), pages 1473-1489, September.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2010.07220. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.