IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2203.09612.html
   My bibliography  Save this paper

Risk-Averse Markov Decision Processes through a Distributional Lens

Author

Listed:
  • Ziteng Cheng
  • Sebastian Jaimungal

Abstract

By adopting a distributional viewpoint on law-invariant convex risk measures, we construct dynamics risk measures (DRMs) at the distributional level. We then apply these DRMs to investigate Markov decision processes, incorporating latent costs, random actions, and weakly continuous transition kernels. Furthermore, the proposed DRMs allow risk aversion to change dynamically. Under mild assumptions, we derive a dynamic programming principle and show the existence of an optimal policy in both finite and infinite time horizons. Moreover, we provide a sufficient condition for the optimality of deterministic actions. For illustration, we conclude the paper with examples from optimal liquidation with limit order books and autonomous driving.

Suggested Citation

  • Ziteng Cheng & Sebastian Jaimungal, 2022. "Risk-Averse Markov Decision Processes through a Distributional Lens," Papers 2203.09612, arXiv.org, revised Apr 2024.
    • Handle: RePEc:arx:papers:2203.09612
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2203.09612
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Riedel, Frank, 2004. "Dynamic coherent risk measures," Stochastic Processes and their Applications, Elsevier, vol. 112(2), pages 185-200, August.
    2. Tomasz R. Bielecki & Igor Cialenco & Marcin Pitera, 2016. "A survey of time consistency of dynamic risk measures and dynamic performance measures in discrete time: LM-measure perspective," Papers 1603.09030, arXiv.org, revised Jan 2017.
    3. Nicole Bäuerle & Alexander Glauner, 2021. "Minimizing spectral risk measures applied to Markov decision processes," Mathematical Methods of Operations Research, Springer;Gesellschaft für Operations Research (GOR);Nederlands Genootschap voor Besliskunde (NGB), vol. 94(1), pages 35-69, August.
    4. Georg Ch. Pflug & Alois Pichler, 2016. "Time-Consistent Decisions and Temporal Decomposition of Coherent Risk Functionals," Mathematics of Operations Research, INFORMS, vol. 41(2), pages 682-699, May.
    5. Tomasz R. Bielecki & Igor Cialenco & Marcin Pitera, 2014. "A unified approach to time consistency of dynamic risk measures and dynamic performance measures in discrete time," Papers 1409.7028, arXiv.org, revised Sep 2017.
    6. Philippe Artzner & Freddy Delbaen & Jean‐Marc Eber & David Heath, 1999. "Coherent Measures of Risk," Mathematical Finance, Wiley Blackwell, vol. 9(3), pages 203-228, July.
    7. Rosazza Gianin, Emanuela, 2006. "Risk measures via g-expectations," Insurance: Mathematics and Economics, Elsevier, vol. 39(1), pages 19-34, August.
    8. Charalambos D. Aliprantis & Kim C. Border, 2006. "Infinite Dimensional Analysis," Springer Books, Springer, edition 0, number 978-3-540-29587-7, June.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Anthony Coache & Sebastian Jaimungal & 'Alvaro Cartea, 2022. "Conditionally Elicitable Dynamic Risk Measures for Deep Reinforcement Learning," Papers 2206.14666, arXiv.org, revised May 2023.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Elisa Mastrogiacomo & Emanuela Rosazza Gianin, 2019. "Time-consistency of risk measures: how strong is such a property?," Decisions in Economics and Finance, Springer;Associazione per la Matematica, vol. 42(1), pages 287-317, June.
    2. Tomasz R. Bielecki & Igor Cialenco & Marcin Pitera, 2018. "A Unified Approach to Time Consistency of Dynamic Risk Measures and Dynamic Performance Measures in Discrete Time," Mathematics of Operations Research, INFORMS, vol. 43(1), pages 204-221, February.
    3. Tomasz R. Bielecki & Igor Cialenco & Tao Chen, 2014. "Dynamic Conic Finance via Backward Stochastic Difference Equations," Papers 1412.6459, arXiv.org, revised Dec 2014.
    4. E. Kromer & L. Overbeck & K. Zilch, 2019. "Dynamic systemic risk measures for bounded discrete time processes," Mathematical Methods of Operations Research, Springer;Gesellschaft für Operations Research (GOR);Nederlands Genootschap voor Besliskunde (NGB), vol. 90(1), pages 77-108, August.
    5. Ji, Ronglin & Shi, Xuejun & Wang, Shijie & Zhou, Jinming, 2019. "Dynamic risk measures for processes via backward stochastic differential equations," Insurance: Mathematics and Economics, Elsevier, vol. 86(C), pages 43-50.
    6. Andreas H Hamel, 2018. "Monetary Measures of Risk," Papers 1812.04354, arXiv.org.
    7. Wing Fung Chong & Ying Hu & Gechun Liang & Thaleia Zariphopoulou, 2019. "An ergodic BSDE approach to forward entropic risk measures: representation and large-maturity behavior," Finance and Stochastics, Springer, vol. 23(1), pages 239-273, January.
    8. Pelsser, Antoon & Salahnejhad Ghalehjooghi, Ahmad, 2016. "Time-consistent actuarial valuations," Insurance: Mathematics and Economics, Elsevier, vol. 66(C), pages 97-112.
    9. c{C}au{g}{i}n Ararat & Zachary Feinstein, 2019. "Set-Valued Risk Measures as Backward Stochastic Difference Inclusions and Equations," Papers 1912.06916, arXiv.org, revised Sep 2020.
    10. Alessandro Doldi & Marco Frittelli, 2020. "Conditional Systemic Risk Measures," Papers 2010.11515, arXiv.org, revised May 2021.
    11. Rohini Kumar & Frederick Forrest Miller & Hussein Nasralah & Stephan Sturm, 2024. "Risk-indifference Pricing of American-style Contingent Claims," Papers 2409.00095, arXiv.org.
    12. Christos E. Kountzakis & Damiano Rossello, 2019. "Acceptability Indices of Performance for Bounded C\`adl\`ag Processes," Papers 1911.02261, arXiv.org.
    13. Strub, Moris S. & Li, Duan & Cui, Xiangyu & Gao, Jianjun, 2019. "Discrete-time mean-CVaR portfolio selection and time-consistency induced term structure of the CVaR," Journal of Economic Dynamics and Control, Elsevier, vol. 108(C).
    14. Daniel Bartl, 2016. "Conditional nonlinear expectations," Papers 1612.09103, arXiv.org, revised Mar 2019.
    15. Zang, Xin & Jiang, Fan & Xia, Chenxi & Yang, Jingping, 2024. "Random distortion risk measures," Insurance: Mathematics and Economics, Elsevier, vol. 116(C), pages 51-73.
    16. Dejian Tian & Xunlian Wang, 2023. "Dynamic star-shaped risk measures and $g$-expectations," Papers 2305.02481, arXiv.org.
    17. Tomasz R. Bielecki & Igor Cialenco & Marcin Pitera, 2014. "A unified approach to time consistency of dynamic risk measures and dynamic performance measures in discrete time," Papers 1409.7028, arXiv.org, revised Sep 2017.
    18. Nicole EL KAROUI & Claudia RAVANELLI, 2008. "Cash Sub-additive Risk Measures and Interest Rate Ambiguity," Swiss Finance Institute Research Paper Series 08-09, Swiss Finance Institute.
    19. Saul Jacka & Seb Armstrong & Abdel Berkaoui, 2017. "Multi-currency reserving for coherent risk measures," Papers 1712.01319, arXiv.org, revised Dec 2017.
    20. Stadje, Mitja, 2010. "Extending dynamic convex risk measures from discrete time to continuous time: A convergence approach," Insurance: Mathematics and Economics, Elsevier, vol. 47(3), pages 391-404, December.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2203.09612. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.