IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2112.13414.html
   My bibliography  Save this paper

Reinforcement Learning with Dynamic Convex Risk Measures

Author

Listed:
  • Anthony Coache
  • Sebastian Jaimungal

Abstract

We develop an approach for solving time-consistent risk-sensitive stochastic optimization problems using model-free reinforcement learning (RL). Specifically, we assume agents assess the risk of a sequence of random variables using dynamic convex risk measures. We employ a time-consistent dynamic programming principle to determine the value of a particular policy, and develop policy gradient update rules that aid in obtaining optimal policies. We further develop an actor-critic style algorithm using neural networks to optimize over policies. Finally, we demonstrate the performance and flexibility of our approach by applying it to three optimization problems: statistical arbitrage trading strategies, financial hedging, and obstacle avoidance robot control.

Suggested Citation

  • Anthony Coache & Sebastian Jaimungal, 2021. "Reinforcement Learning with Dynamic Convex Risk Measures," Papers 2112.13414, arXiv.org, revised Nov 2022.
    • Handle: RePEc:arx:papers:2112.13414
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2112.13414
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Riedel, Frank, 2004. "Dynamic coherent risk measures," Stochastic Processes and their Applications, Elsevier, vol. 112(2), pages 185-200, August.
    2. Paul Milgrom & Ilya Segal, 2002. "Envelope Theorems for Arbitrary Choice Sets," Econometrica, Econometric Society, vol. 70(2), pages 583-601, March.
    3. Steven Campbell & Yichao Chen & Arvind Shrivats & Sebastian Jaimungal, 2021. "Deep Learning for Principal-Agent Mean Field Games," Papers 2110.01127, arXiv.org.
    4. Ben Hambly & Renyuan Xu & Huining Yang, 2020. "Policy Gradient Methods for the Noisy Linear Quadratic Regulator over a Finite Horizon," Papers 2011.10300, arXiv.org, revised Jun 2021.
    5. Erick Delage & Shie Mannor, 2010. "Percentile Optimization for Markov Decision Processes with Parameter Uncertainty," Operations Research, INFORMS, vol. 58(1), pages 203-213, February.
    6. Marek Petrik & Dharmashankar Subramanian, 2012. "An Approximate Solution Method for Large Risk-Averse Markov Decision Processes," Papers 1210.4901, arXiv.org.
    7. Ali Al-Aradi & Adolfo Correia & Danilo Naiff & Gabriel Jardim & Yuri Saporito, 2018. "Solving Nonlinear and High-Dimensional Partial Differential Equations via Deep Learning," Papers 1811.08782, arXiv.org.
    8. Christa Cuchiero & Wahid Khosrawi & Josef Teichmann, 2020. "A Generative Adversarial Network Approach to Calibration of Local Stochastic Volatility Models," Risks, MDPI, vol. 8(4), pages 1-31, September.
    9. Philippe Artzner & Freddy Delbaen & Jean‐Marc Eber & David Heath, 1999. "Coherent Measures of Risk," Mathematical Finance, Wiley Blackwell, vol. 9(3), pages 203-228, July.
    10. Detlefsen, Kai & Scandolo, Giacomo, 2005. "Conditional and dynamic convex risk measures," SFB 649 Discussion Papers 2005-006, Humboldt University Berlin, Collaborative Research Center 649: Economic Risk.
    11. Heston, Steven L, 1993. "A Closed-Form Solution for Options with Stochastic Volatility with Applications to Bond and Currency Options," The Review of Financial Studies, Society for Financial Studies, vol. 6(2), pages 327-343.
    12. Kai Detlefsen & Giacomo Scandolo, 2005. "Conditional and dynamic convex risk measures," Finance and Stochastics, Springer, vol. 9(4), pages 539-561, October.
    13. Christa Cuchiero & Wahid Khosrawi & Josef Teichmann, 2020. "A generative adversarial network approach to calibration of local stochastic volatility models," Papers 2005.02505, arXiv.org, revised Sep 2020.
    14. Stefan Weber, 2006. "Distribution‐Invariant Risk Measures, Information, And Dynamic Consistency," Mathematical Finance, Wiley Blackwell, vol. 16(2), pages 419-441, April.
    15. Sebastian Jaimungal, 2022. "Reinforcement learning and stochastic optimisation," Finance and Stochastics, Springer, vol. 26(1), pages 103-129, January.
    16. David Silver & Aja Huang & Chris J. Maddison & Arthur Guez & Laurent Sifre & George van den Driessche & Julian Schrittwieser & Ioannis Antonoglou & Veda Panneershelvam & Marc Lanctot & Sander Dieleman, 2016. "Mastering the game of Go with deep neural networks and tree search," Nature, Nature, vol. 529(7587), pages 484-489, January.
    17. Brian Ning & Sebastian Jaimungal & Xiaorong Zhang & Maxime Bergeron, 2021. "Arbitrage-Free Implied Volatility Surface Generation with Variational Autoencoders," Papers 2108.04941, arXiv.org, revised Jan 2022.
    18. Blanka Horvath & Aitor Muguruza & Mehdi Tomas, 2021. "Deep learning volatility: a deep neural network perspective on pricing and calibration in (rough) volatility models," Quantitative Finance, Taylor & Francis Journals, vol. 21(1), pages 11-27, January.
    19. Hans Föllmer & Alexander Schied, 2002. "Convex measures of risk and trading constraints," Finance and Stochastics, Springer, vol. 6(4), pages 429-447.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Anthony Coache & Sebastian Jaimungal & 'Alvaro Cartea, 2022. "Conditionally Elicitable Dynamic Risk Measures for Deep Reinforcement Learning," Papers 2206.14666, arXiv.org, revised May 2023.
    2. Nicole EL KAROUI & Claudia RAVANELLI, 2008. "Cash Sub-additive Risk Measures and Interest Rate Ambiguity," Swiss Finance Institute Research Paper Series 08-09, Swiss Finance Institute.
    3. Daniel Lacker, 2015. "Law invariant risk measures and information divergences," Papers 1510.07030, arXiv.org, revised Jun 2016.
    4. Tomasz R. Bielecki & Igor Cialenco & Marcin Pitera, 2014. "A unified approach to time consistency of dynamic risk measures and dynamic performance measures in discrete time," Papers 1409.7028, arXiv.org, revised Sep 2017.
    5. Ji, Ronglin & Shi, Xuejun & Wang, Shijie & Zhou, Jinming, 2019. "Dynamic risk measures for processes via backward stochastic differential equations," Insurance: Mathematics and Economics, Elsevier, vol. 86(C), pages 43-50.
    6. Zachary Feinstein & Birgit Rudloff, 2018. "Scalar multivariate risk measures with a single eligible asset," Papers 1807.10694, arXiv.org, revised Feb 2021.
    7. Qinyu Wu & Fan Yang & Ping Zhang, 2023. "Conditional generalized quantiles based on expected utility model and equivalent characterization of properties," Papers 2301.12420, arXiv.org.
    8. Fei Sun & Jingchao Li & Jieming Zhou, 2018. "Dynamic risk measures with fluctuation of market volatility under Bochne-Lebesgue space," Papers 1806.01166, arXiv.org, revised Mar 2024.
    9. Elisa Mastrogiacomo & Emanuela Rosazza Gianin, 2019. "Time-consistency of risk measures: how strong is such a property?," Decisions in Economics and Finance, Springer;Associazione per la Matematica, vol. 42(1), pages 287-317, June.
    10. Dan A. Iancu & Marek Petrik & Dharmashankar Subramanian, 2015. "Tight Approximations of Dynamic Risk Measures," Mathematics of Operations Research, INFORMS, vol. 40(3), pages 655-682, March.
    11. Acciaio, Beatrice & Föllmer, Hans & Penner, Irina, 2012. "Risk assessment for uncertain cash flows: model ambiguity, discounting ambiguity, and the role of bubbles," LSE Research Online Documents on Economics 50118, London School of Economics and Political Science, LSE Library.
    12. De Lara, Michel & Leclère, Vincent, 2016. "Building up time-consistency for risk measures and dynamic optimization," European Journal of Operational Research, Elsevier, vol. 249(1), pages 177-187.
    13. Yi Shen & Zachary Van Oosten & Ruodu Wang, 2024. "Partial Law Invariance and Risk Measures," Papers 2401.17265, arXiv.org, revised Jun 2024.
    14. Zachary Feinstein & Birgit Rudloff, 2012. "Multiportfolio time consistency for set-valued convex and coherent risk measures," Papers 1212.5563, arXiv.org, revised Oct 2014.
    15. Bellini, Fabio & Laeven, Roger J.A. & Rosazza Gianin, Emanuela, 2021. "Dynamic robust Orlicz premia and Haezendonck–Goovaerts risk measures," European Journal of Operational Research, Elsevier, vol. 291(2), pages 438-446.
    16. Mitja Stadje, 2018. "Representation Results for Law Invariant Recursive Dynamic Deviation Measures and Risk Sharing," Papers 1811.09615, arXiv.org, revised Dec 2018.
    17. Wayne King Ming Chan, 2015. "RAROC-Based Contingent Claim Valuation," PhD Thesis, Finance Discipline Group, UTS Business School, University of Technology, Sydney, number 21, July-Dece.
    18. Zachary Feinstein & Birgit Rudloff, 2018. "Time consistency for scalar multivariate risk measures," Papers 1810.04978, arXiv.org, revised Nov 2021.
    19. Zachary Feinstein & Birgit Rudloff, 2015. "Multi-portfolio time consistency for set-valued convex and coherent risk measures," Finance and Stochastics, Springer, vol. 19(1), pages 67-107, January.
    20. Alexander S. Cherny, 2009. "Capital Allocation And Risk Contribution With Discrete‐Time Coherent Risk," Mathematical Finance, Wiley Blackwell, vol. 19(1), pages 13-40, January.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2112.13414. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.