IDEAS home Printed from https://ideas.repec.org/a/eee/ejores/v311y2023i3p1057-1067.html
   My bibliography  Save this article

A unified algorithm framework for mean-variance optimization in discounted Markov decision processes

Author

Listed:
  • Ma, Shuai
  • Ma, Xiaoteng
  • Xia, Li

Abstract

This paper studies the risk-averse mean-variance optimization in infinite-horizon discounted Markov decision processes (MDPs). The involved variance metric concerns reward variability during the whole process, and future deviations are discounted to their present values. This discounted mean-variance optimization yields a reward function dependent on a discounted mean, and this dependency renders traditional dynamic programming methods inapplicable since it suppresses a crucial property—time-consistency. To deal with this unorthodox problem, we introduce a pseudo mean to transform the untreatable MDP to a standard one with a redefined reward function in standard form and derive a discounted mean-variance performance difference formula. With the pseudo mean, we propose a unified algorithm framework with a bilevel optimization structure for the discounted mean-variance optimization. The framework unifies a variety of algorithms for several variance-related problems, including, but not limited to, risk-averse variance and mean-variance optimizations in discounted and average MDPs. Furthermore, the convergence analyses missing from the literature can be complemented with the proposed framework as well. Taking the value iteration as an example, we develop a discounted mean-variance value iteration algorithm and prove its convergence to a local optimum with the aid of a Bellman local-optimality equation. Finally, we conduct a numerical experiment on portfolio management to validate the proposed algorithm.

Suggested Citation

  • Ma, Shuai & Ma, Xiaoteng & Xia, Li, 2023. "A unified algorithm framework for mean-variance optimization in discounted Markov decision processes," European Journal of Operational Research, Elsevier, vol. 311(3), pages 1057-1067.
  • Handle: RePEc:eee:ejores:v:311:y:2023:i:3:p:1057-1067
    DOI: 10.1016/j.ejor.2023.06.022
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0377221723004757
    Download Restriction: Full text for ScienceDirect subscribers only

    File URL: https://libkey.io/10.1016/j.ejor.2023.06.022?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Kandel, Shmuel & Stambaugh, Robert F, 1989. "A Mean-Variance Framework for Tests of Asset Pricing Models," The Review of Financial Studies, Society for Financial Studies, vol. 2(2), pages 125-156.
    2. Zhuo, Wenyan & Shao, Lusheng & Yang, Honglin, 2018. "Mean–variance analysis of option contracts in a two-echelon supply chain," European Journal of Operational Research, Elsevier, vol. 271(2), pages 535-547.
    3. Tversky, Amos & Kahneman, Daniel, 1992. "Advances in Prospect Theory: Cumulative Representation of Uncertainty," Journal of Risk and Uncertainty, Springer, vol. 5(4), pages 297-323, October.
    4. Caldeira, João F. & Moura, Guilherme V. & Santos, André A.P., 2016. "Bond portfolio optimization using dynamic factor models," Journal of Empirical Finance, Elsevier, vol. 37(C), pages 128-158.
    5. Min Dai & Hanqing Jin & Steven Kou & Yuhong Xu, 2021. "A Dynamic Mean-Variance Analysis for Log Returns," Management Science, INFORMS, vol. 67(2), pages 1093-1108, February.
    6. Panos Kouvelis & Zhan Pang & Qing Ding, 2018. "Integrated Commodity Inventory Management and Financial Hedging: A Dynamic Mean†Variance Analysis," Production and Operations Management, Production and Operations Management Society, vol. 27(6), pages 1052-1073, June.
    7. Kun-Jen Chung, 1994. "Mean-Variance Tradeoffs in an Undiscounted MDP: The Unichain Case," Operations Research, INFORMS, vol. 42(1), pages 184-188, February.
    8. Suleyman Basak & Georgy Chabakauri, 2010. "Dynamic Mean-Variance Asset Allocation," The Review of Financial Studies, Society for Financial Studies, vol. 23(8), pages 2970-3016, August.
    9. Jerzy A. Filar & L. C. M. Kallenberg & Huey-Miin Lee, 1989. "Variance-Penalized Markov Decision Processes," Mathematics of Operations Research, INFORMS, vol. 14(1), pages 147-161, February.
    10. Li Xia, 2020. "Risk‐Sensitive Markov Decision Processes with Combined Metrics of Mean and Variance," Production and Operations Management, Production and Operations Management Society, vol. 29(12), pages 2808-2827, December.
    11. Li, Y.Z. & Wu, Q.H. & Li, M.S. & Zhan, J.P., 2014. "Mean-variance model for power system economic dispatch with wind power integrated," Energy, Elsevier, vol. 72(C), pages 510-520.
    12. Guo, Xianping & Ye, Liuer & Yin, George, 2012. "A mean–variance optimization problem for discounted Markov decision processes," European Journal of Operational Research, Elsevier, vol. 220(2), pages 423-429.
    13. Zhang, Wei-Guo & Liu, Yong-Jun & Xu, Wei-Jun, 2012. "A possibilistic mean-semivariance-entropy model for multi-period portfolio selection with transaction costs," European Journal of Operational Research, Elsevier, vol. 222(2), pages 341-349.
    14. Harry Markowitz, 1952. "Portfolio Selection," Journal of Finance, American Finance Association, vol. 7(1), pages 77-91, March.
    15. Duan Li & Wan‐Lung Ng, 2000. "Optimal Dynamic Portfolio Selection: Multiperiod Mean‐Variance Formulation," Mathematical Finance, Wiley Blackwell, vol. 10(3), pages 387-406, July.
    16. Jonathan Eckstein & Deniz Eskandani & Jingnan Fan, 2016. "Multilevel Optimization Modeling for Risk-Averse Stochastic Programming," INFORMS Journal on Computing, INFORMS, vol. 28(1), pages 112-128, February.
    17. Matthew J. Sobel, 1994. "Mean-Variance Tradeoffs in an Undiscounted MDP," Operations Research, INFORMS, vol. 42(1), pages 175-183, February.
    18. Michael J. Best & Robert R. Grauer, 1991. "Sensitivity Analysis for Mean-Variance Portfolio Problems," Management Science, INFORMS, vol. 37(8), pages 980-989, August.
    19. Xueting Cui & Xiaoling Sun & Shushang Zhu & Rujun Jiang & Duan Li, 2018. "Portfolio Optimization with Nonparametric Value at Risk: A Block Coordinate Descent Method," INFORMS Journal on Computing, INFORMS, vol. 30(3), pages 454-471, August.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Li Xia, 2020. "Risk‐Sensitive Markov Decision Processes with Combined Metrics of Mean and Variance," Production and Operations Management, Production and Operations Management Society, vol. 29(12), pages 2808-2827, December.
    2. van Staden, Pieter M. & Dang, Duy-Minh & Forsyth, Peter A., 2021. "The surprising robustness of dynamic Mean-Variance portfolio optimization to model misspecification errors," European Journal of Operational Research, Elsevier, vol. 289(2), pages 774-792.
    3. De Gennaro Aquino, Luca & Sornette, Didier & Strub, Moris S., 2023. "Portfolio selection with exploration of new investment assets," European Journal of Operational Research, Elsevier, vol. 310(2), pages 773-792.
    4. Xiangyu Cui & Xun Li & Duan Li & Yun Shi, 2014. "Time Consistent Behavior Portfolio Policy for Dynamic Mean-Variance Formulation," Papers 1408.6070, arXiv.org, revised Aug 2015.
    5. Liu, Yong-Jun & Zhang, Wei-Guo, 2015. "A multi-period fuzzy portfolio optimization model with minimum transaction lots," European Journal of Operational Research, Elsevier, vol. 242(3), pages 933-941.
    6. Li, Yongwu & Li, Zhongfei, 2013. "Optimal time-consistent investment and reinsurance strategies for mean–variance insurers with state dependent risk aversion," Insurance: Mathematics and Economics, Elsevier, vol. 53(1), pages 86-97.
    7. F. Cong & C. W. Oosterlee, 2017. "On Robust Multi-Period Pre-Commitment And Time-Consistent Mean-Variance Portfolio Optimization," International Journal of Theoretical and Applied Finance (IJTAF), World Scientific Publishing Co. Pte. Ltd., vol. 20(07), pages 1-26, November.
    8. Jiao, P.H. & Chen, J.J. & Peng, K. & Zhao, Y.L. & Xin, K.F., 2020. "Multi-objective mean-semi-entropy model for optimal standalone micro-grid planning with uncertain renewable energy resources," Energy, Elsevier, vol. 191(C).
    9. Bauder, David & Bodnar, Taras & Parolya, Nestor & Schmid, Wolfgang, 2020. "Bayesian inference of the multi-period optimal portfolio for an exponential utility," Journal of Multivariate Analysis, Elsevier, vol. 175(C).
    10. Alessandro Arlotto & Noah Gans & J. Michael Steele, 2014. "Markov Decision Problems Where Means Bound Variances," Operations Research, INFORMS, vol. 62(4), pages 864-875, August.
    11. Bodnar, Taras & Parolya, Nestor & Schmid, Wolfgang, 2015. "On the exact solution of the multi-period portfolio choice problem for an exponential utility under return predictability," European Journal of Operational Research, Elsevier, vol. 246(2), pages 528-542.
    12. Najafi, Amir Abbas & Mushakhian, Siamak, 2015. "Multi-stage stochastic mean–semivariance–CVaR portfolio optimization under transaction costs," Applied Mathematics and Computation, Elsevier, vol. 256(C), pages 445-458.
    13. Li, Yongwu & Qiao, Han & Wang, Shouyang & Zhang, Ling, 2015. "Time-consistent investment strategy under partial information," Insurance: Mathematics and Economics, Elsevier, vol. 65(C), pages 187-197.
    14. Wu, Huiling & Chen, Hua, 2015. "Nash equilibrium strategy for a multi-period mean–variance portfolio selection problem with regime switching," Economic Modelling, Elsevier, vol. 46(C), pages 79-90.
    15. Xue Dong He & Xun Yu Zhou, 2021. "Who Are I: Time Inconsistency and Intrapersonal Conflict and Reconciliation," Papers 2105.01829, arXiv.org.
    16. Guo, Sini & Yu, Lean & Li, Xiang & Kar, Samarjit, 2016. "Fuzzy multi-period portfolio selection with different investment horizons," European Journal of Operational Research, Elsevier, vol. 254(3), pages 1026-1035.
    17. Agostino Capponi & Sveinn Ólafsson & Thaleia Zariphopoulou, 2022. "Personalized Robo-Advising: Enhancing Investment Through Client Interaction," Management Science, INFORMS, vol. 68(4), pages 2485-2512, April.
    18. Pfiffelmann, Marie & Roger, Tristan & Bourachnikova, Olga, 2016. "When Behavioral Portfolio Theory meets Markowitz theory," Economic Modelling, Elsevier, vol. 53(C), pages 419-435.
    19. Yam, Sheung Chi Phillip & Yang, Hailiang & Yuen, Fei Lung, 2016. "Optimal asset allocation: Risk and information uncertainty," European Journal of Operational Research, Elsevier, vol. 251(2), pages 554-561.
    20. Xiangyu Cui & Xun Li & Duan Li & Yun Shi, 2017. "Time consistent behavioral portfolio policy for dynamic mean–variance formulation," Journal of the Operational Research Society, Palgrave Macmillan;The OR Society, vol. 68(12), pages 1647-1660, December.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:ejores:v:311:y:2023:i:3:p:1057-1067. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/eor .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.