IDEAS home Printed from https://ideas.repec.org/a/plo/pone00/0236178.html
   My bibliography  Save this article

Action-specialized expert ensemble trading system with extended discrete action space using deep reinforcement learning

Author

Listed:
  • JoonBum Leem
  • Ha Young Kim

Abstract

Despite active research on trading systems based on reinforcement learning, the development and performance of research methods require improvements. This study proposes a new action-specialized expert ensemble method consisting of action-specialized expert models designed specifically for each reinforcement learning action: buy, hold, and sell. Models are constructed by examining and defining different reward values that correlate with each action under specific conditions, and investment behavior is reflected with each expert model. To verify the performance of this technique, profits of the proposed system are compared to those of single trading and common ensemble systems. To verify robustness and account for the extension of discrete action space, we compared and analyzed changes in profits of the three actions to our model’s results. Furthermore, we checked for sensitivity with three different reward functions: profit, Sharpe ratio, and Sortino ratio. All experiments were conducted with S&P500, Hang Seng Index, and Eurostoxx50 data. The model was 39.1% and 21.6% more efficient than single and common ensemble models, respectively. Considering the extended discrete action space, the 3-action space was extended to 11- and 21-action spaces, and the cumulative returns increased by 427.2% and 856.7%, respectively. Results on reward functions indicated that our models are well trained; results of the Sharpe and Sortino ratios were better than the implementation of profit only, as in the single-model cases. The Sortino ratio was slightly better than the Sharpe ratio.

Suggested Citation

  • JoonBum Leem & Ha Young Kim, 2020. "Action-specialized expert ensemble trading system with extended discrete action space using deep reinforcement learning," PLOS ONE, Public Library of Science, vol. 15(7), pages 1-39, July.
  • Handle: RePEc:plo:pone00:0236178
    DOI: 10.1371/journal.pone.0236178
    as

    Download full text from publisher

    File URL: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0236178
    Download Restriction: no

    File URL: https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0236178&type=printable
    Download Restriction: no

    File URL: https://libkey.io/10.1371/journal.pone.0236178?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Wei Bao & Jun Yue & Yulei Rao, 2017. "A deep learning framework for financial time series using stacked autoencoders and long-short term memory," PLOS ONE, Public Library of Science, vol. 12(7), pages 1-24, July.
    2. Volodymyr Mnih & Koray Kavukcuoglu & David Silver & Andrei A. Rusu & Joel Veness & Marc G. Bellemare & Alex Graves & Martin Riedmiller & Andreas K. Fidjeland & Georg Ostrovski & Stig Petersen & Charle, 2015. "Human-level control through deep reinforcement learning," Nature, Nature, vol. 518(7540), pages 529-533, February.
    3. Henry H. Huang & Hung-Yi Huang & Jeffrey J. Oxman, 2015. "Stock Liquidity And Corporate Bond Yield Spreads: Theory And Evidence," Journal of Financial Research, Southern Finance Association;Southwestern Finance Association, vol. 38(1), pages 59-91, March.
    4. Jying‐Nan Wang & Hung‐Chun Liu & Jiangze Du & Yuan‐Teng Hsu, 2019. "Economic benefits of technical analysis in portfolio management: Evidence from global stock markets," International Journal of Finance & Economics, John Wiley & Sons, Ltd., vol. 24(2), pages 890-902, April.
    5. Cheng Ju & Aurélien Bibaut & Mark van der Laan, 2018. "The relative performance of ensemble methods with deep convolutional neural networks for image classification," Journal of Applied Statistics, Taylor & Francis Journals, vol. 45(15), pages 2800-2818, November.
    6. Jegadeesh, Narasimhan & Titman, Sheridan, 1993. "Returns to Buying Winners and Selling Losers: Implications for Stock Market Efficiency," Journal of Finance, American Finance Association, vol. 48(1), pages 65-91, March.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Jatin Nainani & Nirman Taterh & Md Ausaf Rashid & Ankit Khivasara, 2022. "Feature-Rich Long-term Bitcoin Trading Assistant," Papers 2209.12664, arXiv.org.
    2. Adrian Millea, 2021. "Deep Reinforcement Learning for Trading—A Critical Survey," Data, MDPI, vol. 6(11), pages 1-25, November.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Wei Pan & Jide Li & Xiaoqiang Li, 2020. "Portfolio Learning Based on Deep Learning," Future Internet, MDPI, vol. 12(11), pages 1-13, November.
    2. Flori, Andrea & Regoli, Daniele, 2021. "Revealing Pairs-trading opportunities with long short-term memory networks," European Journal of Operational Research, Elsevier, vol. 295(2), pages 772-791.
    3. Shuo Sun & Rundong Wang & Bo An, 2021. "Reinforcement Learning for Quantitative Trading," Papers 2109.13851, arXiv.org.
    4. Iwao Maeda & David deGraw & Michiharu Kitano & Hiroyasu Matsushima & Hiroki Sakaji & Kiyoshi Izumi & Atsuo Kato, 2020. "Deep Reinforcement Learning in Agent Based Financial Market Simulation," JRFM, MDPI, vol. 13(4), pages 1-17, April.
    5. Bryan Lim & Stefan Zohren & Stephen Roberts, 2019. "Enhancing Time Series Momentum Strategies Using Deep Neural Networks," Papers 1904.04912, arXiv.org, revised Sep 2020.
    6. Amit Milstein & Haoran Deng & Guy Revach & Hai Morgenstern & Nir Shlezinger, 2022. "Neural Augmented Kalman Filtering with Bollinger Bands for Pairs Trading," Papers 2210.15448, arXiv.org, revised Sep 2023.
    7. Kieran Wood & Stephen Roberts & Stefan Zohren, 2021. "Slow Momentum with Fast Reversion: A Trading Strategy Using Deep Learning and Changepoint Detection," Papers 2105.13727, arXiv.org, revised Dec 2021.
    8. Jifei Wang & Lingjing Wang, 2019. "Residual Switching Network for Portfolio Optimization," Papers 1910.07564, arXiv.org.
    9. Harrison Hong & Terence Lim & Jeremy C. Stein, 2000. "Bad News Travels Slowly: Size, Analyst Coverage, and the Profitability of Momentum Strategies," Journal of Finance, American Finance Association, vol. 55(1), pages 265-295, February.
    10. Berg, Joyce E. & Rietz, Thomas A., 2019. "Longshots, overconfidence and efficiency on the Iowa Electronic Market," International Journal of Forecasting, Elsevier, vol. 35(1), pages 271-287.
    11. Rojahn, Joachim & Röhl, Christian W. & Frère, Eric, 2010. "Optimum Portfolio ETF Indices: Benchmarking für multidimensional diversifizierte Wertpapierportfolios," Berichte aus der Forschung der FOM 75202, FOM Hochschule für Oekonomie & Management.
    12. Shi, Huai-Long & Zhou, Wei-Xing, 2022. "Factor volatility spillover and its implications on factor premia," Journal of International Financial Markets, Institutions and Money, Elsevier, vol. 80(C).
    13. David A. Volkman, 1999. "Market Volatility And Perverse Timing Performance Of Mutual Fund Managers," Journal of Financial Research, Southern Finance Association;Southwestern Finance Association, vol. 22(4), pages 449-470, December.
    14. Pastor, Lubos & Stambaugh, Robert F., 2003. "Liquidity Risk and Expected Stock Returns," Journal of Political Economy, University of Chicago Press, vol. 111(3), pages 642-685, June.
    15. Klaus Grobys & James W. Kolari & Jere Rutanen, 2022. "Factor momentum, option-implied volatility scaling, and investor sentiment," Journal of Asset Management, Palgrave Macmillan, vol. 23(2), pages 138-155, March.
    16. Constantinos Antoniou & John A. Doukas & Avanidhar Subrahmanyam, 2016. "Investor Sentiment, Beta, and the Cost of Equity Capital," Management Science, INFORMS, vol. 62(2), pages 347-367, February.
    17. Agarwal, Vikas & Gay, Gerald D. & Ling, Leng, 2011. "Window dressing in mutual funds," CFR Working Papers 11-07, University of Cologne, Centre for Financial Research (CFR).
    18. Siddiqi, Hammad, 2015. "Anchoring and Adjustment Heuristic: A Unified Explanation for Equity Puzzles," MPRA Paper 68729, University Library of Munich, Germany.
    19. Tulika Saha & Sriparna Saha & Pushpak Bhattacharyya, 2020. "Towards sentiment aided dialogue policy learning for multi-intent conversations using hierarchical reinforcement learning," PLOS ONE, Public Library of Science, vol. 15(7), pages 1-28, July.
    20. Philip A. Stork, 2011. "The intertemporal mechanics of European stock price momentum," Studies in Economics and Finance, Emerald Group Publishing Limited, vol. 28(3), pages 217-232, August.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pone00:0236178. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: plosone (email available below). General contact details of provider: https://journals.plos.org/plosone/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.