IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2109.13851.html
   My bibliography  Save this paper

Reinforcement Learning for Quantitative Trading

Author

Listed:
  • Shuo Sun
  • Rundong Wang
  • Bo An

Abstract

Quantitative trading (QT), which refers to the usage of mathematical models and data-driven techniques in analyzing the financial market, has been a popular topic in both academia and financial industry since 1970s. In the last decade, reinforcement learning (RL) has garnered significant interest in many domains such as robotics and video games, owing to its outstanding ability on solving complex sequential decision making problems. RL's impact is pervasive, recently demonstrating its ability to conquer many challenging QT tasks. It is a flourishing research direction to explore RL techniques' potential on QT tasks. This paper aims at providing a comprehensive survey of research efforts on RL-based methods for QT tasks. More concretely, we devise a taxonomy of RL-based QT models, along with a comprehensive summary of the state of the art. Finally, we discuss current challenges and propose future research directions in this exciting field.

Suggested Citation

  • Shuo Sun & Rundong Wang & Bo An, 2021. "Reinforcement Learning for Quantitative Trading," Papers 2109.13851, arXiv.org.
    • Handle: RePEc:arx:papers:2109.13851
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2109.13851
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Olivier Guéant & Iuliia Manziuk, 2019. "Deep Reinforcement Learning for Market Making in Corporate Bonds: Beating the Curse of Dimensionality," Applied Mathematical Finance, Taylor & Francis Journals, vol. 26(5), pages 387-452, September.
    2. Oriol Vinyals & Igor Babuschkin & Wojciech M. Czarnecki & Michaël Mathieu & Andrew Dudzik & Junyoung Chung & David H. Choi & Richard Powell & Timo Ewalds & Petko Georgiev & Junhyuk Oh & Dan Horgan & M, 2019. "Grandmaster level in StarCraft II using multi-agent reinforcement learning," Nature, Nature, vol. 575(7782), pages 350-354, November.
    3. Zhenhan Huang & Fumihide Tanaka, 2021. "MSPM: A Modularized and Scalable Multi-Agent Reinforcement Learning-based System for Financial Portfolio Management," Papers 2102.03502, arXiv.org, revised Feb 2022.
    4. Tianping Zhang & Yuanqi Li & Yifei Jin & Jian Li, 2020. "AutoAlpha: an Efficient Hierarchical Evolutionary Algorithm for Mining Alpha Factors in Quantitative Investment," Papers 2002.08245, arXiv.org, revised Apr 2020.
    5. Shihao Gu & Bryan Kelly & Dacheng Xiu, 2020. "Empirical Asset Pricing via Machine Learning," Review of Finance, European Finance Association, vol. 33(5), pages 2223-2273.
    6. Thomas Spooner & Rahul Savani, 2020. "Robust Market Making via Adversarial Reinforcement Learning," Papers 2003.01820, arXiv.org, revised Jul 2020.
    7. Panagiotidis, Theodore & Stengos, Thanasis & Vravosinos, Orestis, 2018. "On the determinants of bitcoin returns: A LASSO approach," Finance Research Letters, Elsevier, vol. 27(C), pages 235-240.
    8. Fuli Feng & Xiangnan He & Xiang Wang & Cheng Luo & Yiqun Liu & Tat-Seng Chua, 2018. "Temporal Relational Ranking for Stock Prediction," Papers 1809.09441, arXiv.org, revised Jan 2019.
    9. Shihao Gu & Bryan Kelly & Dacheng Xiu, 2020. "Empirical Asset Pricing via Machine Learning," The Review of Financial Studies, Society for Financial Studies, vol. 33(5), pages 2223-2273.
    10. Gode, Dhananjay K & Sunder, Shyam, 1993. "Allocative Efficiency of Markets with Zero-Intelligence Traders: Market as a Partial Substitute for Individual Rationality," Journal of Political Economy, University of Chicago Press, vol. 101(1), pages 119-137, February.
    11. Xiao-Yang Liu & Zhuoran Xiong & Shan Zhong & Hongyang Yang & Anwar Walid, 2018. "Practical Deep Reinforcement Learning Approach for Stock Trading," Papers 1811.07522, arXiv.org, revised Jul 2022.
    12. Yuchen Fang & Kan Ren & Weiqing Liu & Dong Zhou & Weinan Zhang & Jiang Bian & Yong Yu & Tie-Yan Liu, 2021. "Universal Trading for Order Execution with Oracle Policy Distillation," Papers 2103.10860, arXiv.org.
    13. Svitlana Vyetrenko & David Byrd & Nick Petosa & Mahmoud Mahfouz & Danial Dervovic & Manuela Veloso & Tucker Hybinette Balch, 2019. "Get Real: Realism Metrics for Robust Limit Order Book Market Simulations," Papers 1912.04941, arXiv.org.
    14. Sun, Xiaolei & Liu, Mingxi & Sima, Zeqian, 2020. "A novel cryptocurrency price trend forecasting model based on LightGBM," Finance Research Letters, Elsevier, vol. 32(C).
    15. Fama, Eugene F. & French, Kenneth R., 1993. "Common risk factors in the returns on stocks and bonds," Journal of Financial Economics, Elsevier, vol. 33(1), pages 3-56, February.
    16. William F. Sharpe, 1964. "Capital Asset Prices: A Theory Of Market Equilibrium Under Conditions Of Risk," Journal of Finance, American Finance Association, vol. 19(3), pages 425-442, September.
    17. Omer Berat Sezer & Mehmet Ugur Gudelek & Ahmet Murat Ozbayoglu, 2019. "Financial Time Series Forecasting with Deep Learning : A Systematic Literature Review: 2005-2019," Papers 1911.13288, arXiv.org.
    18. John Moody & Lizhong Wu, "undated". "Optimization of Trading Systems and Portfolios," Computing in Economics and Finance 1997 55, Society for Computational Economics.
    19. Olivier Gu'eant & Iuliia Manziuk, 2019. "Deep reinforcement learning for market making in corporate bonds: beating the curse of dimensionality," Papers 1910.13205, arXiv.org.
    20. Yunan Ye & Hengzhi Pei & Boxin Wang & Pin-Yu Chen & Yada Zhu & Jun Xiao & Bo Li, 2020. "Reinforcement-Learning based Portfolio Management with Augmented Asset Movement Prediction States," Papers 2002.05780, arXiv.org.
    21. Black, Fischer & Scholes, Myron S, 1973. "The Pricing of Options and Corporate Liabilities," Journal of Political Economy, University of Chicago Press, vol. 81(3), pages 637-654, May-June.
    22. Lakshay Chauhan & John Alberg & Zachary C. Lipton, 2020. "Uncertainty-Aware Lookahead Factor Models for Quantitative Investing," Papers 2007.04082, arXiv.org, revised Jul 2020.
    23. Jingyuan Wang & Yang Zhang & Ke Tang & Junjie Wu & Zhang Xiong, 2019. "AlphaStock: A Buying-Winners-and-Selling-Losers Investment Strategy using Interpretable Deep Reinforcement Attention Networks," Papers 1908.02646, arXiv.org.
    24. Stephan K. Chalup & Andreas Mitschele, 2008. "Kernel Methods in Finance," International Handbooks on Information Systems, in: Detlef Seese & Christof Weinhardt & Frank Schlottmann (ed.), Handbook on Information Technology in Finance, chapter 27, pages 655-687, Springer.
    25. Alexei Gaivoronski & Fabio Stella, 2000. "Stochastic Nonstationary Optimization for Finding Universal Portfolios," Annals of Operations Research, Springer, vol. 100(1), pages 165-188, December.
    26. Dieter Hendricks & Diane Wilcox, 2014. "A reinforcement learning extension to the Almgren-Chriss model for optimal trade execution," Papers 1403.2229, arXiv.org.
    27. Nicholas T. Chan and Christian Shelton, 2001. "An Adaptive Electronic Market-Maker," Computing in Economics and Finance 2001 146, Society for Computational Economics.
    28. Wentao Xu & Weiqing Liu & Chang Xu & Jiang Bian & Jian Yin & Tie-Yan Liu, 2021. "REST: Relational Event-driven Stock Trend Forecasting," Papers 2102.07372, arXiv.org, revised Feb 2021.
    29. Bertsimas, Dimitris & Lo, Andrew W., 1998. "Optimal control of execution costs," Journal of Financial Markets, Elsevier, vol. 1(1), pages 1-50, April.
    30. Basak, Suryoday & Kar, Saibal & Saha, Snehanshu & Khaidem, Luckyson & Dey, Sudeepa Roy, 2019. "Predicting the direction of stock market prices using tree-based classifiers," The North American Journal of Economics and Finance, Elsevier, vol. 47(C), pages 552-567.
    31. David P. Helmbold & Robert E. Schapire & Yoram Singer & Manfred K. Warmuth, 1998. "On‐Line Portfolio Selection Using Multiplicative Updates," Mathematical Finance, Wiley Blackwell, vol. 8(4), pages 325-347, October.
    32. Gu, Shihao & Kelly, Bryan & Xiu, Dacheng, 2021. "Autoencoder asset pricing models," Journal of Econometrics, Elsevier, vol. 222(1), pages 429-450.
    33. Fischer, Thomas G., 2018. "Reinforcement learning in financial markets - a survey," FAU Discussion Papers in Economics 12/2018, Friedrich-Alexander University Erlangen-Nuremberg, Institute for Economics.
    34. Zura Kakushadze, 2016. "101 Formulaic Alphas," Papers 1601.00991, arXiv.org, revised Mar 2016.
    35. David Silver & Aja Huang & Chris J. Maddison & Arthur Guez & Laurent Sifre & George van den Driessche & Julian Schrittwieser & Ioannis Antonoglou & Veda Panneershelvam & Marc Lanctot & Sander Dieleman, 2016. "Mastering the game of Go with deep neural networks and tree search," Nature, Nature, vol. 529(7587), pages 484-489, January.
    36. Thomas Spooner & John Fearnley & Rahul Savani & Andreas Koukorinis, 2018. "Market Making via Reinforcement Learning," Papers 1804.04216, arXiv.org.
    37. Eric Benhamou & David Saltiel & Sandrine Ungari & Abhishek Mukhopadhyay, 2020. "Bridging the gap between Markowitz planning and deep reinforcement learning," Papers 2010.09108, arXiv.org.
    38. Zhengyao Jiang & Dixing Xu & Jinjun Liang, 2017. "A Deep Reinforcement Learning Framework for the Financial Portfolio Management Problem," Papers 1706.10059, arXiv.org, revised Jul 2017.
    39. Volodymyr Mnih & Koray Kavukcuoglu & David Silver & Andrei A. Rusu & Joel Veness & Marc G. Bellemare & Alex Graves & Martin Riedmiller & Andreas K. Fidjeland & Georg Ostrovski & Stig Petersen & Charle, 2015. "Human-level control through deep reinforcement learning," Nature, Nature, vol. 518(7540), pages 529-533, February.
    40. Moskowitz, Tobias J. & Ooi, Yao Hua & Pedersen, Lasse Heje, 2012. "Time series momentum," Journal of Financial Economics, Elsevier, vol. 104(2), pages 228-250.
    41. Zhipeng Liang & Hao Chen & Junhao Zhu & Kangkang Jiang & Yanran Li, 2018. "Adversarial Deep Reinforcement Learning in Portfolio Management," Papers 1808.09940, arXiv.org, revised Nov 2018.
    42. Chan, Louis K C & Jegadeesh, Narasimhan & Lakonishok, Josef, 1996. "Momentum Strategies," Journal of Finance, American Finance Association, vol. 51(5), pages 1681-1713, December.
    43. Poterba, James M. & Summers, Lawrence H., 1988. "Mean reversion in stock prices : Evidence and Implications," Journal of Financial Economics, Elsevier, vol. 22(1), pages 27-59, October.
    44. Ahmet Murat Ozbayoglu & Mehmet Ugur Gudelek & Omer Berat Sezer, 2020. "Deep Learning for Financial Applications : A Survey," Papers 2002.05786, arXiv.org.
    45. László Györfi & Gábor Lugosi & Frederic Udina, 2006. "Nonparametric Kernel‐Based Sequential Investment Strategies," Mathematical Finance, Wiley Blackwell, vol. 16(2), pages 337-357, April.
    46. Jegadeesh, Narasimhan & Titman, Sheridan, 1993. "Returns to Buying Winners and Selling Losers: Implications for Stock Market Efficiency," Journal of Finance, American Finance Association, vol. 48(1), pages 65-91, March.
    47. Gaivoronski, A & Stella, F, 2000. "Nonstationary Optimization Approach for Finding Universal Portfolios," MPRA Paper 21913, University Library of Munich, Germany.
    48. Thomas M. Cover, 1991. "Universal Portfolios," Mathematical Finance, Wiley Blackwell, vol. 1(1), pages 1-29, January.
    49. Edoardo Vittori & Michele Trapletti & Marcello Restelli, 2020. "Option Hedging with Risk Averse Reinforcement Learning," Papers 2010.12245, arXiv.org.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Xiao-Yang Liu & Jingyang Rui & Jiechao Gao & Liuqing Yang & Hongyang Yang & Zhaoran Wang & Christina Dan Wang & Jian Guo, 2021. "FinRL-Meta: A Universe of Near-Real Market Environments for Data-Driven Deep Reinforcement Learning in Quantitative Finance," Papers 2112.06753, arXiv.org, revised Mar 2022.
    2. Zechu Li & Xiao-Yang Liu & Jiahao Zheng & Zhaoran Wang & Anwar Walid & Jian Guo, 2021. "FinRL-Podracer: High Performance and Scalable Deep Reinforcement Learning for Quantitative Finance," Papers 2111.05188, arXiv.org.
    3. Eduardo C. Garrido-Merch'an & Sol Mora-Figueroa-Cruz-Guzm'an & Mar'ia Coronado-Vaca, 2023. "Deep Reinforcement Learning for ESG financial portfolio management," Papers 2307.09631, arXiv.org.
    4. Jinan Zou & Qingying Zhao & Yang Jiao & Haiyao Cao & Yanxi Liu & Qingsen Yan & Ehsan Abbasnejad & Lingqiao Liu & Javen Qinfeng Shi, 2022. "Stock Market Prediction via Deep Learning Techniques: A Survey," Papers 2212.12717, arXiv.org, revised Feb 2023.
    5. Ben Hambly & Renyuan Xu & Huining Yang, 2023. "Recent advances in reinforcement learning in finance," Mathematical Finance, Wiley Blackwell, vol. 33(3), pages 437-503, July.
    6. Amit Milstein & Haoran Deng & Guy Revach & Hai Morgenstern & Nir Shlezinger, 2022. "Neural Augmented Kalman Filtering with Bollinger Bands for Pairs Trading," Papers 2210.15448, arXiv.org, revised Sep 2023.
    7. Hui Niu & Siyuan Li & Jian Li, 2022. "MetaTrader: An Reinforcement Learning Approach Integrating Diverse Policies for Portfolio Optimization," Papers 2210.01774, arXiv.org.
    8. Mao Guan & Xiao-Yang Liu, 2021. "Explainable Deep Reinforcement Learning for Portfolio Management: An Empirical Approach," Papers 2111.03995, arXiv.org, revised Dec 2021.
    9. Tian Zhu & Wei Zhu, 2022. "Quantitative Trading through Random Perturbation Q-Network with Nonlinear Transaction Costs," Stats, MDPI, vol. 5(2), pages 1-15, June.
    10. Shuo Sun & Molei Qin & Xinrun Wang & Bo An, 2023. "PRUDEX-Compass: Towards Systematic Evaluation of Reinforcement Learning in Financial Markets," Papers 2302.00586, arXiv.org, revised Mar 2023.
    11. Kim, Seil & Ogawa, Keiichi, 2024. "Who is able or unable to return to school? Exploring the short-term impact of the COVID-19 school closures on students' returning to school in Nigeria," International Journal of Educational Development, Elsevier, vol. 108(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Ben Hambly & Renyuan Xu & Huining Yang, 2021. "Recent Advances in Reinforcement Learning in Finance," Papers 2112.04553, arXiv.org, revised Feb 2023.
    2. Bruno Gašperov & Stjepan Begušić & Petra Posedel Šimović & Zvonko Kostanjčar, 2021. "Reinforcement Learning Approaches to Optimal Market Making," Mathematics, MDPI, vol. 9(21), pages 1-22, October.
    3. Ben Hambly & Renyuan Xu & Huining Yang, 2023. "Recent advances in reinforcement learning in finance," Mathematical Finance, Wiley Blackwell, vol. 33(3), pages 437-503, July.
    4. Jian Guo & Saizhuo Wang & Lionel M. Ni & Heung-Yeung Shum, 2022. "Quant 4.0: Engineering Quantitative Investment with Automated, Explainable and Knowledge-driven Artificial Intelligence," Papers 2301.04020, arXiv.org.
    5. Adel Javanmard & Jingwei Ji & Renyuan Xu, 2024. "Multi-Task Dynamic Pricing in Credit Market with Contextual Information," Papers 2410.14839, arXiv.org, revised Oct 2024.
    6. Paul Handro & Bogdan Dima, 2024. "Analyzing Financial Markets Efficiency: Insights from a Bibliometric and Content Review," Journal of Financial Studies, Institute of Financial Studies, vol. 16(9), pages 119-175, May.
    7. Blanco, Ivan & De Jesus, Miguel & Remesal, Alvaro, 2023. "Overlapping momentum portfolios," Journal of Empirical Finance, Elsevier, vol. 72(C), pages 1-22.
    8. Hui Niu & Siyuan Li & Jian Li, 2022. "MetaTrader: An Reinforcement Learning Approach Integrating Diverse Policies for Portfolio Optimization," Papers 2210.01774, arXiv.org.
    9. Tidor-Vlad Pricope, 2021. "Deep Reinforcement Learning in Quantitative Algorithmic Trading: A Review," Papers 2106.00123, arXiv.org.
    10. Wolfgang Drobetz & Tizian Otto, 2021. "Empirical asset pricing via machine learning: evidence from the European stock market," Journal of Asset Management, Palgrave Macmillan, vol. 22(7), pages 507-538, December.
    11. Azevedo, Vitor, 2023. "Analysts’ underreaction and momentum strategies," Journal of Economic Dynamics and Control, Elsevier, vol. 146(C).
    12. Yuchen Fang & Kan Ren & Weiqing Liu & Dong Zhou & Weinan Zhang & Jiang Bian & Yong Yu & Tie-Yan Liu, 2021. "Universal Trading for Order Execution with Oracle Policy Distillation," Papers 2103.10860, arXiv.org.
    13. Christian Fieberg & Daniel Metko & Thorsten Poddig & Thomas Loy, 2023. "Machine learning techniques for cross-sectional equity returns’ prediction," OR Spectrum: Quantitative Approaches in Management, Springer;Gesellschaft für Operations Research e.V., vol. 45(1), pages 289-323, March.
    14. Jiaju Miao & Pawel Polak, 2023. "Online Ensemble of Models for Optimal Predictive Performance with Applications to Sector Rotation Strategy," Papers 2304.09947, arXiv.org.
    15. Amirhosein Mosavi & Yaser Faghan & Pedram Ghamisi & Puhong Duan & Sina Faizollahzadeh Ardabili & Ely Salwana & Shahab S. Band, 2020. "Comprehensive Review of Deep Reinforcement Learning Methods and Applications in Economics," Mathematics, MDPI, vol. 8(10), pages 1-42, September.
    16. Cakici, Nusret & Zaremba, Adam, 2021. "Liquidity and the cross-section of international stock returns," Journal of Banking & Finance, Elsevier, vol. 127(C).
    17. Constantinos Kardaras & Hyeng Keun Koo & Johannes Ruf, 2022. "Estimation of growth in fund models," Papers 2208.02573, arXiv.org.
    18. Baba-Yara, Fahiz & Boons, Martijn & Tamoni, Andrea, 2024. "Persistent and transitory components of firm characteristics: Implications for asset pricing," Journal of Financial Economics, Elsevier, vol. 154(C).
    19. Svetlana Bryzgalova & Jiantao Huang & Christian Julliard, 2023. "Bayesian Solutions for the Factor Zoo: We Just Ran Two Quadrillion Models," Journal of Finance, American Finance Association, vol. 78(1), pages 487-557, February.
    20. Kim, Jang Ho & Han, Jiwoon & Kang, Taehyeon & Fabozzi, Frank J., 2023. "A machine learning approach for comparing the largest firm effect," Emerging Markets Review, Elsevier, vol. 54(C).

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2109.13851. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.