IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2103.10860.html
   My bibliography  Save this paper

Universal Trading for Order Execution with Oracle Policy Distillation

Author

Listed:
  • Yuchen Fang
  • Kan Ren
  • Weiqing Liu
  • Dong Zhou
  • Weinan Zhang
  • Jiang Bian
  • Yong Yu
  • Tie-Yan Liu

Abstract

As a fundamental problem in algorithmic trading, order execution aims at fulfilling a specific trading order, either liquidation or acquirement, for a given instrument. Towards effective execution strategy, recent years have witnessed the shift from the analytical view with model-based market assumptions to model-free perspective, i.e., reinforcement learning, due to its nature of sequential decision optimization. However, the noisy and yet imperfect market information that can be leveraged by the policy has made it quite challenging to build up sample efficient reinforcement learning methods to achieve effective order execution. In this paper, we propose a novel universal trading policy optimization framework to bridge the gap between the noisy yet imperfect market states and the optimal action sequences for order execution. Particularly, this framework leverages a policy distillation method that can better guide the learning of the common policy towards practically optimal execution by an oracle teacher with perfect information to approximate the optimal trading strategy. The extensive experiments have shown significant improvements of our method over various strong baselines, with reasonable trading actions.

Suggested Citation

  • Yuchen Fang & Kan Ren & Weiqing Liu & Dong Zhou & Weinan Zhang & Jiang Bian & Yong Yu & Tie-Yan Liu, 2021. "Universal Trading for Order Execution with Oracle Policy Distillation," Papers 2103.10860, arXiv.org.
    • Handle: RePEc:arx:papers:2103.10860
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2103.10860
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. �lvaro Cartea & Sebastian Jaimungal, 2015. "Optimal execution with limit and market orders," Quantitative Finance, Taylor & Francis Journals, vol. 15(8), pages 1279-1291, August.
    2. Olivier Guéant & Charles-Albert Lehalle, 2015. "General Intensity Shapes In Optimal Liquidation," Mathematical Finance, Wiley Blackwell, vol. 25(3), pages 457-495, July.
    3. John Moody & Lizhong Wu, "undated". "Optimization of Trading Systems and Portfolios," Computing in Economics and Finance 1997 55, Society for Computational Economics.
    4. Bialkowski, Jedrzej & Darolles, Serge & Le Fol, Gaëlle, 2008. "Improving VWAP strategies: A dynamic volume approach," Journal of Banking & Finance, Elsevier, vol. 32(9), pages 1709-1722, September.
    5. Volodymyr Mnih & Koray Kavukcuoglu & David Silver & Andrei A. Rusu & Joel Veness & Marc G. Bellemare & Alex Graves & Martin Riedmiller & Andreas K. Fidjeland & Georg Ostrovski & Stig Petersen & Charle, 2015. "Human-level control through deep reinforcement learning," Nature, Nature, vol. 518(7540), pages 529-533, February.
    6. Philippe Casgrain & Sebastian Jaimungal, 2019. "Trading algorithms with learning in latent alpha models," Mathematical Finance, Wiley Blackwell, vol. 29(3), pages 735-772, July.
    7. Yunan Ye & Hengzhi Pei & Boxin Wang & Pin-Yu Chen & Yada Zhu & Jun Xiao & Bo Li, 2020. "Reinforcement-Learning based Portfolio Management with Augmented Asset Movement Prediction States," Papers 2002.05780, arXiv.org.
    8. Olivier Guéant & Charles-Albert Lehalle, 2015. "General Intensity Shapes In Optimal Liquidation," Mathematical Finance, Wiley Blackwell, vol. 25(3), pages 457-495, July.
    9. Dieter Hendricks & Diane Wilcox, 2014. "A reinforcement learning extension to the Almgren-Chriss model for optimal trade execution," Papers 1403.2229, arXiv.org.
    10. Bertsimas, Dimitris & Lo, Andrew W., 1998. "Optimal control of execution costs," Journal of Financial Markets, Elsevier, vol. 1(1), pages 1-50, April.
    11. David Silver & Aja Huang & Chris J. Maddison & Arthur Guez & Laurent Sifre & George van den Driessche & Julian Schrittwieser & Ioannis Antonoglou & Veda Panneershelvam & Marc Lanctot & Sander Dieleman, 2016. "Mastering the game of Go with deep neural networks and tree search," Nature, Nature, vol. 529(7587), pages 484-489, January.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Xiaodong Li & Pangjing Wu & Chenxin Zou & Qing Li, 2022. "Hierarchical Deep Reinforcement Learning for VWAP Strategy Optimization," Papers 2212.14670, arXiv.org.
    2. Zitao Song & Xuyang Jin & Chenliang Li, 2022. "Safe-FinRL: A Low Bias and Variance Deep Reinforcement Learning Implementation for High-Freq Stock Trading," Papers 2206.05910, arXiv.org.
    3. Xianfeng Jiao & Zizhong Li & Chang Xu & Yang Liu & Weiqing Liu & Jiang Bian, 2023. "Microstructure-Empowered Stock Factor Extraction and Utilization," Papers 2308.08135, arXiv.org.
    4. Shuo Sun & Molei Qin & Xinrun Wang & Bo An, 2023. "PRUDEX-Compass: Towards Systematic Evaluation of Reinforcement Learning in Financial Markets," Papers 2302.00586, arXiv.org, revised Mar 2023.
    5. Feiyang Pan & Tongzhe Zhang & Ling Luo & Jia He & Shuoling Liu, 2022. "Learn Continuously, Act Discretely: Hybrid Action-Space Reinforcement Learning For Optimal Execution," Papers 2207.11152, arXiv.org.
    6. Shuo Sun & Rundong Wang & Bo An, 2021. "Reinforcement Learning for Quantitative Trading," Papers 2109.13851, arXiv.org.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Shuo Sun & Rundong Wang & Bo An, 2021. "Reinforcement Learning for Quantitative Trading," Papers 2109.13851, arXiv.org.
    2. Olivier Guéant, 2016. "The Financial Mathematics of Market Liquidity: From Optimal Execution to Market Making," Post-Print hal-01393136, HAL.
    3. Schnaubelt, Matthias, 2022. "Deep reinforcement learning for the optimal placement of cryptocurrency limit orders," European Journal of Operational Research, Elsevier, vol. 296(3), pages 993-1006.
    4. Fayc{c}al Drissi, 2022. "Solvability of Differential Riccati Equations and Applications to Algorithmic Trading with Signals," Papers 2202.07478, arXiv.org, revised Aug 2023.
    5. Campi, Luciano & Zabaljauregui, Diego, 2020. "Optimal market making under partial information with general intensities," LSE Research Online Documents on Economics 104612, London School of Economics and Political Science, LSE Library.
    6. Xiaoyue Li & John M. Mulvey, 2023. "Optimal Portfolio Execution in a Regime-switching Market with Non-linear Impact Costs: Combining Dynamic Program and Neural Network," Papers 2306.08809, arXiv.org.
    7. Woo Jae Byun & Bumkyu Choi & Seongmin Kim & Joohyun Jo, 2023. "Practical Application of Deep Reinforcement Learning to Optimal Trade Execution," FinTech, MDPI, vol. 2(3), pages 1-16, June.
    8. Ryan Donnelly & Zi Li, 2022. "Dynamic Inventory Management with Mean-Field Competition," Papers 2210.17208, arXiv.org.
    9. Rama Cont & Arseniy Kukanov, 2012. "Optimal order placement in limit order markets," Papers 1210.1625, arXiv.org, revised Nov 2014.
    10. Yuchao Dong, 2022. "Randomized Optimal Stopping Problem in Continuous time and Reinforcement Learning Algorithm," Papers 2208.02409, arXiv.org, revised Sep 2023.
    11. Alvaro Cartea & Luhui Gan & Sebastian Jaimungal, 2018. "Trading Cointegrated Assets with Price Impact," Papers 1807.01428, arXiv.org.
    12. Alexandre Roch, 2023. "Optimal Liquidation Through a Limit Order Book: A Neural Network and Simulation Approach," Methodology and Computing in Applied Probability, Springer, vol. 25(1), pages 1-29, March.
    13. Soohan Kim & Jimyeong Kim & Hong Kee Sul & Youngjoon Hong, 2023. "An Adaptive Dual-level Reinforcement Learning Approach for Optimal Trade Execution," Papers 2307.10649, arXiv.org.
    14. Rossella Agliardi & Ramazan Gençay, 2017. "Optimal Trading Strategies With Limit Orders," International Journal of Theoretical and Applied Finance (IJTAF), World Scientific Publishing Co. Pte. Ltd., vol. 20(01), pages 1-16, February.
    15. Sadoghi, Amirhossein & Vecer, Jan, 2022. "Optimal liquidation problem in illiquid markets," European Journal of Operational Research, Elsevier, vol. 296(3), pages 1050-1066.
    16. Amirhossein Sadoghi & Jan Vecer, 2022. "Optimal liquidation problem in illiquid markets," Post-Print hal-03696768, HAL.
    17. Schnaubelt, Matthias, 2020. "Deep reinforcement learning for the optimal placement of cryptocurrency limit orders," FAU Discussion Papers in Economics 05/2020, Friedrich-Alexander University Erlangen-Nuremberg, Institute for Economics.
    18. Haoran Wang & Xun Yu Zhou, 2019. "Continuous-Time Mean-Variance Portfolio Selection: A Reinforcement Learning Framework," Papers 1904.11392, arXiv.org, revised May 2019.
    19. Philippe Bergault & Fayc{c}al Drissi & Olivier Gu'eant, 2021. "Multi-asset optimal execution and statistical arbitrage strategies under Ornstein-Uhlenbeck dynamics," Papers 2103.13773, arXiv.org, revised Mar 2022.
    20. Dupret, Jean-Loup & Hainaut, Donatien, 2023. "Optimal liquidation under indirect price impact with propagator," LIDAM Discussion Papers ISBA 2023012, Université catholique de Louvain, Institute of Statistics, Biostatistics and Actuarial Sciences (ISBA).

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2103.10860. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.