IDEAS home Printed from https://ideas.repec.org/p/arx/papers/1812.06600.html
   My bibliography  Save this paper

Double Deep Q-Learning for Optimal Execution

Author

Listed:
  • Brian Ning
  • Franco Ho Ting Lin
  • Sebastian Jaimungal

Abstract

Optimal trade execution is an important problem faced by essentially all traders. Much research into optimal execution uses stringent model assumptions and applies continuous time stochastic control to solve them. Here, we instead take a model free approach and develop a variation of Deep Q-Learning to estimate the optimal actions of a trader. The model is a fully connected Neural Network trained using Experience Replay and Double DQN with input features given by the current state of the limit order book, other trading signals, and available execution actions, while the output is the Q-value function estimating the future rewards under an arbitrary action. We apply our model to nine different stocks and find that it outperforms the standard benchmark approach on most stocks using the measures of (i) mean and median out-performance, (ii) probability of out-performance, and (iii) gain-loss ratios.

Suggested Citation

  • Brian Ning & Franco Ho Ting Lin & Sebastian Jaimungal, 2018. "Double Deep Q-Learning for Optimal Execution," Papers 1812.06600, arXiv.org, revised Jun 2020.
  • Handle: RePEc:arx:papers:1812.06600
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/1812.06600
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Ole E. Barndorff-Nielsen & Neil Shephard, 2002. "Estimating quadratic variation using realized variance," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 17(5), pages 457-477.
    2. Olivier Guéant & Charles-Albert Lehalle, 2015. "General Intensity Shapes In Optimal Liquidation," Mathematical Finance, Wiley Blackwell, vol. 25(3), pages 457-495, July.
    3. Philippe Casgrain & Sebastian Jaimungal, 2018. "Trading algorithms with learning in latent alpha models," Papers 1806.04472, arXiv.org.
    4. Olivier Guéant & Charles-Albert Lehalle, 2015. "General Intensity Shapes In Optimal Liquidation," Mathematical Finance, Wiley Blackwell, vol. 25(3), pages 457-495, July.
    5. Dieter Hendricks & Diane Wilcox, 2014. "A reinforcement learning extension to the Almgren-Chriss model for optimal trade execution," Papers 1403.2229, arXiv.org.
    6. Ole E. Barndorff-Nielsen & Neil Shephard, 2002. "Estimating quadratic variation using realized variance," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 17(5), pages 457-477.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Schnaubelt, Matthias, 2020. "Deep reinforcement learning for the optimal placement of cryptocurrency limit orders," FAU Discussion Papers in Economics 05/2020, Friedrich-Alexander University Erlangen-Nuremberg, Institute for Economics.
    2. Han, Xuefeng & He, Hongwen & Wu, Jingda & Peng, Jiankun & Li, Yuecheng, 2019. "Energy management based on reinforcement learning with double deep Q-learning for a hybrid electric tracked vehicle," Applied Energy, Elsevier, vol. 254(C).
    3. O'Neill, Kay & Burrell, Lori & Peplinski, Kyle & Korfmacher, Jon & Spinosa, Ciara Z. & McGready, John & Duggan, Anne, 2023. "Early childhood home visiting’s initial transition to virtual visits in response to the COVID-19 pandemic," Children and Youth Services Review, Elsevier, vol. 155(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Yuchen Fang & Kan Ren & Weiqing Liu & Dong Zhou & Weinan Zhang & Jiang Bian & Yong Yu & Tie-Yan Liu, 2021. "Universal Trading for Order Execution with Oracle Policy Distillation," Papers 2103.10860, arXiv.org.
    2. Campi, Luciano & Zabaljauregui, Diego, 2020. "Optimal market making under partial information with general intensities," LSE Research Online Documents on Economics 104612, London School of Economics and Political Science, LSE Library.
    3. Härdle, Wolfgang Karl & Hautsch, Nikolaus & Pigorsch, Uta, 2008. "Measuring and modeling risk using high-frequency data," SFB 649 Discussion Papers 2008-045, Humboldt University Berlin, Collaborative Research Center 649: Economic Risk.
    4. Eleftheria Kafousaki & Stavros Degiannakis, 2023. "Forecasting VIX: the illusion of forecast evaluation criteria," Economics and Business Letters, Oviedo University Press, vol. 12(3), pages 231-240.
    5. Lee, Hwang Hee & Hyun, Jung-Soon, 2019. "The asymmetric effect of equity volatility on credit default swap spreads," Journal of Banking & Finance, Elsevier, vol. 98(C), pages 125-136.
    6. Diego Amaya & Jean-François Bégin & Geneviève Gauthier, 2022. "The Informational Content of High-Frequency Option Prices," Management Science, INFORMS, vol. 68(3), pages 2166-2201, March.
    7. Ozcan Ceylan, 2015. "Limited information-processing capacity and asymmetric stock correlations," Quantitative Finance, Taylor & Francis Journals, vol. 15(6), pages 1031-1039, June.
    8. Kamiar Mohaddes & M. Hashem Pesaran, 2013. "One Hundred Years of Oil Income and the Iranian Economy: A Curse or a Blessing?," CESifo Working Paper Series 4118, CESifo.
    9. Yusaku Nishimura & Xuyi Dong & Bianxia Sun, 2021. "Trump's tweets: Sentiment, stock market volatility, and jumps," Journal of Financial Research, Southern Finance Association;Southwestern Finance Association, vol. 44(3), pages 497-512, September.
    10. Ryan Donnelly & Zi Li, 2022. "Dynamic Inventory Management with Mean-Field Competition," Papers 2210.17208, arXiv.org, revised Apr 2025.
    11. Philippe Bergault & Olivier Guéant, 2021. "Size matters for OTC market makers: General results and dimensionality reduction techniques," Mathematical Finance, Wiley Blackwell, vol. 31(1), pages 279-322, January.
    12. Bollerslev, Tim & Kretschmer, Uta & Pigorsch, Christian & Tauchen, George, 2009. "A discrete-time model for daily S & P500 returns and realized variations: Jumps and leverage effects," Journal of Econometrics, Elsevier, vol. 150(2), pages 151-166, June.
    13. Lu, Ran & Xu, Wen & Zeng, Hongjun & Zhou, Xiangjing, 2023. "Volatility connectedness among the Indian equity and major commodity markets under the COVID-19 scenario," Economic Analysis and Policy, Elsevier, vol. 78(C), pages 1465-1481.
    14. Jin, Miao & Liu, Yu-Jane & Meng, Juanjuan, 2019. "Fat-finger event and risk-taking behavior," Journal of Empirical Finance, Elsevier, vol. 53(C), pages 126-143.
    15. Jim Griffin & Jia Liu & John M. Maheu, 2021. "Bayesian Nonparametric Estimation of Ex Post Variance [Out of Sample Forecasts of Quadratic Variation]," Journal of Financial Econometrics, Oxford University Press, vol. 19(5), pages 823-859.
    16. Papantonis, Ioannis & Rompolis, Leonidas & Tzavalis, Elias, 2023. "Improving variance forecasts: The role of Realized Variance features," International Journal of Forecasting, Elsevier, vol. 39(3), pages 1221-1237.
    17. Cesa-Bianchi, Ambrogio & Pesaran, M. Hashem & Rebucci, Alessandro, 2014. "Uncertainty and Economic Activity: A Global Perspective," IDB Publications (Working Papers) 6605, Inter-American Development Bank.
    18. Zdravetz Lazarov, 2005. "Assesing the Economic Significance of the Intra-daily Volatility Seasonalities," School of Economics and Finance Discussion Papers and Working Papers Series 203, School of Economics and Finance, Queensland University of Technology.
    19. Philippe Bergault & David Evangelista & Olivier Gu'eant & Douglas Vieira, 2018. "Closed-form approximations in multi-asset market making," Papers 1810.04383, arXiv.org, revised Sep 2022.
    20. Lux, Thomas & Morales-Arias, Leonardo & Sattarhoff, Cristina, 2011. "A Markov-switching multifractal approach to forecasting realized volatility," Kiel Working Papers 1737, Kiel Institute for the World Economy (IfW Kiel).

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:1812.06600. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.