IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2408.11773.html
   My bibliography  Save this paper

Deviations from the Nash equilibrium and emergence of tacit collusion in a two-player optimal execution game with reinforcement learning

Author

Listed:
  • Fabrizio Lillo
  • Andrea Macr`i

Abstract

The use of reinforcement learning algorithms in financial trading is becoming increasingly prevalent. However, the autonomous nature of these algorithms can lead to unexpected outcomes that deviate from traditional game-theoretical predictions and may even destabilize markets. In this study, we examine a scenario in which two autonomous agents, modeled with Double Deep Q-Learning, learn to liquidate the same asset optimally in the presence of market impact, using the Almgren-Chriss (2000) framework. Our results show that the strategies learned by the agents deviate significantly from the Nash equilibrium of the corresponding market impact game. Notably, the learned strategies exhibit tacit collusion, closely aligning with the Pareto-optimal solution. We further explore how different levels of market volatility influence the agents' performance and the equilibria they discover, including scenarios where volatility differs between the training and testing phases.

Suggested Citation

  • Fabrizio Lillo & Andrea Macr`i, 2024. "Deviations from the Nash equilibrium and emergence of tacit collusion in a two-player optimal execution game with reinforcement learning," Papers 2408.11773, arXiv.org.
    • Handle: RePEc:arx:papers:2408.11773
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2408.11773
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Alexander Schied & Tao Zhang, 2019. "A Market Impact Game Under Transient Price Impact," Mathematics of Operations Research, INFORMS, vol. 44(1), pages 102-121, February.
    2. Matthias Hettich, 2021. "Algorithmic Collusion: Insights from Deep Learning," CQE Working Papers 9421, Center for Quantitative Economics (CQE), University of Muenster.
    3. Alexander Schied & Tao Zhang, 2017. "A State-Constrained Differential Game Arising In Optimal Portfolio Liquidation," Mathematical Finance, Wiley Blackwell, vol. 27(3), pages 779-802, July.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Moritz Voß, 2022. "A two-player portfolio tracking game," Mathematics and Financial Economics, Springer, volume 16, number 6, March.
    2. Moritz Vo{ss}, 2019. "A two-player portfolio tracking game," Papers 1911.05122, arXiv.org, revised Jul 2022.
    3. Samuel Drapeau & Peng Luo & Alexander Schied & Dewen Xiong, 2019. "An FBSDE approach to market impact games with stochastic parameters," Papers 2001.00622, arXiv.org.
    4. Guanxing Fu & Ulrich Horst & Xiaonyu Xia, 2022. "Portfolio liquidation games with self‐exciting order flow," Mathematical Finance, Wiley Blackwell, vol. 32(4), pages 1020-1065, October.
    5. Francesco Cordoni & Fabrizio Lillo, 2022. "Transient impact from the Nash equilibrium of a permanent market impact game," Papers 2205.00494, arXiv.org, revised Mar 2023.
    6. Eyal Neuman & Moritz Voß, 2023. "Trading with the crowd," Mathematical Finance, Wiley Blackwell, vol. 33(3), pages 548-617, July.
    7. Yan Dolinsky & Shir Moshe, 2021. "Utility Indifference Pricing with High Risk Aversion and Small Linear Price Impact," Papers 2111.00451, arXiv.org, revised Jan 2022.
    8. Werner, Tobias, 2021. "Algorithmic and human collusion," DICE Discussion Papers 372, Heinrich Heine University Düsseldorf, Düsseldorf Institute for Competition Economics (DICE).
    9. Masaaki Fujii & Akihiko Takahashi, 2021. "A Mean Field Game Approach to Equilibrium Pricing with Market Clearing Condition," CIRJE F-Series CIRJE-F-1177, CIRJE, Faculty of Economics, University of Tokyo.
    10. Arnoud V. den Boer & Janusz M. Meylahn & Maarten Pieter Schinkel, 2022. "Artificial Collusion: Examining Supracompetitive Pricing by Q-learning Algorithms," Tinbergen Institute Discussion Papers 22-067/VII, Tinbergen Institute.
    11. Eyal Neuman & Moritz Vo{ss}, 2021. "Trading with the Crowd," Papers 2106.09267, arXiv.org, revised Mar 2023.
    12. Dirk Becherer & Todor Bilarev & Peter Frentrup, 2018. "Optimal liquidation under stochastic liquidity," Finance and Stochastics, Springer, vol. 22(1), pages 39-68, January.
    13. Guanxing Fu & Ulrich Horst & Xiaonyu Xia, 2020. "Portfolio Liquidation Games with Self-Exciting Order Flow," Papers 2011.05589, arXiv.org.
    14. Francesco Cordoni & Fabrizio Lillo, 2020. "Instabilities in Multi-Asset and Multi-Agent Market Impact Games," Papers 2004.03546, arXiv.org, revised Nov 2021.
    15. Alexander Schied & Tao Zhang, 2019. "A Market Impact Game Under Transient Price Impact," Mathematics of Operations Research, INFORMS, vol. 44(1), pages 102-121, February.
    16. Masamitsu Ohnishi & Makoto Shimoshimizu, 2022. "Optimal Pair–Trade Execution with Generalized Cross–Impact," Asia-Pacific Financial Markets, Springer;Japanese Association of Financial Economics and Engineering, vol. 29(2), pages 253-289, June.
    17. Xiangge Luo & Alexander Schied, 2018. "Nash equilibrium for risk-averse investors in a market impact game with transient price impact," Papers 1807.03813, arXiv.org, revised Jun 2019.
    18. Masaaki Fujii & Akihiko Takahashi, 2021. "A Mean Field Game Approach to Equilibrium Pricing with Market Clearing Condition," CARF F-Series CARF-F-521, Center for Advanced Research in Finance, Faculty of Economics, The University of Tokyo.
    19. Puru Gupta & Saul D. Jacka, 2023. "Portfolio Choice In Dynamic Thin Markets: Merton Meets Cournot," Papers 2309.16047, arXiv.org.
    20. Epivent, Andréa & Lambin, Xavier, 2024. "On algorithmic collusion and reward–punishment schemes," Economics Letters, Elsevier, vol. 237(C).

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2408.11773. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.