IDEAS home Printed from https://ideas.repec.org/a/spr/dyngam/v13y2023i1d10.1007_s13235-022-00450-2.html
   My bibliography  Save this article

Q-Learning in Regularized Mean-field Games

Author

Listed:
  • Berkay Anahtarci

    (Özyeğin University)

  • Can Deha Kariksiz

    (Özyeğin University)

  • Naci Saldi

    (Bilkent University)

Abstract

In this paper, we introduce a regularized mean-field game and study learning of this game under an infinite-horizon discounted reward function. Regularization is introduced by adding a strongly concave regularization function to the one-stage reward function in the classical mean-field game model. We establish a value iteration based learning algorithm to this regularized mean-field game using fitted Q-learning. The regularization term in general makes reinforcement learning algorithm more robust to the system components. Moreover, it enables us to establish error analysis of the learning algorithm without imposing restrictive convexity assumptions on the system components, which are needed in the absence of a regularization term.

Suggested Citation

  • Berkay Anahtarci & Can Deha Kariksiz & Naci Saldi, 2023. "Q-Learning in Regularized Mean-field Games," Dynamic Games and Applications, Springer, vol. 13(1), pages 89-117, March.
    • Handle: RePEc:spr:dyngam:v:13:y:2023:i:1:d:10.1007_s13235-022-00450-2
    DOI: 10.1007/s13235-022-00450-2
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s13235-022-00450-2
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s13235-022-00450-2?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Piotr Więcek & Eitan Altman, 2015. "Stationary Anonymous Sequential Games with Undiscounted Rewards," Journal of Optimization Theory and Applications, Springer, vol. 166(2), pages 686-710, August.
    2. A. Bensoussan & K. Sung & S. Yam, 2013. "Linear–Quadratic Time-Inconsistent Mean Field Games," Dynamic Games and Applications, Springer, vol. 3(4), pages 537-552, December.
    3. Naci Saldi & Tamer Başar & Maxim Raginsky, 2019. "Approximate Nash Equilibria in Partially Observed Stochastic Games with Mean-Field Interactions," Mathematics of Operations Research, INFORMS, vol. 44(3), pages 1006-1033, August.
    4. Diogo Gomes & João Saúde, 2014. "Mean Field Games Models—A Brief Survey," Dynamic Games and Applications, Springer, vol. 4(2), pages 110-154, June.
    5. Adlakha, Sachin & Johari, Ramesh & Weintraub, Gabriel Y., 2015. "Equilibria of dynamic games with many players: Existence, approximation, and market structure," Journal of Economic Theory, Elsevier, vol. 156(C), pages 269-316.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Naci Saldi & Tamer Bas¸ ar & Maxim Raginsky, 2020. "Approximate Markov-Nash Equilibria for Discrete-Time Risk-Sensitive Mean-Field Games," Mathematics of Operations Research, INFORMS, vol. 45(4), pages 1596-1620, November.
    2. Naci Saldi & Tamer Başar & Maxim Raginsky, 2023. "Partially Observed Discrete-Time Risk-Sensitive Mean Field Games," Dynamic Games and Applications, Springer, vol. 13(3), pages 929-960, September.
    3. Naci Saldi & Tamer Başar & Maxim Raginsky, 2019. "Approximate Nash Equilibria in Partially Observed Stochastic Games with Mean-Field Interactions," Mathematics of Operations Research, INFORMS, vol. 44(3), pages 1006-1033, August.
    4. Régis Chenavaz & Corina Paraschiv & Gabriel Turinici, 2021. "Dynamic Pricing of New Products in Competitive Markets: A Mean-Field Game Approach," Dynamic Games and Applications, Springer, vol. 11(3), pages 463-490, September.
    5. Piotr Więcek, 2020. "Discrete-Time Ergodic Mean-Field Games with Average Reward on Compact Spaces," Dynamic Games and Applications, Springer, vol. 10(1), pages 222-256, March.
    6. Piotr Więcek, 2017. "Total Reward Semi-Markov Mean-Field Games with Complementarity Properties," Dynamic Games and Applications, Springer, vol. 7(3), pages 507-529, September.
    7. Paulo B. Brito, 2022. "The dynamics of growth and distribution in a spatially heterogeneous world," Portuguese Economic Journal, Springer;Instituto Superior de Economia e Gestao, vol. 21(3), pages 311-350, September.
    8. Régis Chenavaz & Corina Paraschiv & Gabriel Turinici, 2017. "Dynamic Pricing of New Products in Competitive Markets: A Mean-Field Game Approach," Working Papers hal-01592958, HAL.
    9. Light, Bar & Weintraub, Gabriel, 2018. "Mean Field Equilibrium: Uniqueness, Existence, and Comparative Statics," Research Papers 3731, Stanford University, Graduate School of Business.
    10. Romuald Élie & Emma Hubert & Thibaut Mastrolia & Dylan Possamaï, 2021. "Mean–field moral hazard for optimal energy demand response management," Mathematical Finance, Wiley Blackwell, vol. 31(1), pages 399-473, January.
    11. Fu, Guanxing & Horst, Ulrich, 2017. "Mean Field Games with Singular Controls," Rationality and Competition Discussion Paper Series 22, CRC TRR 190 Rationality and Competition.
    12. Vassili N. Kolokoltsov, 2021. "Quantum Mean-Field Games with the Observations of Counting Type," Games, MDPI, vol. 12(1), pages 1-14, January.
    13. Qinglong Zhou & Gaofeng Zong, 2016. "Time-Inconsistent Stochastic Linear-quadratic Differential Game," Papers 1607.00638, arXiv.org.
    14. Dianetti, Jodi & Ferrari, Giorgio & Fischer, Markus & Nendel, Max, 2022. "A Unifying Framework for Submodular Mean Field Games," Center for Mathematical Economics Working Papers 661, Center for Mathematical Economics, Bielefeld University.
    15. Alain Bensoussan & Guiyuan Ma & Chi Chung Siu & Sheung Chi Phillip Yam, 2022. "Dynamic mean–variance problem with frictions," Finance and Stochastics, Springer, vol. 26(2), pages 267-300, April.
    16. Aïd, René & Basei, Matteo & Ferrari, Giorgio, 2023. "A Stationary Mean-Field Equilibrium Model of Irreversible Investment in a Two-Regime Economy," Center for Mathematical Economics Working Papers 679, Center for Mathematical Economics, Bielefeld University.
    17. Dianetti, Jodi & Ferrari, Giorgio & Tzouanas, Ioannis, 2023. "Ergodic Mean-Field Games of Singular Control with Regime-Switching (extended version)," Center for Mathematical Economics Working Papers 681, Center for Mathematical Economics, Bielefeld University.
    18. Noha Almulla & Rita Ferreira & Diogo Gomes, 2017. "Two Numerical Approaches to Stationary Mean-Field Games," Dynamic Games and Applications, Springer, vol. 7(4), pages 657-682, December.
    19. Arvind V. Shrivats & Dena Firoozi & Sebastian Jaimungal, 2022. "A mean‐field game approach to equilibrium pricing in solar renewable energy certificate markets," Mathematical Finance, Wiley Blackwell, vol. 32(3), pages 779-824, July.
    20. Deepanshu Vasal & Randall Berry, 2022. "Master Equation for Discrete-Time Stackelberg Mean Field Games with single leader," Papers 2201.05959, arXiv.org.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:dyngam:v:13:y:2023:i:1:d:10.1007_s13235-022-00450-2. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.