IDEAS home Printed from https://ideas.repec.org/a/eee/jetheo/v189y2020ics0022053120300892.html
   My bibliography  Save this article

Best-response dynamics in zero-sum stochastic games

Author

Listed:
  • Leslie, David S.
  • Perkins, Steven
  • Xu, Zibo

Abstract

We define and analyse three learning dynamics for two-player zero-sum discounted-payoff stochastic games. A continuous-time best-response dynamic in mixed strategies is proved to converge to the set of Nash equilibrium stationary strategies. Extending this, we introduce a fictitious-play-like process in a continuous-time embedding of a stochastic zero-sum game, which is again shown to converge to the set of Nash equilibrium strategies. Finally, we present a modified δ-converging best-response dynamic, in which the discount rate converges to 1, and the learned value converges to the asymptotic value of the zero-sum stochastic game. The critical feature of all the dynamic processes is a separation of adaption rates: beliefs about the value of states adapt more slowly than the strategies adapt, and in the case of the δ-converging dynamic the discount rate adapts more slowly than everything else.

Suggested Citation

  • Leslie, David S. & Perkins, Steven & Xu, Zibo, 2020. "Best-response dynamics in zero-sum stochastic games," Journal of Economic Theory, Elsevier, vol. 189(C).
    • Handle: RePEc:eee:jetheo:v:189:y:2020:i:c:s0022053120300892
    DOI: 10.1016/j.jet.2020.105095
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0022053120300892
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.jet.2020.105095?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Fudenberg, Drew & Levine, David, 1998. "Learning in games," European Economic Review, Elsevier, vol. 42(3-5), pages 631-639, May.
    2. William H. Sandholm, 2001. "Preference Evolution, Two-Speed Dynamics, and Rapid Social Change," Review of Economic Dynamics, Elsevier for the Society for Economic Dynamics, vol. 4(3), pages 637-679, July.
    3. Viossat, Yannick & Zapechelnyuk, Andriy, 2013. "No-regret dynamics and fictitious play," Journal of Economic Theory, Elsevier, vol. 148(2), pages 825-842.
    4. Yehuda Levy, 2013. "Continuous-Time Stochastic Games of Fixed Duration," Dynamic Games and Applications, Springer, vol. 3(2), pages 279-312, June.
    5. Hopkins, Ed, 1999. "A Note on Best Response Dynamics," Games and Economic Behavior, Elsevier, vol. 29(1-2), pages 138-150, October.
    6. Michel Benaim & Josef Hofbauer & Sylvain Sorin, 2005. "Stochastic Approximations and Differential Inclusions II: Applications," Levine's Bibliography 784828000000000098, UCLA Department of Economics.
    7. Fudenberg, Drew & Levine, David K., 2014. "Recency, Consistent Learning, and Nash Equilibrium," Scholarly Articles 13477947, Harvard University Department of Economics.
    8. Neyman, Abraham, 2017. "Continuous-time stochastic games," Games and Economic Behavior, Elsevier, vol. 104(C), pages 92-130.
    9. Gilboa, Itzhak & Matsui, Akihiko, 1991. "Social Stability and Equilibrium," Econometrica, Econometric Society, vol. 59(3), pages 859-867, May.
    10. Michel Benaïm & Josef Hofbauer & Sylvain Sorin, 2005. "Stochastic Approximations and Differential Inclusions; Part II: Applications," Working Papers hal-00242974, HAL.
    11. Leslie, David S. & Collins, E.J., 2006. "Generalised weakened fictitious play," Games and Economic Behavior, Elsevier, vol. 56(2), pages 285-298, August.
    12. Harris, Christopher, 1998. "On the Rate of Convergence of Continuous-Time Fictitious Play," Games and Economic Behavior, Elsevier, vol. 22(2), pages 238-259, February.
    13. Balkenborg, Dieter G. & Hofbauer, Josef & Kuzmics, Christoph, 2013. "Refined best-response correspondence and dynamics," Theoretical Economics, Econometric Society, vol. 8(1), January.
    14. Ely, Jeffrey C. & Yilankaya, Okan, 2001. "Nash Equilibrium and the Evolution of Preferences," Journal of Economic Theory, Elsevier, vol. 97(2), pages 255-272, April.
    15. Johannes Hörner & Takuo Sugaya & Satoru Takahashi & Nicolas Vieille, 2011. "Recursive Methods in Discounted Stochastic Games: An Algorithm for δ→ 1 and a Folk Theorem," Econometrica, Econometric Society, vol. 79(4), pages 1277-1318, July.
    16. V. S. Borkar, 2002. "Reinforcement Learning In Markovian Evolutionary Games," Advances in Complex Systems (ACS), World Scientific Publishing Co. Pte. Ltd., vol. 5(01), pages 55-72.
    17. Vrieze, O.J. & Tijs, S.H., 1982. "Fictitious play applied to sequences of games and discounted stochastic games," Other publications TiSEM da21d287-bc00-4a8e-a18f-0, Tilburg University, School of Economics and Management.
    18. Xu, Zibo, 2016. "Convergence of best-response dynamics in extensive-form games," Journal of Economic Theory, Elsevier, vol. 162(C), pages 21-54.
    19. Truman Bewley & Elon Kohlberg, 1976. "The Asymptotic Theory of Stochastic Games," Mathematics of Operations Research, INFORMS, vol. 1(3), pages 197-208, August.
    20. G. Schoenmakers & J. Flesch & F. Thuijsman, 2007. "Fictitious play in stochastic games," Mathematical Methods of Operations Research, Springer;Gesellschaft für Operations Research (GOR);Nederlands Genootschap voor Besliskunde (NGB), vol. 66(2), pages 315-325, October.
    21. Hofbauer, Josef & Sandholm, William H., 2009. "Stable games and their dynamics," Journal of Economic Theory, Elsevier, vol. 144(4), pages 1665-1693.4, July.
    22. Drew Fudenberg & David K. Levine, 1998. "The Theory of Learning in Games," MIT Press Books, The MIT Press, edition 1, volume 1, number 0262061945, April.
    23. Michael J. Fox & Jeff S. Shamma, 2013. "Population Games, Stable Games, and Passivity," Games, MDPI, vol. 4(4), pages 1-23, October.
    24. Miquel Oliu-Barton, 2014. "The Asymptotic Value in Finite Stochastic Games," Mathematics of Operations Research, INFORMS, vol. 39(3), pages 712-721, August.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Lucas Baudin & Rida Laraki, 2022. "Fictitious Play and Best-Response Dynamics in Identical Interest and Zero Sum Stochastic Games," Post-Print hal-03767937, HAL.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Sandholm, William H., 2015. "Population Games and Deterministic Evolutionary Dynamics," Handbook of Game Theory with Economic Applications,, Elsevier.
    2. Panayotis Mertikopoulos & William H. Sandholm, 2016. "Learning in Games via Reinforcement and Regularization," Mathematics of Operations Research, INFORMS, vol. 41(4), pages 1297-1324, November.
    3. Benaïm, Michel & Hofbauer, Josef & Hopkins, Ed, 2009. "Learning in games with unstable equilibria," Journal of Economic Theory, Elsevier, vol. 144(4), pages 1694-1709, July.
    4. Ewerhart, Christian & Valkanova, Kremena, 2020. "Fictitious play in networks," Games and Economic Behavior, Elsevier, vol. 123(C), pages 182-206.
    5. Saeed Hadikhanloo & Rida Laraki & Panayotis Mertikopoulos & Sylvain Sorin, 2022. "Learning in nonatomic games, part Ⅰ: Finite action spaces and population games," Post-Print hal-03767995, HAL.
    6. Leslie, David S. & Collins, E.J., 2006. "Generalised weakened fictitious play," Games and Economic Behavior, Elsevier, vol. 56(2), pages 285-298, August.
    7. Swenson, Brian & Murray, Ryan & Kar, Soummya, 2020. "Regular potential games," Games and Economic Behavior, Elsevier, vol. 124(C), pages 432-453.
    8. Ratul, Lahkar, 2011. "The dynamic instability of dispersed price equilibria," Journal of Economic Theory, Elsevier, vol. 146(5), pages 1796-1827, September.
    9. Berger, Ulrich, 2007. "Two more classes of games with the continuous-time fictitious play property," Games and Economic Behavior, Elsevier, vol. 60(2), pages 247-261, August.
    10. Cason, Timothy N. & Friedman, Daniel & Hopkins, Ed, 2010. "Testing the TASP: An experimental investigation of learning in games with unstable equilibria," Journal of Economic Theory, Elsevier, vol. 145(6), pages 2309-2331, November.
    11. Heller, Yuval & Mohlin, Erik, 2019. "Coevolution of deception and preferences: Darwin and Nash meet Machiavelli," Games and Economic Behavior, Elsevier, vol. 113(C), pages 223-247.
    12. Ulrich Berger, 2004. "Two More Classes of Games with the Fictitious Play Property," Game Theory and Information 0408003, University Library of Munich, Germany.
    13. Sandholm, William H., 2007. "Evolution in Bayesian games II: Stability of purified equilibria," Journal of Economic Theory, Elsevier, vol. 136(1), pages 641-667, September.
    14. Reinoud Joosten, 2009. "Paul Samuelson's critique and equilibrium concepts in evolutionary game theory," Papers on Economics and Evolution 2009-16, Philipps University Marburg, Department of Geography.
    15. Michel Benaïm & Josef Hofbauer & Sylvain Sorin, 2012. "Perturbations of Set-Valued Dynamical Systems, with Applications to Game Theory," Dynamic Games and Applications, Springer, vol. 2(2), pages 195-205, June.
    16. Mathieu Faure & Gregory Roth, 2010. "Stochastic Approximations of Set-Valued Dynamical Systems: Convergence with Positive Probability to an Attractor," Mathematics of Operations Research, INFORMS, vol. 35(3), pages 624-640, August.
    17. Sandholm,W.H., 2003. "Excess payoff dynamics, potential dynamics, and stable games," Working papers 5, Wisconsin Madison - Social Systems.
    18. Hofbauer, Josef & Hopkins, Ed, 2005. "Learning in perturbed asymmetric games," Games and Economic Behavior, Elsevier, vol. 52(1), pages 133-152, July.
    19. Lahkar, Ratul & Mukherjee, Sayan & Roy, Souvik, 2022. "Generalized perturbed best response dynamics with a continuum of strategies," Journal of Economic Theory, Elsevier, vol. 200(C).
    20. Dai Zusai, 2018. "Evolutionary dynamics in heterogeneous populations: a general framework for an arbitrary type distribution," Papers 1805.04897, arXiv.org, revised May 2019.

    More about this item

    Keywords

    Stochastic games; Best-response dynamics; Zero-sum games; Convergence;
    All these keywords.

    JEL classification:

    • C73 - Mathematical and Quantitative Methods - - Game Theory and Bargaining Theory - - - Stochastic and Dynamic Games; Evolutionary Games

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:jetheo:v:189:y:2020:i:c:s0022053120300892. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/inca/622869 .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.