IDEAS home Printed from https://ideas.repec.org/a/the/publsh/3843.html
   My bibliography  Save this article

Equilibrium in misspecified Markov decision processes

Author

Listed:
  • Esponda, Ignacio

    (Department of Economics, University of California, Santa Barbara)

  • Pouzo, Demian

    (Department of Economics, UC Berkeley)

Abstract

We provide an equilibrium framework for modeling the behavior of an agent who holds a simplified view of a dynamic optimization problem. The agent faces a Markov Decision Process, where a transition probability function determines the evolution of a state variable as a function of the previous state and the agent’s action. The agent is uncertain about the true transition function and has a prior over a set of possible transition functions; this set reflects the agent’s (possibly simplified) view of her environment and may not contain the true function. We define an equilibrium concept and provide conditions under which it characterizes steady-state behavior when the agent updates her beliefs using Bayes’ rule.

Suggested Citation

  • Esponda, Ignacio & Pouzo, Demian, 2021. "Equilibrium in misspecified Markov decision processes," Theoretical Economics, Econometric Society, vol. 16(2), May.
    • Handle: RePEc:the:publsh:3843
    as

    Download full text from publisher

    File URL: http://econtheory.org/ojs/index.php/te/article/viewFile/20210717/30649/886
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Joshua Schwartzstein, 2014. "Selective Attention And Learning," Journal of the European Economic Association, European Economic Association, vol. 12(6), pages 1423-1452, December.
    2. Fildes, Robert, 1986. "Sensitivity analyses would help : Edward E. Learner, American Economic Review 75 (1985) 308-313," International Journal of Forecasting, Elsevier, vol. 2(2), pages 237-238.
    3. Philippe Aghion & Patrick Bolton & Christopher Harris & Bruno Jullien, 1991. "Optimal Learning by Experimentation," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 58(4), pages 621-654.
    4. Blume, Lawrence E. & Easley, David, 1982. "Learning to be rational," Journal of Economic Theory, Elsevier, vol. 26(2), pages 340-351, April.
    5. , & ,, 2007. "Valuation equilibrium," Theoretical Economics, Econometric Society, vol. 2(2), June.
    6. Nyarko, Yaw, 1991. "Learning in mis-specified models and the possibility of cycles," Journal of Economic Theory, Elsevier, vol. 55(2), pages 416-427, December.
    7. Fudenberg, Drew & Levine, David K, 1993. "Self-Confirming Equilibrium," Econometrica, Econometric Society, vol. 61(3), pages 523-545, May.
    8. Erik Eyster & Matthew Rabin, 2005. "Cursed Equilibrium," Econometrica, Econometric Society, vol. 73(5), pages 1623-1672, September.
    9. Michele Piccione & Ariel Rubinstein, 2003. "Modeling the Economic Interaction of Agents With Diverse Abilities to Recognize Equilibrium Patterns," Journal of the European Economic Association, MIT Press, vol. 1(1), pages 212-223, March.
    10. Jehiel, Philippe, 2005. "Analogy-based expectation equilibrium," Journal of Economic Theory, Elsevier, vol. 123(2), pages 81-104, August.
    11. Rothschild, Michael, 1974. "A two-armed bandit theory of market pricing," Journal of Economic Theory, Elsevier, vol. 9(2), pages 185-202, October.
    12. Kalai, Ehud & Lehrer, Ehud, 1993. "Rational Learning Leads to Nash Equilibrium," Econometrica, Econometric Society, vol. 61(5), pages 1019-1045, September.
    13. Dekel, Eddie & Fudenberg, Drew & Levine, David K., 2004. "Learning to play Bayesian games," Games and Economic Behavior, Elsevier, vol. 46(2), pages 282-303, February.
    14. Enriqueta Aragones & Itzhak Gilboa & Andrew Postlewaite & David Schmeidler, 2012. "Fact-Free Learning," World Scientific Book Chapters, in: Case-Based Predictions An Axiomatic Approach to Prediction, Classification and Statistical Learning, chapter 8, pages 185-210, World Scientific Publishing Co. Pte. Ltd..
    15. Fudenberg, Drew & Levine, David K, 1993. "Steady State Learning and Nash Equilibrium," Econometrica, Econometric Society, vol. 61(3), pages 547-573, May.
    16. Barberis, Nicholas & Shleifer, Andrei & Vishny, Robert, 1998. "A model of investor sentiment," Journal of Financial Economics, Elsevier, vol. 49(3), pages 307-343, September.
    17. McLennan, Andrew, 1984. "Price dispersion and incomplete learning in the long run," Journal of Economic Dynamics and Control, Elsevier, vol. 7(3), pages 331-347, September.
    18. Ignacio Esponda, 2008. "Behavioral Equilibrium in Economies with Adverse Selection," American Economic Review, American Economic Association, vol. 98(4), pages 1269-1291, September.
    19. Fudenberg Drew & Kreps David M., 1993. "Learning Mixed Equilibria," Games and Economic Behavior, Elsevier, vol. 5(3), pages 320-367, July.
    20. Osborne, Martin J & Rubinstein, Ariel, 1998. "Games with Procedurally Rational Players," American Economic Review, American Economic Association, vol. 88(4), pages 834-847, September.
    21. , & ,, 2010. "A theory of regular Markov perfect equilibria in dynamic stochastic games: genericity, stability, and purification," Theoretical Economics, Econometric Society, vol. 5(3), September.
    22. Bray, Margaret, 1982. "Learning, estimation, and the stability of rational expectations," Journal of Economic Theory, Elsevier, vol. 26(2), pages 318-339, April.
    23. Blume, Lawrence E. & Easley, David, 1984. "Rational expectations equilibrium: An alternative approach," Journal of Economic Theory, Elsevier, vol. 34(1), pages 116-129, October.
    24. Sobel, Joel, 1984. "Non-linear prices and price-taking behavior," Journal of Economic Behavior & Organization, Elsevier, vol. 5(3-4), pages 387-396.
    25. Nabil I. Al-Najjar, 2009. "Decision Makers as Statisticians: Diversity, Ambiguity, and Learning," Econometrica, Econometric Society, vol. 77(5), pages 1371-1401, September.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Esponda, Ignacio & Pouzo, Demian & Yamamoto, Yuichi, 2021. "Asymptotic behavior of Bayesian learners with misspecified models," Journal of Economic Theory, Elsevier, vol. 195(C).
    2. Yingkai Li & Aleksandrs Slivkins, 2022. "Exploration and Incentivizing Participation in Randomized Trials," Papers 2202.06191, arXiv.org, revised Mar 2025.
    3. Fudenberg, Drew & Romanyuk, Gleb & Strack, Philipp, 2017. "Active learning with a misspecified prior," Theoretical Economics, Econometric Society, vol. 12(3), September.
    4. Anderson, Robert M. & Duanmu, Haosui & Ghosh, Aniruddha & Khan, M. Ali, 2024. "On existence of Berk-Nash equilibria in misspecified Markov decision processes with infinite spaces," Journal of Economic Theory, Elsevier, vol. 217(C).
    5. Thomas J. Sargent & John Stachurski, 2024. "Dynamic Programming: Finite States," Papers 2401.10473, arXiv.org.
    6. Esponda, Ignacio & Pouzo, Demian & Yamamoto, Yuichi, 2021. "Asymptotic behavior of Bayesian learners with misspecified models," Journal of Economic Theory, Elsevier, vol. 195(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Ignacio Esponda & Demian Pouzo, 2016. "Berk–Nash Equilibrium: A Framework for Modeling Agents With Misspecified Models," Econometrica, Econometric Society, vol. 84, pages 1093-1130, May.
    2. Ignacio Esponda & Demian Pouzo, 2016. "Berk–Nash Equilibrium: A Framework for Modeling Agents With Misspecified Models," Econometrica, Econometric Society, vol. 84, pages 1093-1130, May.
    3. Esponda, Ignacio & Pouzo, Demian & Yamamoto, Yuichi, 2021. "Asymptotic behavior of Bayesian learners with misspecified models," Journal of Economic Theory, Elsevier, vol. 195(C).
    4. Esponda, Ignacio & Pouzo, Demian & Yamamoto, Yuichi, 2021. "Asymptotic behavior of Bayesian learners with misspecified models," Journal of Economic Theory, Elsevier, vol. 195(C).
    5. Philippe Jehiel, 2022. "Analogy-Based Expectation Equilibrium and Related Concepts:Theory, Applications, and Beyond," PSE Working Papers halshs-03735680, HAL.
    6. Topi Miettinen, 2012. "Paying attention to payoffs in analogy-based learning," Economic Theory, Springer;Society for the Advancement of Economic Theory (SAET), vol. 50(1), pages 193-222, May.
    7. Christoph March, 2011. "Adaptive social learning," PSE Working Papers halshs-00572528, HAL.
    8. Fudenberg, Drew & Romanyuk, Gleb & Strack, Philipp, 2017. "Active learning with a misspecified prior," Theoretical Economics, Econometric Society, vol. 12(3), September.
    9. Mario Gilli, 2002. "Rational Learning in Imperfect Monitoring Games," Working Papers 46, University of Milano-Bicocca, Department of Economics, revised Mar 2002.
    10. Jean-Michel Grandmont, 1998. "Expectations Formation and Stability of Large Socioeconomic Systems," Econometrica, Econometric Society, vol. 66(4), pages 741-782, July.
    11. Sobel, Joel, 2000. "Economists' Models of Learning," Journal of Economic Theory, Elsevier, vol. 94(2), pages 241-261, October.
    12. S. Nageeb Ali, 2011. "Learning Self-Control," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 126(2), pages 857-893.
    13. Philippe Jehiel & Erik Mohlin, 2023. "Categorization in Games: A Bias-Variance Perspective," Working Papers halshs-04154272, HAL.
    14. Zacharias Maniadis, 2014. "Selective revelation of public information and self-confirming equilibrium," International Journal of Game Theory, Springer;Game Theory Society, vol. 43(4), pages 991-1008, November.
    15. Ran Spiegler, 2016. "Bayesian Networks and Boundedly Rational Expectations," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 131(3), pages 1243-1290.
    16. Liu, Zhen, 2016. "Games with incomplete information when players are partially aware of others’ signals," Journal of Mathematical Economics, Elsevier, vol. 65(C), pages 58-70.
    17. Mira Frick & Ryota Iijima & Yuhta Ishii, 2020. "Stability and Robustness in Misspecified Learning Models," Cowles Foundation Discussion Papers 2235, Cowles Foundation for Research in Economics, Yale University.
    18. Mira Frick & Ryota Iijima & Yuhta Ishii, 2020. "Belief Convergence under Misspecified Learning: A Martingale Approach," Cowles Foundation Discussion Papers 2235R2, Cowles Foundation for Research in Economics, Yale University, revised Dec 2021.
    19. Manxi Wu & Saurabh Amin & Asuman Ozdaglar, 2021. "Multi-agent Bayesian Learning with Best Response Dynamics: Convergence and Stability," Papers 2109.00719, arXiv.org.
    20. Miettinen, Topi, 2009. "The partially cursed and the analogy-based expectation equilibrium," Economics Letters, Elsevier, vol. 105(2), pages 162-164, November.

    More about this item

    Keywords

    Misspecified model; Markov decision process; equilibrium;
    All these keywords.

    JEL classification:

    • C61 - Mathematical and Quantitative Methods - - Mathematical Methods; Programming Models; Mathematical and Simulation Modeling - - - Optimization Techniques; Programming Models; Dynamic Analysis
    • D83 - Microeconomics - - Information, Knowledge, and Uncertainty - - - Search; Learning; Information and Knowledge; Communication; Belief; Unawareness

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:the:publsh:3843. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Martin J. Osborne (email available below). General contact details of provider: http://econtheory.org .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.