IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2407.04227.html
   My bibliography  Save this paper

Simple method for efficiently solving dynamic models with continuous actions using policy gradient

Author

Listed:
  • Takeshi Fukasawa

Abstract

This study proposes the Value Function-Policy Gradient Iteration-Spectral (VF-PGI-Spectral) algorithm, which efficiently solves discrete-time infinite-horizon dynamic models with continuous actions. It combines the spectral algorithm to accelerate convergence. The method is applicable not only to single-agent dynamic optimization problems, but also to multi-agent dynamic games, which previously proposed methods cannot deal with. Moreover, the proposed algorithm is not limited to models with specific functional forms, and applies to models with multiple continuous actions. This study shows the results of numerical experiments, showing the effective performance of the proposed algorithm.

Suggested Citation

  • Takeshi Fukasawa, 2024. "Simple method for efficiently solving dynamic models with continuous actions using policy gradient," Papers 2407.04227, arXiv.org.
    • Handle: RePEc:arx:papers:2407.04227
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2407.04227
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Arellano, Cristina & Maliar, Lilia & Maliar, Serguei & Tsyrennikov, Viktor, 2016. "Envelope condition method with an application to default risk models," Journal of Economic Dynamics and Control, Elsevier, vol. 69(C), pages 436-459.
    2. Iskhakov, Fedor, 2015. "Multidimensional endogenous gridpoint method: Solving triangular dynamic stochastic optimization problems without root-finding operations," Economics Letters, Elsevier, vol. 135(C), pages 72-76.
    3. Ariel Pakes & Paul McGuire, 1994. "Computing Markov-Perfect Nash Equilibria: Numerical Implications of a Dynamic Differentiated Product Model," RAND Journal of Economics, The RAND Corporation, vol. 25(4), pages 555-589, Winter.
    4. Ronald L. Goettler & Brett R. Gordon, 2011. "Does AMD Spur Intel to Innovate More?," Journal of Political Economy, University of Chicago Press, vol. 119(6), pages 1141-1200.
    5. Kenneth L. Judd, 1998. "Numerical Methods in Economics," MIT Press Books, The MIT Press, edition 1, volume 1, number 0262100711, April.
    6. Ben Mermelstein & Volker Nocke & Mark A. Satterthwaite & Michael D. Whinston, 2020. "Internal versus External Growth in Industries with Scale Economies: A Computational Model of Optimal Merger Policy," Journal of Political Economy, University of Chicago Press, vol. 128(1), pages 301-341.
    7. Ravi Varadhan & Christophe Roland, 2008. "Simple and Globally Convergent Methods for Accelerating the Convergence of Any EM Algorithm," Scandinavian Journal of Statistics, Danish Society for Theoretical Statistics;Finnish Statistical Society;Norwegian Statistical Association;Swedish Statistical Association, vol. 35(2), pages 335-353, June.
    8. Maliar, Lilia & Maliar, Serguei, 2013. "Envelope condition method versus endogenous grid method for solving dynamic programming problems," Economics Letters, Elsevier, vol. 120(2), pages 262-266.
    9. Kenneth L. Judd & Lilia Maliar & Serguei Maliar & Inna Tsener, 2017. "How to solve dynamic stochastic models computing expectations just once," Quantitative Economics, Econometric Society, vol. 8(3), pages 851-893, November.
    10. Berry, Steven & Levinsohn, James & Pakes, Ariel, 1995. "Automobile Prices in Market Equilibrium," Econometrica, Econometric Society, vol. 63(4), pages 841-890, July.
    11. Victor Aguirregabiria & Mathieu Marcoux, 2021. "Imposing equilibrium restrictions in the estimation of dynamic discrete games," Quantitative Economics, Econometric Society, vol. 12(4), pages 1223-1271, November.
    12. Judd, Kenneth L. & Maliar, Lilia & Maliar, Serguei & Valero, Rafael, 2014. "Smolyak method for solving dynamic economic models: Lagrange interpolation, anisotropic grid and adaptive domain," Journal of Economic Dynamics and Control, Elsevier, vol. 44(C), pages 92-123.
    13. Carroll, Christopher D., 2006. "The method of endogenous gridpoints for solving dynamic stochastic optimization problems," Economics Letters, Elsevier, vol. 91(3), pages 312-320, June.
    14. Marlon Azinovic & Luca Gaegauf & Simon Scheidegger, 2022. "Deep Equilibrium Nets," International Economic Review, Department of Economics, University of Pennsylvania and Osaka University Institute of Social and Economic Research Association, vol. 63(4), pages 1471-1525, November.
    15. Mario J. Miranda & Paul L. Fackler, 2004. "Applied Computational Economics and Finance," MIT Press Books, The MIT Press, edition 1, volume 1, number 0262633094, April.
    16. Judd, Kenneth L., 1992. "Projection methods for solving aggregate growth models," Journal of Economic Theory, Elsevier, vol. 58(2), pages 410-452, December.
    17. FUKASAWA Takeshi & OHASHI Hiroshi, 2023. "Long-run Effect of a Horizontal Merger and Its Remedial Standards," Discussion papers 23001, Research Institute of Economy, Trade and Industry (RIETI).
    18. Martin L. Puterman & Shelby L. Brumelle, 1979. "On the Convergence of Policy Iteration in Stationary Dynamic Programming," Mathematics of Operations Research, INFORMS, vol. 4(1), pages 60-69, February.
    19. Christopher Conlon & Jeff Gortmaker, 2020. "Best practices for differentiated products demand estimation with PyBLP," RAND Journal of Economics, RAND Corporation, vol. 51(4), pages 1108-1161, December.
    20. Russell W. Cooper & John C. Haltiwanger, 2006. "On the Nature of Capital Adjustment Costs," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 73(3), pages 611-633.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. FUKASAWA Takeshi & OHASHI Hiroshi, 2023. "Long-run Effect of a Horizontal Merger and Its Remedial Standards," Discussion papers 23001, Research Institute of Economy, Trade and Industry (RIETI).
    2. Ayşe Kabukçuoğlu & Enrique Martínez-García, 2021. "A Generalized Time Iteration Method for Solving Dynamic Optimization Problems with Occasionally Binding Constraints," Computational Economics, Springer;Society for Computational Economics, vol. 58(2), pages 435-460, August.
    3. Takeshi Fukasawa, 2024. "Fast and simple inner-loop algorithms of static / dynamic BLP estimations," Papers 2404.04494, arXiv.org, revised Oct 2024.
    4. Lilia Maliar & Serguei Maliar & John B. Taylor & Inna Tsener, 2020. "A tractable framework for analyzing a class of nonstationary Markov models," Quantitative Economics, Econometric Society, vol. 11(4), pages 1289-1323, November.
    5. Serguei Maliar & John Taylor & Lilia Maliar, 2016. "The Impact of Alternative Transitions to Normalized Monetary Policy," 2016 Meeting Papers 794, Society for Economic Dynamics.
    6. Alexander Ludwig & Matthias Schön, 2018. "Endogenous Grids in Higher Dimensions: Delaunay Interpolation and Hybrid Methods," Computational Economics, Springer;Society for Computational Economics, vol. 51(3), pages 463-492, March.
    7. Guerra Vallejos, Ernesto & Bobenrieth Hochfarber, Eugenio & Bobenrieth Hochfarber, Juan & Wright, Brian D., 2021. "Solving dynamic stochastic models with multiple occasionally binding constraints," Economic Modelling, Elsevier, vol. 105(C).
    8. Ayse Kabukcuoglu & Enrique Martínez-García, 2016. "The Market Resources Method for Solving Dynamic Optimization Problems," Koç University-TUSIAD Economic Research Forum Working Papers 1607, Koc University-TUSIAD Economic Research Forum.
    9. Maliar, Lilia & Maliar, Serguei & Winant, Pablo, 2021. "Deep learning for solving dynamic economic models," Journal of Monetary Economics, Elsevier, vol. 122(C), pages 76-101.
    10. Schesch, Constantin, 2024. "Pseudospectral methods for continuous-time heterogeneous-agent models," Journal of Economic Dynamics and Control, Elsevier, vol. 163(C).
    11. Hull, Isaiah, 2015. "Approximate dynamic programming with post-decision states as a solution method for dynamic economic models," Journal of Economic Dynamics and Control, Elsevier, vol. 55(C), pages 57-70.
    12. Victor Aguirregabiria & Allan Collard-Wexler & Stephen P. Ryan, 2021. "Dynamic Games in Empirical Industrial Organization," NBER Working Papers 29291, National Bureau of Economic Research, Inc.
    13. Pál, László & Sándor, Zsolt, 2023. "Comparing procedures for estimating random coefficient logit demand models with a special focus on obtaining global optima," International Journal of Industrial Organization, Elsevier, vol. 88(C).
    14. Chase Coleman & Spencer Lyon & Lilia Maliar & Serguei Maliar, 2021. "Matlab, Python, Julia: What to Choose in Economics?," Computational Economics, Springer;Society for Computational Economics, vol. 58(4), pages 1263-1288, December.
    15. Judd, Kenneth L. & Maliar, Lilia & Maliar, Serguei & Valero, Rafael, 2014. "Smolyak method for solving dynamic economic models: Lagrange interpolation, anisotropic grid and adaptive domain," Journal of Economic Dynamics and Control, Elsevier, vol. 44(C), pages 92-123.
    16. White, Matthew N., 2015. "The method of endogenous gridpoints in theory and practice," Journal of Economic Dynamics and Control, Elsevier, vol. 60(C), pages 26-41.
    17. Arellano, Cristina & Maliar, Lilia & Maliar, Serguei & Tsyrennikov, Viktor, 2016. "Envelope condition method with an application to default risk models," Journal of Economic Dynamics and Control, Elsevier, vol. 69(C), pages 436-459.
    18. Fabian Goessling, 2019. "Exact Expectations: Efficient Calculation of DSGE Models," Computational Economics, Springer;Society for Computational Economics, vol. 53(3), pages 977-990, March.
    19. Yasuo Hirose & Takeki Sunakawa, 2019. "Review of Solution and Estimation Methods for Nonlinear Dynamic Stochastic General Equilibrium Models with the Zero Lower Bound," The Japanese Economic Review, Japanese Economic Association, vol. 70(1), pages 51-104, March.
    20. Karolos Arapakis, 2023. "A Method to Pre-compile Numerical Integrals When Solving Stochastic Dynamic Problems," Computational Economics, Springer;Society for Computational Economics, vol. 61(2), pages 593-610, February.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2407.04227. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.