IDEAS home Printed from https://ideas.repec.org/a/spr/coopap/v56y2013i3p635-674.html
   My bibliography  Save this article

Particle methods for stochastic optimal control problems

Author

Listed:
  • Pierre Carpentier
  • Guy Cohen
  • Anes Dallagi

Abstract

When dealing with numerical solution of stochastic optimal control problems, stochastic dynamic programming is the natural framework. In order to try to overcome the so-called curse of dimensionality, the stochastic programming school promoted another approach based on scenario trees which can be seen as the combination of Monte Carlo sampling ideas on the one hand, and of a heuristic technique to handle causality (or nonanticipativeness) constraints on the other hand. However, if one considers that the solution of a stochastic optimal control problem is a feedback law which relates control to state variables, the numerical resolution of the optimization problem over a scenario tree should be completed by a feedback synthesis stage in which, at each time step of the scenario tree, control values at nodes are plotted against corresponding state values to provide a first discrete shape of this feedback law from which a continuous function can be finally inferred. From this point of view, the scenario tree approach faces an important difficulty: at the first time stages (close to the tree root), there are a few nodes (or Monte-Carlo particles), and therefore a relatively scarce amount of information to guess a feedback law, but this information is generally of a good quality (that is, viewed as a set of control value estimates for some particular state values, it has a small variance because the future of those nodes is rich enough); on the contrary, at the final time stages (near the tree leaves), the number of nodes increases but the variance gets large because the future of each node gets poor (and sometimes even deterministic). After this dilemma has been confirmed by numerical experiments, we have tried to derive new variational approaches. First of all, two different formulations of the essential constraint of nonanticipativeness are considered: one is called algebraic and the other one is called functional. Next, in both settings, we obtain optimality conditions for the corresponding optimal control problem. For the numerical resolution of those optimality conditions, an adaptive mesh discretization method is used in the state space in order to provide information for feedback synthesis. This mesh is naturally derived from a bunch of sample noise trajectories which need not to be put into the form of a tree prior to numerical resolution. In particular, an important consequence of this discrepancy with the scenario tree approach is that the same number of nodes (or points) are available from the beginning to the end of the time horizon. And this will be obtained without sacrifying the quality of the results (that is, the variance of the estimates). Results of experiments with a hydro-electric dam production management problem will be presented and will demonstrate the claimed improvements. A more realistic problem will also be presented in order to demonstrate the effectiveness of the method for high dimensional problems. Copyright Springer Science+Business Media New York 2013

Suggested Citation

  • Pierre Carpentier & Guy Cohen & Anes Dallagi, 2013. "Particle methods for stochastic optimal control problems," Computational Optimization and Applications, Springer, vol. 56(3), pages 635-674, December.
  • Handle: RePEc:spr:coopap:v:56:y:2013:i:3:p:635-674
    DOI: 10.1007/s10589-013-9579-y
    as

    Download full text from publisher

    File URL: http://hdl.handle.net/10.1007/s10589-013-9579-y
    Download Restriction: Access to full text is restricted to subscribers.

    File URL: https://libkey.io/10.1007/s10589-013-9579-y?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. K. Barty & J.-P. Chancelier & G. Cohen & M. Lara & T. Guilbaud & P. Carpentier, 2006. "Dual effect free stochastic controls," Annals of Operations Research, Springer, vol. 142(1), pages 41-62, February.
    2. Z. L. Chen & W. B. Powell, 1999. "Convergent Cutting-Plane and Partial-Sampling Algorithm for Multistage Stochastic Linear Programs with Recourse," Journal of Optimization Theory and Applications, Springer, vol. 102(3), pages 497-524, September.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. de Queiroz, Anderson Rodrigo, 2016. "Stochastic hydro-thermal scheduling optimization: An overview," Renewable and Sustainable Energy Reviews, Elsevier, vol. 62(C), pages 382-395.
    2. Lee, Jinkyu & Bae, Sanghyeon & Kim, Woo Chang & Lee, Yongjae, 2023. "Value function gradient learning for large-scale multistage stochastic programming problems," European Journal of Operational Research, Elsevier, vol. 308(1), pages 321-335.
    3. Saif Benjaafar & Daniel Jiang & Xiang Li & Xiaobo Li, 2022. "Dynamic Inventory Repositioning in On-Demand Rental Networks," Management Science, INFORMS, vol. 68(11), pages 7861-7878, November.
    4. Davi Valladão & Thuener Silva & Marcus Poggi, 2019. "Time-consistent risk-constrained dynamic portfolio optimization with transactional costs and time-dependent returns," Annals of Operations Research, Springer, vol. 282(1), pages 379-405, November.
    5. Murwan Siddig & Yongjia Song, 2022. "Adaptive partition-based SDDP algorithms for multistage stochastic linear programming with fixed recourse," Computational Optimization and Applications, Springer, vol. 81(1), pages 201-250, January.
    6. Andre Luiz Diniz & Maria Elvira P. Maceira & Cesar Luis V. Vasconcellos & Debora Dias J. Penna, 2020. "A combined SDDP/Benders decomposition approach with a risk-averse surface concept for reservoir operation in long term power generation planning," Annals of Operations Research, Springer, vol. 292(2), pages 649-681, September.
    7. Leah K. Lakdawala & David Simon, 2016. "The Intergenerational Consequences of Tobacco Policy," Working papers 2016-27, University of Connecticut, Department of Economics.
    8. Pedro Borges, 2022. "Cut-sharing across trees and efficient sequential sampling for SDDP with uncertainty in the RHS," Computational Optimization and Applications, Springer, vol. 82(3), pages 617-647, July.
    9. Martin Šmíd & Václav Kozmík, 2024. "Approximation of multistage stochastic programming problems by smoothed quantization," Review of Managerial Science, Springer, vol. 18(7), pages 2079-2114, July.
    10. Wim Ackooij & Welington Oliveira & Yongjia Song, 2019. "On level regularization with normal solutions in decomposition methods for multistage stochastic programming problems," Computational Optimization and Applications, Springer, vol. 74(1), pages 1-42, September.
    11. Guigues, Vincent & Sagastizábal, Claudia, 2012. "The value of rolling-horizon policies for risk-averse hydro-thermal planning," European Journal of Operational Research, Elsevier, vol. 217(1), pages 129-140.
    12. W. Ackooij & X. Warin, 2020. "On conditional cuts for stochastic dual dynamic programming," EURO Journal on Computational Optimization, Springer;EURO - The Association of European Operational Research Societies, vol. 8(2), pages 173-199, June.
    13. Michelle Bandarra & Vincent Guigues, 2021. "Single cut and multicut stochastic dual dynamic programming with cut selection for multistage stochastic linear programs: convergence proof and numerical experiments," Computational Management Science, Springer, vol. 18(2), pages 125-148, June.
    14. Cheng Kai-Wen & Kenkel Don S, 2010. "U.S. Cigarette Demand: 1944-2004," The B.E. Journal of Economic Analysis & Policy, De Gruyter, vol. 10(1), pages 1-21, August.
    15. Steeger, Gregory & Rebennack, Steffen, 2017. "Dynamic convexification within nested Benders decomposition using Lagrangian relaxation: An application to the strategic bidding problem," European Journal of Operational Research, Elsevier, vol. 257(2), pages 669-686.
    16. Soares, Murilo Pereira & Street, Alexandre & Valladão, Davi Michel, 2017. "On the solution variability reduction of Stochastic Dual Dynamic Programming applied to energy planning," European Journal of Operational Research, Elsevier, vol. 258(2), pages 743-760.
    17. Warren Powell & Andrzej Ruszczyński & Huseyin Topaloglu, 2004. "Learning Algorithms for Separable Approximations of Discrete Stochastic Optimization Problems," Mathematics of Operations Research, INFORMS, vol. 29(4), pages 814-836, November.
    18. Juliana M. Nascimento & Warren B. Powell, 2009. "An Optimal Approximate Dynamic Programming Algorithm for the Lagged Asset Acquisition Problem," Mathematics of Operations Research, INFORMS, vol. 34(1), pages 210-237, February.
    19. Guan, Z. & Philpott, A.B., 2011. "A multistage stochastic programming model for the New Zealand dairy industry," International Journal of Production Economics, Elsevier, vol. 134(2), pages 289-299, December.
    20. K. Linowsky & A. B. Philpott, 2005. "On the Convergence of Sampling-Based Decomposition Algorithms for Multistage Stochastic Programs," Journal of Optimization Theory and Applications, Springer, vol. 125(2), pages 349-366, May.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:coopap:v:56:y:2013:i:3:p:635-674. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.