IDEAS home Printed from https://ideas.repec.org/a/spr/annopr/v241y2016i1d10.1007_s10479-012-1077-6.html
   My bibliography  Save this article

Perspectives of approximate dynamic programming

Author

Listed:
  • Warren B. Powell

    (Princeton University)

Abstract

Approximate dynamic programming has evolved, initially independently, within operations research, computer science and the engineering controls community, all searching for practical tools for solving sequential stochastic optimization problems. More so than other communities, operations research continued to develop the theory behind the basic model introduced by Bellman with discrete states and actions, even while authors as early as Bellman himself recognized its limits due to the “curse of dimensionality” inherent in discrete state spaces. In response to these limitations, subcommunities in computer science, control theory and operations research have developed a variety of methods for solving different classes of stochastic, dynamic optimization problems, creating the appearance of a jungle of competing approaches. In this article, we show that there is actually a common theme to these strategies, and underpinning the entire field remains the fundamental algorithmic strategies of value and policy iteration that were first introduced in the 1950’s and 60’s.

Suggested Citation

  • Warren B. Powell, 2016. "Perspectives of approximate dynamic programming," Annals of Operations Research, Springer, vol. 241(1), pages 319-356, June.
    • Handle: RePEc:spr:annopr:v:241:y:2016:i:1:d:10.1007_s10479-012-1077-6
    DOI: 10.1007/s10479-012-1077-6
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s10479-012-1077-6
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s10479-012-1077-6?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Powell, Warren B., 1987. "An operational planning model for the dynamic vehicle allocation problem with uncertain demands," Transportation Research Part B: Methodological, Elsevier, vol. 21(3), pages 217-232, June.
    2. Hugo P. Simão & Abraham George & Warren B. Powell & Ted Gifford & John Nienow & Jeff Day, 2010. "Approximate Dynamic Programming Captures Fleet Operations for Schneider National," Interfaces, INFORMS, vol. 40(5), pages 342-352, October.
    3. Gregory A. Godfrey & Warren B. Powell, 2002. "An Adaptive Dynamic Programming Algorithm for Dynamic Fleet Management, I: Single Period Travel Times," Transportation Science, INFORMS, vol. 36(1), pages 21-39, February.
    4. Apostolos N. Burnetas & Michael N. Katehakis, 1997. "Optimal Adaptive Policies for Markov Decision Processes," Mathematics of Operations Research, INFORMS, vol. 22(1), pages 222-255, February.
    5. Suvrajeet Sen & Julia L. Higle, 1999. "An Introductory Tutorial on Stochastic Linear Programming Models," Interfaces, INFORMS, vol. 29(2), pages 33-61, April.
    6. Gregory A. Godfrey & Warren B. Powell, 2002. "An Adaptive Dynamic Programming Algorithm for Dynamic Fleet Management, II: Multiperiod Travel Times," Transportation Science, INFORMS, vol. 36(1), pages 40-54, February.
    7. Peter Frazier & Warren Powell & Savas Dayanik, 2009. "The Knowledge-Gradient Policy for Correlated Normal Beliefs," INFORMS Journal on Computing, INFORMS, vol. 21(4), pages 599-613, November.
    8. Warren B. Powell, 2010. "Feature Article ---Merging AI and OR to Solve High-Dimensional Stochastic Optimization Problems Using Approximate Dynamic Programming," INFORMS Journal on Computing, INFORMS, vol. 22(1), pages 2-17, February.
    9. Ilya O. Ryzhov & Warren B. Powell & Peter I. Frazier, 2012. "The Knowledge Gradient Algorithm for a General Class of Online Learning Problems," Operations Research, INFORMS, vol. 60(1), pages 180-195, February.
    10. Stephen E. Chick & Noah Gans, 2009. "Economic Analysis of Simulation Selection Problems," Management Science, INFORMS, vol. 55(3), pages 421-437, March.
    11. Justin Boesel & Barry L. Nelson & Seong-Hee Kim, 2003. "Using Ranking and Selection to “Clean Up” after Simulation Optimization," Operations Research, INFORMS, vol. 51(5), pages 814-825, October.
    12. Raymond K. Cheung & Warren B. Powell, 1996. "An Algorithm for Multistage Dynamic Networks with Random Arc Capacities, with an Application to Dynamic Fleet Management," Operations Research, INFORMS, vol. 44(6), pages 951-963, December.
    13. Diana M. Negoescu & Peter I. Frazier & Warren B. Powell, 2011. "The Knowledge-Gradient Algorithm for Sequencing Experiments in Drug Discovery," INFORMS Journal on Computing, INFORMS, vol. 23(3), pages 346-363, August.
    14. Jitka Dupačová & Giorgio Consigli & Stein Wallace, 2000. "Scenarios for Multistage Stochastic Programs," Annals of Operations Research, Springer, vol. 100(1), pages 25-53, December.
    15. Allen R. Ferguson & George B. Dantzig, 1956. "The Allocation of Aircraft to Routes--An Example of Linear Programming Under Uncertain Demand," Management Science, INFORMS, vol. 3(1), pages 45-73, October.
    16. Hugo P. Simão & Jeff Day & Abraham P. George & Ted Gifford & John Nienow & Warren B. Powell, 2009. "An Approximate Dynamic Programming Algorithm for Large-Scale Fleet Management: A Case Application," Transportation Science, INFORMS, vol. 43(2), pages 178-197, May.
    17. Kenneth L. Judd, 1998. "Numerical Methods in Economics," MIT Press Books, The MIT Press, edition 1, volume 1, number 0262100711, April.
    18. Cyrus Derman, 1962. "On Sequential Decisions and Markov Chains," Management Science, INFORMS, vol. 9(1), pages 16-24, October.
    19. Linos F. Frantzeskakis & Warren B. Powell, 1990. "A Successive Linear Approximation Procedure for Stochastic, Dynamic Vehicle Allocation Problems," Transportation Science, INFORMS, vol. 24(1), pages 40-57, February.
    20. George B. Dantzig, 1955. "Linear Programming under Uncertainty," Management Science, INFORMS, vol. 1(3-4), pages 197-206, 04-07.
    21. Huseyin Topaloglu & Warren B. Powell, 2006. "Dynamic-Programming Approximations for Stochastic Time-Staged Integer Multicommodity-Flow Problems," INFORMS Journal on Computing, INFORMS, vol. 18(1), pages 31-42, February.
    22. Michael N. Katehakis & Arthur F. Veinott, 1987. "The Multi-Armed Bandit Problem: Decomposition and Computation," Mathematics of Operations Research, INFORMS, vol. 12(2), pages 262-268, May.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Voelkel, Michael A. & Sachs, Anna-Lena & Thonemann, Ulrich W., 2020. "An aggregation-based approximate dynamic programming approach for the periodic review model with random yield," European Journal of Operational Research, Elsevier, vol. 281(2), pages 286-298.
    2. Waßmuth, Katrin & Köhler, Charlotte & Agatz, Niels & Fleischmann, Moritz, 2023. "Demand management for attended home delivery—A literature review," European Journal of Operational Research, Elsevier, vol. 311(3), pages 801-815.
    3. Lee, Younsoo & Lee, Kyungsik, 2022. "New integer optimization models and an approximate dynamic programming algorithm for the lot-sizing and scheduling problem with sequence-dependent setups," European Journal of Operational Research, Elsevier, vol. 302(1), pages 230-243.
    4. Rempel, M. & Cai, J., 2021. "A review of approximate dynamic programming applications within military operations research," Operations Research Perspectives, Elsevier, vol. 8(C).
    5. Tahir Ekin & Stephen Walker & Paul Damien, 2023. "Augmented simulation methods for discrete stochastic optimization with recourse," Annals of Operations Research, Springer, vol. 320(2), pages 771-793, January.
    6. Alexandre Forel & Martin Grunow, 2023. "Dynamic stochastic lot sizing with forecast evolution in rolling‐horizon planning," Production and Operations Management, Production and Operations Management Society, vol. 32(2), pages 449-468, February.
    7. Heydar, Mojtaba & Mardaneh, Elham & Loxton, Ryan, 2022. "Approximate dynamic programming for an energy-efficient parallel machine scheduling problem," European Journal of Operational Research, Elsevier, vol. 302(1), pages 363-380.
    8. Yang Zou & Hecheng Wu & Yunqiang Yin & Lalitha Dhamotharan & Daqiang Chen & Aviral Kumar Tiwari, 2024. "An improved transformer model with multi-head attention and attention to attention for low-carbon multi-depot vehicle routing problem," Annals of Operations Research, Springer, vol. 339(1), pages 517-536, August.
    9. Silva, Thiago A.O. & de Souza, Mauricio C., 2020. "Surgical scheduling under uncertainty by approximate dynamic programming," Omega, Elsevier, vol. 95(C).
    10. Victor F. Araman & René A. Caldentey, 2022. "Diffusion Approximations for a Class of Sequential Experimentation Problems," Management Science, INFORMS, vol. 68(8), pages 5958-5979, August.
    11. Liu, Xinbao & Yang, Tianji & Pei, Jun & Liao, Haitao & Pohl, Edward A., 2019. "Replacement and inventory control for a multi-customer product service system with decreasing replacement costs," European Journal of Operational Research, Elsevier, vol. 273(2), pages 561-574.
    12. Mojtaba Heydar & Małgorzata M. O’Reilly & Erin Trainer & Mark Fackrell & Peter G. Taylor & Ali Tirdad, 2022. "A stochastic model for the patient-bed assignment problem with random arrivals and departures," Annals of Operations Research, Springer, vol. 315(2), pages 813-845, August.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Powell, Warren B., 2019. "A unified framework for stochastic optimization," European Journal of Operational Research, Elsevier, vol. 275(3), pages 795-821.
    2. Ilya O. Ryzhov & Martijn R. K. Mes & Warren B. Powell & Gerald van den Berg, 2019. "Bayesian Exploration for Approximate Dynamic Programming," Operations Research, INFORMS, vol. 67(1), pages 198-214, January.
    3. Antoine Sauré & Jonathan Patrick & Martin L. Puterman, 2015. "Simulation-Based Approximate Policy Iteration with Generalized Logistic Functions," INFORMS Journal on Computing, INFORMS, vol. 27(3), pages 579-595, August.
    4. Zolfagharinia, Hossein & Haughton, Michael, 2018. "The importance of considering non-linear layover and delay costs for local truckers," Transportation Research Part E: Logistics and Transportation Review, Elsevier, vol. 109(C), pages 331-355.
    5. Song, Haiqing & Cheung, Raymond K. & Wang, Haiyan, 2014. "An arc-exchange decomposition method for multistage dynamic networks with random arc capacities," European Journal of Operational Research, Elsevier, vol. 233(3), pages 474-487.
    6. Gabor, A.F. & Dekker, R. & van Dijk, T. & van Scheepstal, P., 2009. "Scheduling deliveries under uncertainty," ERIM Report Series Research in Management ERS-2009-040-LIS, Erasmus Research Institute of Management (ERIM), ERIM is the joint research institute of the Rotterdam School of Management, Erasmus University and the Erasmus School of Economics (ESE) at Erasmus University Rotterdam.
    7. Gregory A. Godfrey & Warren B. Powell, 2002. "An Adaptive Dynamic Programming Algorithm for Dynamic Fleet Management, I: Single Period Travel Times," Transportation Science, INFORMS, vol. 36(1), pages 21-39, February.
    8. George, David K. & Xia, Cathy H., 2011. "Fleet-sizing and service availability for a vehicle rental system via closed queueing networks," European Journal of Operational Research, Elsevier, vol. 211(1), pages 198-207, May.
    9. Zolfagharinia, Hossein & Haughton, Michael, 2016. "Effective truckload dispatch decision methods with incomplete advance load information," European Journal of Operational Research, Elsevier, vol. 252(1), pages 103-121.
    10. Zolfagharinia, Hossein & Haughton, Michael, 2014. "The benefit of advance load information for truckload carriers," Transportation Research Part E: Logistics and Transportation Review, Elsevier, vol. 70(C), pages 34-54.
    11. Schmid, Verena, 2012. "Solving the dynamic ambulance relocation and dispatching problem using approximate dynamic programming," European Journal of Operational Research, Elsevier, vol. 219(3), pages 611-621.
    12. Alan L. Erera & Juan C. Morales & Martin Savelsbergh, 2009. "Robust Optimization for Empty Repositioning Problems," Operations Research, INFORMS, vol. 57(2), pages 468-483, April.
    13. Zolfagharinia, Hossein & Haughton, Michael A., 2017. "Operational flexibility in the truckload trucking industry," Transportation Research Part B: Methodological, Elsevier, vol. 104(C), pages 437-460.
    14. Ulmer, Marlin W. & Thomas, Barrett W., 2020. "Meso-parametric value function approximation for dynamic customer acceptances in delivery routing," European Journal of Operational Research, Elsevier, vol. 285(1), pages 183-195.
    15. G J King & H Topaloglu, 2007. "Incorporating the pricing decisions into the dynamic fleet management problem," Journal of the Operational Research Society, Palgrave Macmillan;The OR Society, vol. 58(8), pages 1065-1074, August.
    16. Felix Papier & Ulrich W. Thonemann, 2008. "Queuing Models for Sizing and Structuring Rental Fleets," Transportation Science, INFORMS, vol. 42(3), pages 302-317, August.
    17. Rempel, M. & Cai, J., 2021. "A review of approximate dynamic programming applications within military operations research," Operations Research Perspectives, Elsevier, vol. 8(C).
    18. Song, Haiqing & Huang, Huei-Chuen, 2008. "A successive convex approximation method for multistage workforce capacity planning problem with turnover," European Journal of Operational Research, Elsevier, vol. 188(1), pages 29-48, July.
    19. Warren B. Powell & Abraham George & Hugo Simão & Warren Scott & Alan Lamont & Jeffrey Stewart, 2012. "SMART: A Stochastic Multiscale Model for the Analysis of Energy Resources, Technology, and Policy," INFORMS Journal on Computing, INFORMS, vol. 24(4), pages 665-682, November.
    20. Shi, Ning & Song, Haiqing & Powell, Warren B., 2014. "The dynamic fleet management problem with uncertain demand and customer chosen service level," International Journal of Production Economics, Elsevier, vol. 148(C), pages 110-121.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:annopr:v:241:y:2016:i:1:d:10.1007_s10479-012-1077-6. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.