IDEAS home Printed from https://ideas.repec.org/a/eee/ejores/v298y2022i3p939-952.html
   My bibliography  Save this article

Deep Q-learning for same-day delivery with vehicles and drones

Author

Listed:
  • Chen, Xinwei
  • Ulmer, Marlin W.
  • Thomas, Barrett W.

Abstract

In this paper, we consider same-day delivery with vehicles and drones. Customers make delivery requests over the course of the day, and the dispatcher dynamically dispatches vehicles and drones to deliver the goods to customers before their delivery deadline. Vehicles can deliver multiple packages in one route but travel relatively slowly due to the urban traffic. Drones travel faster, but they have limited capacity and require charging or battery swaps. To exploit the different strengths of the fleets, we propose a deep Q-learning approach. Our method learns the value of assigning a new customer to either drones or vehicles as well as the option to not offer service at all. In a systematic computational analysis, we show the superiority of our policy compared to benchmark policies and the effectiveness of our deep Q-learning approach. We also show that the combination of state and action features is very valuable and that our policy can maintain effectiveness when demand data and the fleet size change moderately.

Suggested Citation

  • Chen, Xinwei & Ulmer, Marlin W. & Thomas, Barrett W., 2022. "Deep Q-learning for same-day delivery with vehicles and drones," European Journal of Operational Research, Elsevier, vol. 298(3), pages 939-952.
    • Handle: RePEc:eee:ejores:v:298:y:2022:i:3:p:939-952
    DOI: 10.1016/j.ejor.2021.06.021
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0377221721005361
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.ejor.2021.06.021?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Mathias A. Klapp & Alan L. Erera & Alejandro Toriello, 2018. "The One-Dimensional Dynamic Dispatch Waves Problem," Transportation Science, INFORMS, vol. 52(2), pages 402-415, March.
    2. Stacy A. Voccia & Ann Melissa Campbell & Barrett W. Thomas, 2019. "The Same-Day Delivery Problem for Online Purchases," Service Science, INFORMS, vol. 53(1), pages 167-184, February.
    3. Iman Dayarian & Martin Savelsbergh & John-Paul Clarke, 2020. "Same-Day Delivery with Drone Resupply," Transportation Science, INFORMS, vol. 54(1), pages 229-249, January.
    4. Klapp, Mathias A. & Erera, Alan L. & Toriello, Alejandro, 2018. "The Dynamic Dispatch Waves Problem for same-day delivery," European Journal of Operational Research, Elsevier, vol. 271(2), pages 519-534.
    5. Powell, Warren B., 2019. "A unified framework for stochastic optimization," European Journal of Operational Research, Elsevier, vol. 275(3), pages 795-821.
    6. Nabila Azi & Michel Gendreau & Jean-Yves Potvin, 2012. "A dynamic vehicle routing problem with multiple delivery routes," Annals of Operations Research, Springer, vol. 199(1), pages 103-112, October.
    7. Marlin W. Ulmer & Dirk C. Mattfeld & Felix Köster, 2018. "Budgeting Time for Dynamic Vehicle Routing with Stochastic Customer Requests," Transportation Science, INFORMS, vol. 52(1), pages 20-37, January.
    8. Alejandro Toriello & William B. Haskell & Michael Poremba, 2014. "A Dynamic Traveling Salesman Problem with Stochastic Arc Costs," Operations Research, INFORMS, vol. 62(5), pages 1107-1125, October.
    9. Pérez Rivera, Arturo E. & Mes, Martijn R.K., 2017. "Anticipatory freight selection in intermodal long-haul round-trips," Transportation Research Part E: Logistics and Transportation Review, Elsevier, vol. 105(C), pages 176-194.
    10. Russell W. Bent & Pascal Van Hentenryck, 2004. "Scenario-Based Planning for Partially Dynamic Vehicle Routing with Stochastic Customers," Operations Research, INFORMS, vol. 52(6), pages 977-987, December.
    11. Ulmer, Marlin W. & Thomas, Barrett W., 2020. "Meso-parametric value function approximation for dynamic customer acceptances in delivery routing," European Journal of Operational Research, Elsevier, vol. 285(1), pages 183-195.
    12. Klapp, Mathias A. & Erera, Alan L. & Toriello, Alejandro, 2020. "Request acceptance in same-day delivery," Transportation Research Part E: Logistics and Transportation Review, Elsevier, vol. 143(C).
    13. Iman Dayarian & Martin Savelsbergh, 2020. "Crowdshipping and Same‐day Delivery: Employing In‐store Customers to Deliver Online Orders," Production and Operations Management, Production and Operations Management Society, vol. 29(9), pages 2153-2174, September.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Bosse, Alexander & Ulmer, Marlin W. & Manni, Emanuele & Mattfeld, Dirk C., 2023. "Dynamic priority rules for combining on-demand passenger transportation and transportation of goods," European Journal of Operational Research, Elsevier, vol. 309(1), pages 399-408.
    2. Zhang, Jian & Woensel, Tom Van, 2023. "Dynamic vehicle routing with random requests: A literature review," International Journal of Production Economics, Elsevier, vol. 256(C).
    3. Chen, Xinwei & Wang, Tong & Thomas, Barrett W. & Ulmer, Marlin W., 2023. "Same-day delivery with fair customer service," European Journal of Operational Research, Elsevier, vol. 308(2), pages 738-751.
    4. Liu, Zeyu & Li, Xueping & Khojandi, Anahita, 2022. "The flying sidekick traveling salesman problem with stochastic travel time: A reinforcement learning approach," Transportation Research Part E: Logistics and Transportation Review, Elsevier, vol. 164(C).
    5. Zhitao Xu & Adel Elomri & Roberto Baldacci & Laoucine Kerbache & Zhenyong Wu, 2024. "Frontiers and trends of supply chain optimization in the age of industry 4.0: an operations research perspective," Annals of Operations Research, Springer, vol. 338(2), pages 1359-1401, July.
    6. Pina-Pardo, Juan C. & Silva, Daniel F. & Smith, Alice E. & Gatica, Ricardo A., 2024. "Fleet resupply by drones for last-mile delivery," European Journal of Operational Research, Elsevier, vol. 316(1), pages 168-182.
    7. Banerjee, Dipayan & Erera, Alan L. & Stroh, Alexander M. & Toriello, Alejandro, 2023. "Who has access to e-commerce and when? Time-varying service regions in same-day delivery," Transportation Research Part B: Methodological, Elsevier, vol. 170(C), pages 148-168.
    8. Nikola Mardešić & Tomislav Erdelić & Tonči Carić & Marko Đurasević, 2023. "Review of Stochastic Dynamic Vehicle Routing in the Evolving Urban Logistics Environment," Mathematics, MDPI, vol. 12(1), pages 1-44, December.
    9. Quy Ta-Dinh & Tu-San Pham & Minh Hoàng Hà & Louis-Martin Rousseau, 2024. "A reinforcement learning approach for the online dynamic home health care scheduling problem," Health Care Management Science, Springer, vol. 27(4), pages 650-664, December.
    10. Peter Dieter & Philipp Speckenmeyer & Guido Schryen, 2024. "The On-Demand Delivery Problem: Assignment of Orders to Warehouses and Couriers," Working Papers Dissertations 126, Paderborn University, Faculty of Business Administration and Economics.
    11. Abdul Aziz Khan Niazi & Tehmina Fiaz Qazi & Maryam Aziz & Abdul Basit & Ifra Aziz Khan Niazi, 2023. "Using the Binary Matrices for Modeling the Supply Chain Issues of Virtual Shops: An Evidence from Pakistan," Journal of Policy Research (JPR), Research Foundation for Humanity (RFH), vol. 9(2), pages 548-564.
    12. Fleckenstein, David & Klein, Robert & Steinhardt, Claudius, 2023. "Recent advances in integrating demand management and vehicle routing: A methodological review," European Journal of Operational Research, Elsevier, vol. 306(2), pages 499-518.
    13. Anna Konovalenko & Lars Magnus Hvattum, 2024. "Optimizing a Dynamic Vehicle Routing Problem with Deep Reinforcement Learning: Analyzing State-Space Components," Logistics, MDPI, vol. 8(4), pages 1-18, October.
    14. Zhu, Waiming & Hu, Xiaoxuan & Pei, Jun & Pardalos, Panos M., 2024. "Minimizing the total travel distance for the locker-based drone delivery: A branch-and-cut-based method," Transportation Research Part B: Methodological, Elsevier, vol. 184(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Zhang, Jian & Woensel, Tom Van, 2023. "Dynamic vehicle routing with random requests: A literature review," International Journal of Production Economics, Elsevier, vol. 256(C).
    2. Fleckenstein, David & Klein, Robert & Steinhardt, Claudius, 2023. "Recent advances in integrating demand management and vehicle routing: A methodological review," European Journal of Operational Research, Elsevier, vol. 306(2), pages 499-518.
    3. Klein, Vienna & Steinhardt, Claudius, 2023. "Dynamic demand management and online tour planning for same-day delivery," European Journal of Operational Research, Elsevier, vol. 307(2), pages 860-886.
    4. Côté, Jean-François & Alves de Queiroz, Thiago & Gallesi, Francesco & Iori, Manuel, 2023. "A branch-and-regret algorithm for the same-day delivery problem," Transportation Research Part E: Logistics and Transportation Review, Elsevier, vol. 177(C).
    5. Soeffker, Ninja & Ulmer, Marlin W. & Mattfeld, Dirk C., 2022. "Stochastic dynamic vehicle routing in the light of prescriptive analytics: A review," European Journal of Operational Research, Elsevier, vol. 298(3), pages 801-820.
    6. Chen, Xinwei & Wang, Tong & Thomas, Barrett W. & Ulmer, Marlin W., 2023. "Same-day delivery with fair customer service," European Journal of Operational Research, Elsevier, vol. 308(2), pages 738-751.
    7. Zhang, Jian & Luo, Kelin & Florio, Alexandre M. & Van Woensel, Tom, 2023. "Solving large-scale dynamic vehicle routing problems with stochastic requests," European Journal of Operational Research, Elsevier, vol. 306(2), pages 596-614.
    8. Ritzinger, Ulrike & Puchinger, Jakob & Rudloff, Christian & Hartl, Richard F., 2022. "Comparison of anticipatory algorithms for a dial-a-ride problem," European Journal of Operational Research, Elsevier, vol. 301(2), pages 591-608.
    9. Peter Dieter & Philipp Speckenmeyer & Guido Schryen, 2024. "The On-Demand Delivery Problem: Assignment of Orders to Warehouses and Couriers," Working Papers Dissertations 126, Paderborn University, Faculty of Business Administration and Economics.
    10. Banerjee, Dipayan & Erera, Alan L. & Stroh, Alexander M. & Toriello, Alejandro, 2023. "Who has access to e-commerce and when? Time-varying service regions in same-day delivery," Transportation Research Part B: Methodological, Elsevier, vol. 170(C), pages 148-168.
    11. Liu, Zeyu & Li, Xueping & Khojandi, Anahita, 2022. "The flying sidekick traveling salesman problem with stochastic travel time: A reinforcement learning approach," Transportation Research Part E: Logistics and Transportation Review, Elsevier, vol. 164(C).
    12. Klapp, Mathias A. & Erera, Alan L. & Toriello, Alejandro, 2020. "Request acceptance in same-day delivery," Transportation Research Part E: Logistics and Transportation Review, Elsevier, vol. 143(C).
    13. Ulmer, Marlin W. & Soeffker, Ninja & Mattfeld, Dirk C., 2018. "Value function approximation for dynamic multi-period vehicle routing," European Journal of Operational Research, Elsevier, vol. 269(3), pages 883-899.
    14. Bosse, Alexander & Ulmer, Marlin W. & Manni, Emanuele & Mattfeld, Dirk C., 2023. "Dynamic priority rules for combining on-demand passenger transportation and transportation of goods," European Journal of Operational Research, Elsevier, vol. 309(1), pages 399-408.
    15. Nikola Mardešić & Tomislav Erdelić & Tonči Carić & Marko Đurasević, 2023. "Review of Stochastic Dynamic Vehicle Routing in the Evolving Urban Logistics Environment," Mathematics, MDPI, vol. 12(1), pages 1-44, December.
    16. Waßmuth, Katrin & Köhler, Charlotte & Agatz, Niels & Fleischmann, Moritz, 2023. "Demand management for attended home delivery—A literature review," European Journal of Operational Research, Elsevier, vol. 311(3), pages 801-815.
    17. Ouyang, Zhiyuan & Leung, Eric K.H. & Huang, George Q., 2023. "Community logistics and dynamic community partitioning: A new approach for solving e-commerce last mile delivery," European Journal of Operational Research, Elsevier, vol. 307(1), pages 140-156.
    18. Avraham, Edison & Raviv, Tal, 2021. "The steady-state mobile personnel booking problem," Transportation Research Part B: Methodological, Elsevier, vol. 154(C), pages 266-288.
    19. Marlin W. Ulmer & Alan Erera & Martin Savelsbergh, 2022. "Dynamic service area sizing in urban delivery," OR Spectrum: Quantitative Approaches in Management, Springer;Gesellschaft für Operations Research e.V., vol. 44(3), pages 763-793, September.
    20. Stacy A. Voccia & Ann Melissa Campbell & Barrett W. Thomas, 2019. "The Same-Day Delivery Problem for Online Purchases," Service Science, INFORMS, vol. 53(1), pages 167-184, February.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:ejores:v:298:y:2022:i:3:p:939-952. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/eor .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.