IDEAS home Printed from https://ideas.repec.org/a/gam/jmathe/v10y2022i20p3902-d948477.html
   My bibliography  Save this article

Deep Reinforcement Learning for Crowdshipping Last-Mile Delivery with Endogenous Uncertainty

Author

Listed:
  • Marco Silva

    (Industrial Engineering and Management, Institute for Systems and Computer Engineering, Technology and Science, 4200-465 Porto, Portugal)

  • João Pedro Pedroso

    (Industrial Engineering and Management, Institute for Systems and Computer Engineering, Technology and Science, 4200-465 Porto, Portugal
    Department of Computer Science, Faculty of Sciences, University of Porto, 4169-007 Porto, Portugal)

Abstract

In this work, we study a flexible compensation scheme for last-mile delivery where a company outsources part of the activity of delivering products to its customers to occasional drivers (ODs), under a scheme named crowdshipping. All deliveries are completed at the minimum total cost incurred with their vehicles and drivers plus the compensation paid to the ODs. The company decides on the best compensation scheme to offer to the ODs at the planning stage. We model our problem based on a stochastic and dynamic environment where delivery orders and ODs volunteering to make deliveries present themselves randomly within fixed time windows. The uncertainty is endogenous in the sense that the compensation paid to ODs influences their availability. We develop a deep reinforcement learning (DRL) algorithm that can deal with large instances while focusing on the quality of the solution: we combine the combinatorial structure of the action space with the neural network of the approximated value function, involving techniques from machine learning and integer optimization. The results show the effectiveness of the DRL approach by examining out-of-sample performance and that it is suitable to process large samples of uncertain data, which induces better solutions.

Suggested Citation

  • Marco Silva & João Pedro Pedroso, 2022. "Deep Reinforcement Learning for Crowdshipping Last-Mile Delivery with Endogenous Uncertainty," Mathematics, MDPI, vol. 10(20), pages 1-23, October.
  • Handle: RePEc:gam:jmathe:v:10:y:2022:i:20:p:3902-:d:948477
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/2227-7390/10/20/3902/pdf
    Download Restriction: no

    File URL: https://www.mdpi.com/2227-7390/10/20/3902/
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Patrick Jaillet, 1988. "A Priori Solution of a Traveling Salesman Problem in Which a Random Subset of the Customers Are Visited," Operations Research, INFORMS, vol. 36(6), pages 929-936, December.
    2. Gilbert Laporte & François V. Louveaux & Hélène Mercure, 1994. "A Priori Optimization of the Probabilistic Traveling Salesman Problem," Operations Research, INFORMS, vol. 42(3), pages 543-549, June.
    3. Dimitris J. Bertsimas, 1992. "A Vehicle Routing Problem with Stochastic Demand," Operations Research, INFORMS, vol. 40(3), pages 574-585, June.
    4. Maksim Balashov & Anton Kiselev & Alena Kuryleva, 2021. "Reinforcement Learning Approach for Dynamic Pricing," Studies on Entrepreneurship, Structural Change and Industrial Dynamics, in: Tessaleno Devezas & João Leitão & Askar Sarygulov (ed.), The Economics of Digital Transformation, edition 1, pages 123-141, Springer.
    5. Xin Chen & Melvyn Sim & Peng Sun, 2007. "A Robust Optimization Perspective on Stochastic Programming," Operations Research, INFORMS, vol. 55(6), pages 1058-1071, December.
    6. Joel Goh & Melvyn Sim, 2010. "Distributionally Robust Optimization and Its Tractable Approximations," Operations Research, INFORMS, vol. 58(4-part-1), pages 902-917, August.
    7. Tore Jonsbråten & Roger Wets & David Woodruff, 1998. "A class of stochastic programs withdecision dependent random elements," Annals of Operations Research, Springer, vol. 82(0), pages 83-106, August.
    8. Erick Delage & Yinyu Ye, 2010. "Distributionally Robust Optimization Under Moment Uncertainty with Application to Data-Driven Problems," Operations Research, INFORMS, vol. 58(3), pages 595-612, June.
    9. Gilbert Laporte & FranÇois V. Louveaux & Luc van Hamme, 2002. "An Integer L -Shaped Algorithm for the Capacitated Vehicle Routing Problem with Stochastic Demands," Operations Research, INFORMS, vol. 50(3), pages 415-423, June.
    10. Archetti, Claudia & Savelsbergh, Martin & Speranza, M. Grazia, 2016. "The Vehicle Routing Problem with Occasional Drivers," European Journal of Operational Research, Elsevier, vol. 254(2), pages 472-480.
    11. Iman Dayarian & Martin Savelsbergh, 2020. "Crowdshipping and Same‐day Delivery: Employing In‐store Customers to Deliver Online Orders," Production and Operations Management, Production and Operations Management Society, vol. 29(9), pages 2153-2174, September.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Silva, Marco & Pedroso, João Pedro & Viana, Ana, 2023. "Stochastic crowd shipping last-mile delivery with correlated marginals and probabilistic constraints," European Journal of Operational Research, Elsevier, vol. 307(1), pages 249-265.
    2. Zhi Chen & Melvyn Sim & Huan Xu, 2019. "Distributionally Robust Optimization with Infinitely Constrained Ambiguity Sets," Operations Research, INFORMS, vol. 67(5), pages 1328-1344, September.
    3. Mathias A. Klapp & Alan L. Erera & Alejandro Toriello, 2018. "The One-Dimensional Dynamic Dispatch Waves Problem," Transportation Science, INFORMS, vol. 52(2), pages 402-415, March.
    4. Jinil Han & Chungmok Lee & Sungsoo Park, 2014. "A Robust Scenario Approach for the Vehicle Routing Problem with Uncertain Travel Times," Transportation Science, INFORMS, vol. 48(3), pages 373-390, August.
    5. L. Jeff Hong & Zhiyuan Huang & Henry Lam, 2021. "Learning-Based Robust Optimization: Procedures and Statistical Guarantees," Management Science, INFORMS, vol. 67(6), pages 3447-3467, June.
    6. Fanwen Meng & Jin Qi & Meilin Zhang & James Ang & Singfat Chu & Melvyn Sim, 2015. "A Robust Optimization Model for Managing Elective Admission in a Public Hospital," Operations Research, INFORMS, vol. 63(6), pages 1452-1467, December.
    7. Ann M. Campbell & Barrett W. Thomas, 2008. "Probabilistic Traveling Salesman Problem with Deadlines," Transportation Science, INFORMS, vol. 42(1), pages 1-21, February.
    8. Karels, Vincent C.G. & Rei, Walter & Veelenturf, Lucas P. & Van Woensel, Tom, 2024. "A vehicle routing problem with multiple service agreements," European Journal of Operational Research, Elsevier, vol. 313(1), pages 129-145.
    9. Soumia Ichoua & Michel Gendreau & Jean-Yves Potvin, 2006. "Exploiting Knowledge About Future Demands for Real-Time Vehicle Dispatching," Transportation Science, INFORMS, vol. 40(2), pages 211-225, May.
    10. Jian Yang & Patrick Jaillet & Hani Mahmassani, 2004. "Real-Time Multivehicle Truckload Pickup and Delivery Problems," Transportation Science, INFORMS, vol. 38(2), pages 135-148, May.
    11. Postek, Krzysztof & Ben-Tal, A. & den Hertog, Dick & Melenberg, Bertrand, 2015. "Exact Robust Counterparts of Ambiguous Stochastic Constraints Under Mean and Dispersion Information," Other publications TiSEM d718e419-a375-4707-b206-e, Tilburg University, School of Economics and Management.
    12. Gendreau, Michel & Laporte, Gilbert & Seguin, Rene, 1996. "Stochastic vehicle routing," European Journal of Operational Research, Elsevier, vol. 88(1), pages 3-12, January.
    13. Liu, Kanglin & Li, Qiaofeng & Zhang, Zhi-Hai, 2019. "Distributionally robust optimization of an emergency medical service station location and sizing problem with joint chance constraints," Transportation Research Part B: Methodological, Elsevier, vol. 119(C), pages 79-101.
    14. Joel Goh & Melvyn Sim, 2011. "Robust Optimization Made Easy with ROME," Operations Research, INFORMS, vol. 59(4), pages 973-985, August.
    15. Shipra Agrawal & Yichuan Ding & Amin Saberi & Yinyu Ye, 2012. "Price of Correlations in Stochastic Optimization," Operations Research, INFORMS, vol. 60(1), pages 150-162, February.
    16. Florent Hernandez & Michel Gendreau & Ola Jabali & Walter Rei, 2019. "A local branching matheuristic for the multi-vehicle routing problem with stochastic demands," Journal of Heuristics, Springer, vol. 25(2), pages 215-245, April.
    17. Gabrel, Virginie & Murat, Cécile & Thiele, Aurélie, 2014. "Recent advances in robust optimization: An overview," European Journal of Operational Research, Elsevier, vol. 235(3), pages 471-483.
    18. Long He & Ho-Yin Mak & Ying Rong & Zuo-Jun Max Shen, 2017. "Service Region Design for Urban Electric Vehicle Sharing Systems," Manufacturing & Service Operations Management, INFORMS, vol. 19(2), pages 309-327, May.
    19. Dimitris Bertsimas & Melvyn Sim & Meilin Zhang, 2019. "Adaptive Distributionally Robust Optimization," Management Science, INFORMS, vol. 65(2), pages 604-618, February.
    20. Allahviranloo, Mahdieh & Chow, Joseph Y.J. & Recker, Will W., 2014. "Selective vehicle routing problems under uncertainty without recourse," Transportation Research Part E: Logistics and Transportation Review, Elsevier, vol. 62(C), pages 68-88.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jmathe:v:10:y:2022:i:20:p:3902-:d:948477. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager (email available below). General contact details of provider: https://www.mdpi.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.