IDEAS home Printed from https://ideas.repec.org/a/eee/transe/v164y2022ics136655452200179x.html
   My bibliography  Save this article

Online model-based reinforcement learning for decision-making in long distance routes

Author

Listed:
  • Alcaraz, Juan J.
  • Losilla, Fernando
  • Caballero-Arnaldos, Luis

Abstract

In road transportation, long-distance routes require scheduled driving times, breaks, and rest periods, in compliance with the regulations on working conditions for truck drivers, while ensuring goods are delivered within the time windows of each customer. However, routes are subject to uncertain travel and service times, and incidents may cause additional delays, making predefined schedules ineffective in many real-life situations. This paper presents a reinforcement learning (RL) algorithm capable of making en-route decisions regarding driving times, breaks, and rest periods, under uncertain conditions. Our proposal aims at maximizing the likelihood of on-time delivery while complying with drivers’ work regulations. We use an online model-based RL strategy that needs no prior training and is more flexible than model-free RL approaches, where the agent must be trained offline before making online decisions. Our proposal combines model predictive control with a rollout strategy and Monte Carlo tree search. At each decision stage, our algorithm anticipates the consequences of all the possible decisions in a number of future stages (the lookahead horizon), and then uses a base policy to generate a sequence of decisions beyond the lookahead horizon. This base policy could be, for example, a set of decision rules based on the experience and expertise of the transportation company covering the routes. Our numerical results show that the policy obtained using our algorithm outperforms not only the base policy (up to 83%), but also a policy obtained offline using deep Q networks (DQN), a state-of-the-art, model-free RL algorithm.

Suggested Citation

  • Alcaraz, Juan J. & Losilla, Fernando & Caballero-Arnaldos, Luis, 2022. "Online model-based reinforcement learning for decision-making in long distance routes," Transportation Research Part E: Logistics and Transportation Review, Elsevier, vol. 164(C).
  • Handle: RePEc:eee:transe:v:164:y:2022:i:c:s136655452200179x
    DOI: 10.1016/j.tre.2022.102790
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S136655452200179X
    Download Restriction: Full text for ScienceDirect subscribers only

    File URL: https://libkey.io/10.1016/j.tre.2022.102790?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Gilbert Laporte, 2016. "Scheduling issues in vehicle routing," Annals of Operations Research, Springer, vol. 236(2), pages 463-474, January.
    2. A. L. Kok & C. M. Meyer & H. Kopfer & J. M. J. Schutten, 2010. "A Dynamic Programming Heuristic for the Vehicle Routing Problem with Time Windows and European Community Social Legislation," Transportation Science, INFORMS, vol. 44(4), pages 442-454, November.
    3. Asvin Goel, 2009. "Vehicle Scheduling and Routing with Drivers' Working Hours," Transportation Science, INFORMS, vol. 43(1), pages 17-26, February.
    4. Gilbert Laporte, 2016. "Scheduling issues in vehicle routing," Annals of Operations Research, Springer, vol. 236(2), pages 463-474, January.
    5. Volodymyr Mnih & Koray Kavukcuoglu & David Silver & Andrei A. Rusu & Joel Veness & Marc G. Bellemare & Alex Graves & Martin Riedmiller & Andreas K. Fidjeland & Georg Ostrovski & Stig Petersen & Charle, 2015. "Human-level control through deep reinforcement learning," Nature, Nature, vol. 518(7540), pages 529-533, February.
    6. Eric Prescott-Gagnon & Guy Desaulniers & Michael Drexl & Louis-Martin Rousseau, 2010. "European Driver Rules in Vehicle Routing with Time Windows," Transportation Science, INFORMS, vol. 44(4), pages 455-473, November.
    7. Alcaraz, Juan J. & Caballero-Arnaldos, Luis & Vales-Alonso, Javier, 2019. "Rich vehicle routing problem with last-mile outsourcing decisions," Transportation Research Part E: Logistics and Transportation Review, Elsevier, vol. 129(C), pages 263-286.
    8. Li, Xiangyong & Tian, Peng & Leung, Stephen C.H., 2010. "Vehicle routing problems with time windows and stochastic travel and service times: Models and algorithm," International Journal of Production Economics, Elsevier, vol. 125(1), pages 137-145, May.
    9. Anastasios D. Vareias & Panagiotis P. Repoussis & Panagiotis P. Repoussi, 2019. "Assessing Customer Service Reliability in Route Planning with Self-Imposed Time Windows and Stochastic Travel Times," Service Science, INFORMS, vol. 53(1), pages 256-281, February.
    10. Goel, Asvin, 2018. "Legal aspects in road transport optimization in Europe," Transportation Research Part E: Logistics and Transportation Review, Elsevier, vol. 114(C), pages 144-162.
    11. Bernhardt, A. & Melo, Teresa & Bousonville, Thomas & Kopfer, Herbert, 2016. "Scheduling of driver activities with multiple soft time windows considering European regulations on rest periods and breaks," Technical Reports on Logistics of the Saarland Business School 12, Saarland University of Applied Sciences (htw saar), Saarland Business School.
    12. Liu, Shan & Jiang, Hai & Chen, Shuiping & Ye, Jing & He, Renqing & Sun, Zhizhao, 2020. "Integrating Dijkstra’s algorithm into deep inverse reinforcement learning for food delivery route planning," Transportation Research Part E: Logistics and Transportation Review, Elsevier, vol. 142(C).
    13. Bernhardt, A. & Melo, Teresa & Bousonville, Thomas & Kopfer, Herbert, 2017. "Truck driver scheduling with combined planning of rest periods, breaks and vehicle refueling," Technical Reports on Logistics of the Saarland Business School 14, Saarland University of Applied Sciences (htw saar), Saarland Business School.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Ding, Yida & Wandelt, Sebastian & Wu, Guohua & Xu, Yifan & Sun, Xiaoqian, 2023. "Towards efficient airline disruption recovery with reinforcement learning," Transportation Research Part E: Logistics and Transportation Review, Elsevier, vol. 179(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Daiane Maria Genaro Chiroli & Sérgio Fernando Mayerle & João Neiva Figueiredo, 2022. "Using state-space shortest-path heuristics to solve the long-haul point-to-point vehicle routing and driver scheduling problem subject to hours-of-service regulatory constraints," Journal of Heuristics, Springer, vol. 28(1), pages 23-59, February.
    2. Mayerle, Sérgio Fernando & De Genaro Chiroli, Daiane Maria & Neiva de Figueiredo, João & Rodrigues, Hidelbrando Ferreira, 2020. "The long-haul full-load vehicle routing and truck driver scheduling problem with intermediate stops: An economic impact evaluation of Brazilian policy," Transportation Research Part A: Policy and Practice, Elsevier, vol. 140(C), pages 36-51.
    3. Ostermeier, Manuel, 2024. "The supply of convenience stores: Challenges of short-distance routing within the constraints of working time regulations," European Journal of Operational Research, Elsevier, vol. 314(3), pages 997-1012.
    4. Christian Tilk & Asvin Goel, 2019. "Bidirectional labeling for solving vehicle routing and truck driver scheduling problems," Working Papers 1914, Gutenberg School of Management and Economics, Johannes Gutenberg-Universität Mainz.
    5. Alcaraz, Juan J. & Caballero-Arnaldos, Luis & Vales-Alonso, Javier, 2019. "Rich vehicle routing problem with last-mile outsourcing decisions," Transportation Research Part E: Logistics and Transportation Review, Elsevier, vol. 129(C), pages 263-286.
    6. Tilk, Christian & Goel, Asvin, 2020. "Bidirectional labeling for solving vehicle routing and truck driver scheduling problems," European Journal of Operational Research, Elsevier, vol. 283(1), pages 108-124.
    7. Asvin Goel & Thibaut Vidal & Adrianus Leendert Kok, 2021. "To team up or not: single versus team driving in European road freight transport," Flexible Services and Manufacturing Journal, Springer, vol. 33(4), pages 879-913, December.
    8. Mor, Andrea & Archetti, Claudia & Jabali, Ola & Simonetto, Alberto & Speranza, M. Grazia, 2022. "The Bi-objective Long-haul Transportation Problem on a Road Network," Omega, Elsevier, vol. 106(C).
    9. Asvin Goel & Thibaut Vidal, 2014. "Hours of Service Regulations in Road Freight Transport: An Optimization-Based International Assessment," Transportation Science, INFORMS, vol. 48(3), pages 391-412, August.
    10. Stehbeck, Florian, 2019. "Designing and Scheduling Cost-Efficient Tours by Using the Concept of Truck Platooning," Junior Management Science (JUMS), Junior Management Science e. V., vol. 4(4), pages 566-634.
    11. Derigs, Ulrich & Kurowsky, René & Vogel, Ulrich, 2011. "Solving a real-world vehicle routing problem with multiple use of tractors and trailers and EU-regulations for drivers arising in air cargo road feeder services," European Journal of Operational Research, Elsevier, vol. 213(1), pages 309-319, August.
    12. Gilbert Laporte, 2016. "Scheduling issues in vehicle routing," Annals of Operations Research, Springer, vol. 236(2), pages 463-474, January.
    13. Koç, Çağrı & Bektaş, Tolga & Jabali, Ola & Laporte, Gilbert, 2016. "A comparison of three idling options in long-haul truck scheduling," Transportation Research Part B: Methodological, Elsevier, vol. 93(PA), pages 631-647.
    14. Asvin Goel & Leendert Kok, 2012. "Truck Driver Scheduling in the United States," Transportation Science, INFORMS, vol. 46(3), pages 317-326, August.
    15. Vidal, Thibaut & Laporte, Gilbert & Matl, Piotr, 2020. "A concise guide to existing and emerging vehicle routing problem variants," European Journal of Operational Research, Elsevier, vol. 286(2), pages 401-416.
    16. Asvin Goel & Stefan Irnich, 2017. "An Exact Method for Vehicle Routing and Truck Driver Scheduling Problems," Transportation Science, INFORMS, vol. 51(2), pages 737-754, May.
    17. Andrew Lim & Zhenzhen Zhang & Hu Qin, 2017. "Pickup and Delivery Service with Manpower Planning in Hong Kong Public Hospitals," Transportation Science, INFORMS, vol. 51(2), pages 688-705, May.
    18. Maximilian Schiffer & Michael Schneider & Grit Walther & Gilbert Laporte, 2019. "Vehicle Routing and Location Routing with Intermediate Stops: A Review," Transportation Science, INFORMS, vol. 53(2), pages 319-343, March.
    19. Gilbert Laporte, 2016. "Scheduling issues in vehicle routing," Annals of Operations Research, Springer, vol. 236(2), pages 463-474, January.
    20. Goel, Asvin, 2018. "Legal aspects in road transport optimization in Europe," Transportation Research Part E: Logistics and Transportation Review, Elsevier, vol. 114(C), pages 144-162.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:transe:v:164:y:2022:i:c:s136655452200179x. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/wps/find/journaldescription.cws_home/600244/description#description .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.