IDEAS home Printed from https://ideas.repec.org/a/eee/transe/v157y2022ics1366554521002581.html
   My bibliography  Save this article

Dynamic stochastic electric vehicle routing with safe reinforcement learning

Author

Listed:
  • Basso, Rafael
  • Kulcsár, Balázs
  • Sanchez-Diaz, Ivan
  • Qu, Xiaobo

Abstract

Dynamic routing of electric commercial vehicles can be a challenging problem since besides the uncertainty of energy consumption there are also random customer requests. This paper introduces the Dynamic Stochastic Electric Vehicle Routing Problem (DS-EVRP). A Safe Reinforcement Learning method is proposed for solving the problem. The objective is to minimize expected energy consumption in a safe way, which means also minimizing the risk of battery depletion while en route by planning charging whenever necessary. The key idea is to learn offline about the stochastic customer requests and energy consumption using Monte Carlo simulations, to be able to plan the route predictively and safely online. The method is evaluated using simulations based on energy consumption data from a realistic traffic model for the city of Luxembourg and a high-fidelity vehicle model. The results indicate that it is possible to save energy at the same time maintaining reliability by planning the routes and charging in an anticipative way. The proposed method has the potential to improve transport operations with electric commercial vehicles capitalizing on their environmental benefits.

Suggested Citation

  • Basso, Rafael & Kulcsár, Balázs & Sanchez-Diaz, Ivan & Qu, Xiaobo, 2022. "Dynamic stochastic electric vehicle routing with safe reinforcement learning," Transportation Research Part E: Logistics and Transportation Review, Elsevier, vol. 157(C).
    • Handle: RePEc:eee:transe:v:157:y:2022:i:c:s1366554521002581
    DOI: 10.1016/j.tre.2021.102496
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S1366554521002581
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.tre.2021.102496?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Gendreau, Michel & Laporte, Gilbert & Seguin, Rene, 1996. "Stochastic vehicle routing," European Journal of Operational Research, Elsevier, vol. 88(1), pages 3-12, January.
    2. Gilbert Laporte & François Louveaux & Hélène Mercure, 1992. "The Vehicle Routing Problem with Stochastic Travel Times," Transportation Science, INFORMS, vol. 26(3), pages 161-170, August.
    3. Tolga Bektaş & Emrah Demir & Gilbert Laporte, 2016. "Green Vehicle Routing," International Series in Operations Research & Management Science, in: Harilaos N. Psaraftis (ed.), Green Transportation Logistics, edition 127, chapter 0, pages 243-265, Springer.
    4. Bektas, Tolga & Laporte, Gilbert, 2011. "The Pollution-Routing Problem," Transportation Research Part B: Methodological, Elsevier, vol. 45(8), pages 1232-1250, September.
    5. Ivan Sanchez-Diaz & Laura Palacios-Argüello & Anders Levandi & Jimmy Mardberg & Rafael Basso, 2020. "A Time-Efficiency Study of Medium-Duty Trucks Delivering in Urban Environments," Sustainability, MDPI, vol. 12(1), pages 1-15, January.
    6. Novoa, Clara & Storer, Robert, 2009. "An approximate dynamic programming approach for the vehicle routing problem with stochastic demands," European Journal of Operational Research, Elsevier, vol. 196(2), pages 509-515, July.
    7. Tahami, Hesamoddin & Rabadi, Ghaith & Haouari, Mohamed, 2020. "Exact approaches for routing capacitated electric vehicles," Transportation Research Part E: Logistics and Transportation Review, Elsevier, vol. 144(C).
    8. Erdoğan, Sevgi & Miller-Hooks, Elise, 2012. "A Green Vehicle Routing Problem," Transportation Research Part E: Logistics and Transportation Review, Elsevier, vol. 48(1), pages 100-114.
    9. Demir, Emrah & Bektaş, Tolga & Laporte, Gilbert, 2014. "A review of recent research on green road freight transportation," European Journal of Operational Research, Elsevier, vol. 237(3), pages 775-793.
    10. Justin C. Goodson & Jeffrey W. Ohlmann & Barrett W. Thomas, 2013. "Rollout Policies for Dynamic Solutions to the Multivehicle Routing Problem with Stochastic Demand and Duration Limits," Operations Research, INFORMS, vol. 61(1), pages 138-154, February.
    11. Basso, Rafael & Kulcsár, Balázs & Sanchez-Diaz, Ivan, 2021. "Electric vehicle routing problem with machine learning for energy prediction," Transportation Research Part B: Methodological, Elsevier, vol. 145(C), pages 24-55.
    12. Bektaş, Tolga & Ehmke, Jan Fabian & Psaraftis, Harilaos N. & Puchinger, Jakob, 2019. "The role of operational research in green freight transportation," European Journal of Operational Research, Elsevier, vol. 274(3), pages 807-823.
    13. Timothy M. Sweda & Irina S. Dolinskaya & Diego Klabjan, 2017. "Adaptive Routing and Recharging Policies for Electric Vehicles," Transportation Science, INFORMS, vol. 51(4), pages 1326-1348, November.
    14. Nicola Secomandi, 2001. "A Rollout Policy for the Vehicle Routing Problem with Stochastic Demands," Operations Research, INFORMS, vol. 49(5), pages 796-802, October.
    15. Pelletier, Samuel & Jabali, Ola & Laporte, Gilbert, 2019. "The electric vehicle routing problem with energy consumption uncertainty," Transportation Research Part B: Methodological, Elsevier, vol. 126(C), pages 225-255.
    16. Michael Schneider & Andreas Stenger & Dominik Goeke, 2014. "The Electric Vehicle-Routing Problem with Time Windows and Recharging Stations," Transportation Science, INFORMS, vol. 48(4), pages 500-520, November.
    17. Cuma, Mehmet Ugras & Koroglu, Tahsin, 2015. "A comprehensive review on estimation strategies used in hybrid and battery electric vehicles," Renewable and Sustainable Energy Reviews, Elsevier, vol. 42(C), pages 517-531.
    18. Adler, Jonathan D. & Mirchandani, Pitu B., 2014. "Online routing and battery reservations for electric vehicles with swappable batteries," Transportation Research Part B: Methodological, Elsevier, vol. 70(C), pages 285-302.
    19. Michel Gendreau & Ola Jabali & Walter Rei, 2016. "50th Anniversary Invited Article—Future Research Directions in Stochastic Vehicle Routing," Transportation Science, INFORMS, vol. 50(4), pages 1163-1173, November.
    20. Felipe, Ángel & Ortuño, M. Teresa & Righini, Giovanni & Tirado, Gregorio, 2014. "A heuristic approach for the green vehicle routing problem with multiple technologies and partial recharges," Transportation Research Part E: Logistics and Transportation Review, Elsevier, vol. 71(C), pages 111-128.
    21. Jorge Oyola & Halvard Arntzen & David L. Woodruff, 2018. "The stochastic vehicle routing problem, a literature review, part I: models," EURO Journal on Transportation and Logistics, Springer;EURO - The Association of European Operational Research Societies, vol. 7(3), pages 193-221, September.
    22. Jochem, Patrick & Doll, Claus & Fichtner, Wolf, 2016. "External costs of electric vehicles," MPRA Paper 91602, University Library of Munich, Germany.
    23. Marlin W. Ulmer & Justin C. Goodson & Dirk C. Mattfeld & Marco Hennig, 2019. "Offline–Online Approximate Dynamic Programming for Dynamic Vehicle Routing with Stochastic Requests," Service Science, INFORMS, vol. 53(1), pages 185-202, February.
    24. Schneider, M. & Stenger, A. & Goeke, D., 2014. "The Electric Vehicle Routing Problem with Time Windows and Recharging Stations," Publications of Darmstadt Technical University, Institute for Business Studies (BWL) 62382, Darmstadt Technical University, Department of Business Administration, Economics and Law, Institute for Business Studies (BWL).
    25. Liu, Shan & Jiang, Hai & Chen, Shuiping & Ye, Jing & He, Renqing & Sun, Zhizhao, 2020. "Integrating Dijkstra’s algorithm into deep inverse reinforcement learning for food delivery route planning," Transportation Research Part E: Logistics and Transportation Review, Elsevier, vol. 142(C).
    26. Masmoudi, Mohamed Amine & Hosny, Manar & Demir, Emrah & Genikomsakis, Konstantinos N. & Cheikhrouhou, Naoufel, 2018. "The dial-a-ride problem with electric vehicles and battery swapping stations," Transportation Research Part E: Logistics and Transportation Review, Elsevier, vol. 118(C), pages 392-420.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Xabier A. Martin & Marc Escoto & Antoni Guerrero & Angel A. Juan, 2024. "Battery Management in Electric Vehicle Routing Problems: A Review," Energies, MDPI, vol. 17(5), pages 1-25, February.
    2. Ding, Yida & Wandelt, Sebastian & Wu, Guohua & Xu, Yifan & Sun, Xiaoqian, 2023. "Towards efficient airline disruption recovery with reinforcement learning," Transportation Research Part E: Logistics and Transportation Review, Elsevier, vol. 179(C).
    3. Guo, Feng & Wei, Qu & Wang, Miao & Guo, Zhaoxia & Wallace, Stein W., 2023. "Deep attention models with dimension-reduction and gate mechanisms for solving practical time-dependent vehicle routing problems," Transportation Research Part E: Logistics and Transportation Review, Elsevier, vol. 173(C).
    4. Pourvaziri, H. & Sarhadi, H. & Azad, N. & Afshari, H. & Taghavi, M., 2024. "Planning of electric vehicle charging stations: An integrated deep learning and queueing theory approach," Transportation Research Part E: Logistics and Transportation Review, Elsevier, vol. 186(C).
    5. Fescioglu-Unver, Nilgun & Yıldız Aktaş, Melike, 2023. "Electric vehicle charging service operations: A review of machine learning applications for infrastructure planning, control, pricing and routing," Renewable and Sustainable Energy Reviews, Elsevier, vol. 188(C).
    6. Garside, Annisa Kesy & Ahmad, Robiah & Muhtazaruddin, Mohd Nabil Bin, 2024. "A recent review of solution approaches for green vehicle routing problem and its variants," Operations Research Perspectives, Elsevier, vol. 12(C).
    7. Marouane Adnane & Ahmed Khoumsi & João Pedro F. Trovão, 2023. "Efficient Management of Energy Consumption of Electric Vehicles Using Machine Learning—A Systematic and Comprehensive Survey," Energies, MDPI, vol. 16(13), pages 1-39, June.
    8. Amine Masmoudi, M. & Coelho, Leandro C. & Demir, Emrah, 2022. "Plug-in hybrid electric refuse vehicle routing problem for waste collection," Transportation Research Part E: Logistics and Transportation Review, Elsevier, vol. 166(C).
    9. Wang, Jianxin & Lim, Ming K. & Liu, Weihua, 2024. "Promoting intelligent IoT-driven logistics through integrating dynamic demand and sustainable logistics operations," Transportation Research Part E: Logistics and Transportation Review, Elsevier, vol. 185(C).
    10. Tan, Zhen & Liu, Fan & Chan, Hing Kai & Gao, H. Oliver, 2022. "Transportation systems management considering dynamic wireless charging electric vehicles: Review and prospects," Transportation Research Part E: Logistics and Transportation Review, Elsevier, vol. 163(C).
    11. Gu, Jianqiang & Wu, Zhan & Song, Yubing & Nicolescu, Ana-Cristina, 2024. "A win-win relationship? New evidence on artificial intelligence and new energy vehicles," Energy Economics, Elsevier, vol. 134(C).
    12. Zhao, Zhonghao & Lee, Carman K.M. & Huo, Jiage, 2023. "EV charging station deployment on coupled transportation and power distribution networks via reinforcement learning," Energy, Elsevier, vol. 267(C).
    13. Yong Wang & Jiayi Zhe & Xiuwen Wang & Yaoyao Sun & Haizhong Wang, 2022. "Collaborative Multidepot Vehicle Routing Problem with Dynamic Customer Demands and Time Windows," Sustainability, MDPI, vol. 14(11), pages 1-37, May.
    14. Moiz Ahmad & Muhammad Babar Ramzan & Muhammad Omair & Muhammad Salman Habib, 2024. "Integrating Risk-Averse and Constrained Reinforcement Learning for Robust Decision-Making in High-Stakes Scenarios," Mathematics, MDPI, vol. 12(13), pages 1-32, June.
    15. Nikola Mardešić & Tomislav Erdelić & Tonči Carić & Marko Đurasević, 2023. "Review of Stochastic Dynamic Vehicle Routing in the Evolving Urban Logistics Environment," Mathematics, MDPI, vol. 12(1), pages 1-44, December.
    16. Li Ma & Minghan Xin & Yi-Jia Wang & Yanjiao Zhang, 2022. "Dynamic Scheduling Strategy for Shared Agricultural Machinery for On-Demand Farming Services," Mathematics, MDPI, vol. 10(21), pages 1-22, October.
    17. Seyfi, Majid & Alinaghian, Mahdi & Ghorbani, Erfan & Çatay, Bülent & Saeid Sabbagh, Mohammad, 2022. "Multi-mode hybrid electric vehicle routing problem," Transportation Research Part E: Logistics and Transportation Review, Elsevier, vol. 166(C).
    18. Liu, Zeyu & Li, Xueping & Khojandi, Anahita, 2022. "The flying sidekick traveling salesman problem with stochastic travel time: A reinforcement learning approach," Transportation Research Part E: Logistics and Transportation Review, Elsevier, vol. 164(C).
    19. Xie, Jiaohong & Yang, Zhenyu & Lai, Xiongfei & Liu, Yang & Yang, Xiao Bo & Teng, Teck-Hou & Tham, Chen-Khong, 2022. "Deep reinforcement learning for dynamic incident-responsive traffic information dissemination," Transportation Research Part E: Logistics and Transportation Review, Elsevier, vol. 166(C).
    20. Chen, Xi & Li, Kaiwen & Lin, Sidian & Ding, Xiaosong, 2024. "Technician routing and scheduling with employees’ learning through implicit cross-training strategy," International Journal of Production Economics, Elsevier, vol. 271(C).
    21. Chen, Xinyuan & Wu, Shining & Liu, Yannick & Wu, Weiwei & Wang, Shuaian, 2022. "A patrol routing problem for maritime Crime-Fighting," Transportation Research Part E: Logistics and Transportation Review, Elsevier, vol. 168(C).
    22. Guo, Yuhan & Li, Jinning & Xiao, Linfan & Allaoui, Hamid & Choudhary, Alok & Zhang, Lufang, 2024. "Efficient inventory routing for Bike-Sharing Systems: A combinatorial reinforcement learning framework," Transportation Research Part E: Logistics and Transportation Review, Elsevier, vol. 182(C).
    23. Meryem Abid & Mohamed Tabaa & Hanaa Hachimi, 2024. "Electric Vehicle Routing Problem with an Enhanced Vehicle Dispatching Approach Considering Real-Life Data," Energies, MDPI, vol. 17(7), pages 1-27, March.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Asghari, Mohammad & Mirzapour Al-e-hashem, S. Mohammad J., 2021. "Green vehicle routing problem: A state-of-the-art review," International Journal of Production Economics, Elsevier, vol. 231(C).
    2. Mohammad Asghari & Seyed Mohammad Javad Mirzapour Al-E-Hashem, 2021. "Green vehicle routing problem: A state-of-the-art review," Post-Print hal-03182944, HAL.
    3. Raeesi, Ramin & Zografos, Konstantinos G., 2022. "Coordinated routing of electric commercial vehicles with intra-route recharging and en-route battery swapping," European Journal of Operational Research, Elsevier, vol. 301(1), pages 82-109.
    4. Amine Masmoudi, M. & Coelho, Leandro C. & Demir, Emrah, 2022. "Plug-in hybrid electric refuse vehicle routing problem for waste collection," Transportation Research Part E: Logistics and Transportation Review, Elsevier, vol. 166(C).
    5. Raeesi, Ramin & Zografos, Konstantinos G., 2020. "The electric vehicle routing problem with time windows and synchronised mobile battery swapping," Transportation Research Part B: Methodological, Elsevier, vol. 140(C), pages 101-129.
    6. Leandro do C. Martins & Rafael D. Tordecilla & Juliana Castaneda & Angel A. Juan & Javier Faulin, 2021. "Electric Vehicle Routing, Arc Routing, and Team Orienteering Problems in Sustainable Transportation," Energies, MDPI, vol. 14(16), pages 1-30, August.
    7. Masmoudi, Mohamed Amine & Hosny, Manar & Demir, Emrah & Genikomsakis, Konstantinos N. & Cheikhrouhou, Naoufel, 2018. "The dial-a-ride problem with electric vehicles and battery swapping stations," Transportation Research Part E: Logistics and Transportation Review, Elsevier, vol. 118(C), pages 392-420.
    8. Seyfi, Majid & Alinaghian, Mahdi & Ghorbani, Erfan & Çatay, Bülent & Saeid Sabbagh, Mohammad, 2022. "Multi-mode hybrid electric vehicle routing problem," Transportation Research Part E: Logistics and Transportation Review, Elsevier, vol. 166(C).
    9. Goeke, Dominik & Schneider, Michael, 2015. "Routing a mixed fleet of electric and conventional vehicles," European Journal of Operational Research, Elsevier, vol. 245(1), pages 81-99.
    10. Dönmez, Sercan & Koç, Çağrı & Altıparmak, Fulya, 2022. "The mixed fleet vehicle routing problem with partial recharging by multiple chargers: Mathematical model and adaptive large neighborhood search," Transportation Research Part E: Logistics and Transportation Review, Elsevier, vol. 167(C).
    11. Alexandre M. Florio & Nabil Absi & Dominique Feillet, 2021. "Routing Electric Vehicles on Congested Street Networks," Transportation Science, INFORMS, vol. 55(1), pages 238-256, 1-2.
    12. Goeke, D. & Schneider, M., 2015. "Routing a Mixed Fleet of Electric and Conventional Vehicles," Publications of Darmstadt Technical University, Institute for Business Studies (BWL) 65939, Darmstadt Technical University, Department of Business Administration, Economics and Law, Institute for Business Studies (BWL).
    13. Tahami, Hesamoddin & Rabadi, Ghaith & Haouari, Mohamed, 2020. "Exact approaches for routing capacitated electric vehicles," Transportation Research Part E: Logistics and Transportation Review, Elsevier, vol. 144(C).
    14. Zhang, Shuai & Gajpal, Yuvraj & Appadoo, S.S. & Abdulkader, M.M.S., 2018. "Electric vehicle routing problem with recharging stations for minimizing energy consumption," International Journal of Production Economics, Elsevier, vol. 203(C), pages 404-413.
    15. Leggieri, Valeria & Haouari, Mohamed, 2017. "A practical solution approach for the green vehicle routing problem," Transportation Research Part E: Logistics and Transportation Review, Elsevier, vol. 104(C), pages 97-112.
    16. Bektaş, Tolga & Ehmke, Jan Fabian & Psaraftis, Harilaos N. & Puchinger, Jakob, 2019. "The role of operational research in green freight transportation," European Journal of Operational Research, Elsevier, vol. 274(3), pages 807-823.
    17. Cortés-Murcia, David L. & Prodhon, Caroline & Murat Afsar, H., 2019. "The electric vehicle routing problem with time windows, partial recharges and satellite customers," Transportation Research Part E: Logistics and Transportation Review, Elsevier, vol. 130(C), pages 184-206.
    18. Alberto Ceselli & Ángel Felipe & M. Teresa Ortuño & Giovanni Righini & Gregorio Tirado, 2021. "A Branch-and-Cut-and-Price Algorithm for the Electric Vehicle Routing Problem with Multiple Technologies," SN Operations Research Forum, Springer, vol. 2(1), pages 1-33, March.
    19. Singh, Nitish & Dang, Quang-Vinh & Akcay, Alp & Adan, Ivo & Martagan, Tugce, 2022. "A matheuristic for AGV scheduling with battery constraints," European Journal of Operational Research, Elsevier, vol. 298(3), pages 855-873.
    20. Koyuncu, Işıl & Yavuz, Mesut, 2019. "Duplicating nodes or arcs in green vehicle routing: A computational comparison of two formulations," Transportation Research Part E: Logistics and Transportation Review, Elsevier, vol. 122(C), pages 605-623.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:transe:v:157:y:2022:i:c:s1366554521002581. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/wps/find/journaldescription.cws_home/600244/description#description .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.