IDEAS home Printed from https://ideas.repec.org/a/eee/energy/v284y2023ics0360544223027299.html
   My bibliography  Save this article

Multi-agent reinforcement learning for electric vehicle decarbonized routing and scheduling

Author

Listed:
  • Wang, Yi
  • Qiu, Dawei
  • He, Yinglong
  • Zhou, Quan
  • Strbac, Goran

Abstract

Low-carbon transitions require joint efforts from electricity grid and transport network, where electric vehicles (EVs) play a key role. Particularly, EVs can reduce the carbon emissions of transport networks through eco-routing while providing the carbon intensity service for power networks via vehicle-to-grid technique. Distinguishing from previous research that focused on EV routing and scheduling problems separately, this paper studies their coordinated effect with the objective of carbon emission reduction on both sides. To solve this problem, we propose a multi-agent reinforcement learning method that does not rely on prior knowledge of the system and can adapt to various uncertainties and dynamics. The proposed method learns a hierarchical structure for the mutually exclusive discrete routing and continuous scheduling decisions via a hybrid policy. Extensive case studies based on a virtual 7-node 10-edge transport and 15-bus power network as well as a coupled real-world central London transport and 33-bus power network are developed to demonstrate the effectiveness of the proposed MARL method on reducing carbon emissions in transport network and providing carbon intensity service in power network.

Suggested Citation

  • Wang, Yi & Qiu, Dawei & He, Yinglong & Zhou, Quan & Strbac, Goran, 2023. "Multi-agent reinforcement learning for electric vehicle decarbonized routing and scheduling," Energy, Elsevier, vol. 284(C).
    • Handle: RePEc:eee:energy:v:284:y:2023:i:c:s0360544223027299
    DOI: 10.1016/j.energy.2023.129335
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0360544223027299
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.energy.2023.129335?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Heymann, Fabian & Milojevic, Tatjana & Covatariu, Andrei & Verma, Piyush, 2023. "Digitalization in decarbonizing electricity systems – Phenomena, regional aspects, stakeholders, use cases, challenges and policy options," Energy, Elsevier, vol. 262(PB).
    2. Franceschetti, Anna & Honhon, Dorothée & Van Woensel, Tom & Bektaş, Tolga & Laporte, Gilbert, 2013. "The time-dependent pollution-routing problem," Transportation Research Part B: Methodological, Elsevier, vol. 56(C), pages 265-293.
    3. Dixon, James & Bukhsh, Waqquas & Edmunds, Calum & Bell, Keith, 2020. "Scheduling electric vehicle charging to minimise carbon emissions and wind curtailment," Renewable Energy, Elsevier, vol. 161(C), pages 1072-1091.
    4. Daryabari, Mohamad K. & Keypour, Reza & Golmohamadi, Hessam, 2020. "Stochastic energy management of responsive plug-in electric vehicles characterizing parking lot aggregators," Applied Energy, Elsevier, vol. 279(C).
    5. Qiu, Dawei & Wang, Yi & Hua, Weiqi & Strbac, Goran, 2023. "Reinforcement learning for electric vehicle applications in power systems:A critical review," Renewable and Sustainable Energy Reviews, Elsevier, vol. 173(C).
    6. Jiang, C.X. & Jing, Z.X. & Cui, X.R. & Ji, T.Y. & Wu, Q.H., 2018. "Multiple agents and reinforcement learning for modelling charging loads of electric taxis," Applied Energy, Elsevier, vol. 222(C), pages 158-168.
    7. Colmenar-Santos, Antonio & Muñoz-Gómez, Antonio-Miguel & Rosales-Asensio, Enrique & López-Rey, África, 2019. "Electric vehicle charging strategy to support renewable energy sources in Europe 2050 low-carbon scenario," Energy, Elsevier, vol. 183(C), pages 61-74.
    8. Alqahtani, Mohammed & Hu, Mengqi, 2022. "Dynamic energy scheduling and routing of multiple electric vehicles using deep reinforcement learning," Energy, Elsevier, vol. 244(PA).
    9. Tim Schwanen, 2021. "Achieving just transitions to low-carbon urban mobility," Nature Energy, Nature, vol. 6(7), pages 685-687, July.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Garside, Annisa Kesy & Ahmad, Robiah & Muhtazaruddin, Mohd Nabil Bin, 2024. "A recent review of solution approaches for green vehicle routing problem and its variants," Operations Research Perspectives, Elsevier, vol. 12(C).
    2. Wang, Hongfei & Guan, Hongzhi & Qin, Huanmei & Zhao, Pengfei, 2024. "Assessing the sustainability of time-dependent electric demand responsive transit service through deep reinforcement learning," Energy, Elsevier, vol. 296(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Zhang, Gang & Wen, Jiaxing & Xie, Tuo & Zhang, Kaoshe & Jia, Rong, 2023. "Bi-layer economic scheduling for integrated energy system based on source-load coordinated carbon reduction," Energy, Elsevier, vol. 280(C).
    2. Qiu, Dawei & Wang, Yi & Sun, Mingyang & Strbac, Goran, 2022. "Multi-service provision for electric vehicles in power-transportation networks towards a low-carbon transition: A hierarchical and hybrid multi-agent reinforcement learning approach," Applied Energy, Elsevier, vol. 313(C).
    3. Qiu, Dawei & Wang, Yi & Hua, Weiqi & Strbac, Goran, 2023. "Reinforcement learning for electric vehicle applications in power systems:A critical review," Renewable and Sustainable Energy Reviews, Elsevier, vol. 173(C).
    4. Zhang, Rui & Yu, Jilai, 2024. "Evaluating multi-dimensional response capability of electric bus considering carbon emissions and traffic index," Energy, Elsevier, vol. 286(C).
    5. Yin, Linfei & Zhang, Bin, 2021. "Time series generative adversarial network controller for long-term smart generation control of microgrids," Applied Energy, Elsevier, vol. 281(C).
    6. Jean-Michel Clairand & Paulo Guerra-Terán & Xavier Serrano-Guerrero & Mario González-Rodríguez & Guillermo Escrivá-Escrivá, 2019. "Electric Vehicles for Public Transportation in Power Systems: A Review of Methodologies," Energies, MDPI, vol. 12(16), pages 1-22, August.
    7. Youssef Amry & Elhoussin Elbouchikhi & Franck Le Gall & Mounir Ghogho & Soumia El Hani, 2022. "Electric Vehicle Traction Drives and Charging Station Power Electronics: Current Status and Challenges," Energies, MDPI, vol. 15(16), pages 1-30, August.
    8. Ehmke, Jan Fabian & Campbell, Ann M. & Thomas, Barrett W., 2018. "Optimizing for total costs in vehicle routing in urban areas," Transportation Research Part E: Logistics and Transportation Review, Elsevier, vol. 116(C), pages 242-265.
    9. Yagcitekin, Bunyamin & Uzunoglu, Mehmet, 2016. "A double-layer smart charging strategy of electric vehicles taking routing and charge scheduling into account," Applied Energy, Elsevier, vol. 167(C), pages 407-419.
    10. Furini, Fabio & Persiani, Carlo Alfredo & Toth, Paolo, 2016. "The Time Dependent Traveling Salesman Planning Problem in Controlled Airspace," Transportation Research Part B: Methodological, Elsevier, vol. 90(C), pages 38-55.
    11. Xuanjing Fang & Yanan Du & Yuzhuo Qiu, 2017. "Reducing Carbon Emissions in a Closed-Loop Production Routing Problem with Simultaneous Pickups and Deliveries under Carbon Cap-and-Trade," Sustainability, MDPI, vol. 9(12), pages 1-15, November.
    12. Zhu, Dafeng & Yang, Bo & Ma, Chengbin & Wang, Zhaojian & Zhu, Shanying & Ma, Kai & Guan, Xinping, 2022. "Stochastic gradient-based fast distributed multi-energy management for an industrial park with temporally-coupled constraints," Applied Energy, Elsevier, vol. 317(C).
    13. Raeesi, Ramin & Zografos, Konstantinos G., 2020. "The electric vehicle routing problem with time windows and synchronised mobile battery swapping," Transportation Research Part B: Methodological, Elsevier, vol. 140(C), pages 101-129.
    14. Paraskevopoulos, Dimitris C. & Gürel, Sinan & Bektaş, Tolga, 2016. "The congested multicommodity network design problem," Transportation Research Part E: Logistics and Transportation Review, Elsevier, vol. 85(C), pages 166-187.
    15. Roberto Tadei & Guido Perboli & Francesca Perfetti, 2017. "The multi-path Traveling Salesman Problem with stochastic travel costs," EURO Journal on Transportation and Logistics, Springer;EURO - The Association of European Operational Research Societies, vol. 6(1), pages 3-23, March.
    16. Liu, Xuan & Xue, Jilai, 2019. "The role of Al2Gd cuboids in the discharge performance and electrochemical behaviors of AZ31-Gd anode for Mg-air batteries," Energy, Elsevier, vol. 189(C).
    17. Yu, Yang & Wu, Yuting & Wang, Junwei, 2019. "Bi-objective green ride-sharing problem: Model and exact method," International Journal of Production Economics, Elsevier, vol. 208(C), pages 472-482.
    18. Benjamin K. Sovacool & Paul Upham & Mari Martiskainen & Kirsten E. H. Jenkins & Gerardo A. Torres Contreras & Neil Simcock, 2023. "Policy prescriptions to address energy and transport poverty in the United Kingdom," Nature Energy, Nature, vol. 8(3), pages 273-283, March.
    19. Zhang, Zhenzhen & Wei, Lijun & Lim, Andrew, 2015. "An evolutionary local search for the capacitated vehicle routing problem minimizing fuel consumption under three-dimensional loading constraints," Transportation Research Part B: Methodological, Elsevier, vol. 82(C), pages 20-35.
    20. Chengzhou Li & Ningling Wang & Hongyuan Zhang & Qingxin Liu & Youguo Chai & Xiaohu Shen & Zhiping Yang & Yongping Yang, 2019. "Environmental Impact Evaluation of Distributed Renewable Energy System Based on Life Cycle Assessment and Fuzzy Rough Sets," Energies, MDPI, vol. 12(21), pages 1-17, November.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:energy:v:284:y:2023:i:c:s0360544223027299. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.journals.elsevier.com/energy .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.