IDEAS home Printed from https://ideas.repec.org/a/eee/appene/v285y2021ics0306261920317566.html
   My bibliography  Save this article

Development and Evaluation of a Smart Charging Strategy for an Electric Vehicle Fleet Based on Reinforcement Learning

Author

Listed:
  • Tuchnitz, Felix
  • Ebell, Niklas
  • Schlund, Jonas
  • Pruckner, Marco

Abstract

Governments are currently subsidizing growth in the electric car market and the associated infrastructure in order to accelerate the transition to more sustainable mobility. To avoid the grid overload that results from simultaneously charging too many electric vehicles, there is a need for smart charging coordination systems. In this paper, we propose a charging coordination system based on Reinforcement Learning using an artificial neural network as a function approximator. Taking into account the baseload present in the power grid, a central agent creates forward-looking, coordinated charging schedules for an electric vehicle fleet of any size. In contrast to optimization-based charging strategies, system dynamics such as future arrivals, departures, and energy consumption do not have to be known beforehand. We implement and compare a range of parameter variants that differ in terms of the reward function and prioritized experience. Subsequently, we use a case study to compare our Reinforcement Learning algorithm with several other charging strategies. The Reinforcement Learning-based charging coordination system is shown to perform very well. All electric vehicles have enough energy for their next trip on departure and charging is carried out almost exclusively during the load valleys at night. Compared with an uncontrolled charging strategy, the Reinforcement Learning algorithm reduces the variance of the total load by 65%. The performance of our Reinforcement Learning concept comes close to that of an optimization-based charging strategy. However, an optimization algorithm needs to know certain information beforehand, such as the vehicle’s departure time and its energy requirement on arriving at the charging station. Our novel Reinforcement Learning-based charging coordination system therefore offers a flexible, easily adaptable, and scalable approach for an electric vehicle fleet under realistic operating conditions.

Suggested Citation

  • Tuchnitz, Felix & Ebell, Niklas & Schlund, Jonas & Pruckner, Marco, 2021. "Development and Evaluation of a Smart Charging Strategy for an Electric Vehicle Fleet Based on Reinforcement Learning," Applied Energy, Elsevier, vol. 285(C).
    • Handle: RePEc:eee:appene:v:285:y:2021:i:c:s0306261920317566
    DOI: 10.1016/j.apenergy.2020.116382
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0306261920317566
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.apenergy.2020.116382?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Martin Spitzer & Jonas Schlund & Elpiniki Apostolaki-Iosifidou & Marco Pruckner, 2019. "Optimized Integration of Electric Vehicles in Low Voltage Distribution Grids," Energies, MDPI, vol. 12(21), pages 1-19, October.
    2. Monica Alonso & Hortensia Amaris & Jean Gardy Germain & Juan Manuel Galan, 2014. "Optimal Charging Scheduling of Electric Vehicles in Smart Grids by Heuristic Algorithms," Energies, MDPI, vol. 7(4), pages 1-27, April.
    3. Hu, Junjie & Morais, Hugo & Sousa, Tiago & Lind, Morten, 2016. "Electric vehicle fleet management in smart grids: A review of services, optimization and control aspects," Renewable and Sustainable Energy Reviews, Elsevier, vol. 56(C), pages 1207-1226.
    4. Hannan, M.A. & Lipu, M.S.H. & Hussain, A. & Mohamed, A., 2017. "A review of lithium-ion battery state of charge estimation and management system in electric vehicle applications: Challenges and recommendations," Renewable and Sustainable Energy Reviews, Elsevier, vol. 78(C), pages 834-854.
    5. Yang, Zhile & Li, Kang & Foley, Aoife, 2015. "Computational scheduling methods for integrating plug-in electric vehicles with power systems: A review," Renewable and Sustainable Energy Reviews, Elsevier, vol. 51(C), pages 396-416.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Paudel, Diwas & Das, Tapas K., 2023. "A deep reinforcement learning approach for power management of battery-assisted fast-charging EV hubs participating in day-ahead and real-time electricity markets," Energy, Elsevier, vol. 283(C).
    2. Jin, Ruiyang & Zhou, Yuke & Lu, Chao & Song, Jie, 2022. "Deep reinforcement learning-based strategy for charging station participating in demand response," Applied Energy, Elsevier, vol. 328(C).
    3. Samuel M. Muhindo & Roland P. Malhamé & Geza Joos, 2021. "A Novel Mean Field Game-Based Strategy for Charging Electric Vehicles in Solar Powered Parking Lots," Energies, MDPI, vol. 14(24), pages 1-21, December.
    4. Qiu, Dawei & Wang, Yi & Hua, Weiqi & Strbac, Goran, 2023. "Reinforcement learning for electric vehicle applications in power systems:A critical review," Renewable and Sustainable Energy Reviews, Elsevier, vol. 173(C).
    5. Wang, Yi & Qiu, Dawei & Strbac, Goran, 2022. "Multi-agent deep reinforcement learning for resilience-driven routing and scheduling of mobile energy storage systems," Applied Energy, Elsevier, vol. 310(C).
    6. Li, Xiaohui & Wang, Zhenpo & Zhang, Lei & Sun, Fengchun & Cui, Dingsong & Hecht, Christopher & Figgener, Jan & Sauer, Dirk Uwe, 2023. "Electric vehicle behavior modeling and applications in vehicle-grid integration: An overview," Energy, Elsevier, vol. 268(C).
    7. Abid, Md. Shadman & Apon, Hasan Jamil & Hossain, Salman & Ahmed, Ashik & Ahshan, Razzaqul & Lipu, M.S. Hossain, 2024. "A novel multi-objective optimization based multi-agent deep reinforcement learning approach for microgrid resources planning," Applied Energy, Elsevier, vol. 353(PA).
    8. Zeynali, Saeed & Nasiri, Nima & Marzband, Mousa & Ravadanegh, Sajad Najafi, 2021. "A hybrid robust-stochastic framework for strategic scheduling of integrated wind farm and plug-in hybrid electric vehicle fleets," Applied Energy, Elsevier, vol. 300(C).
    9. Zhao, Zhonghao & Lee, Carman K.M. & Huo, Jiage, 2023. "EV charging station deployment on coupled transportation and power distribution networks via reinforcement learning," Energy, Elsevier, vol. 267(C).
    10. Ahmed M. Abed & Ali AlArjani, 2022. "The Neural Network Classifier Works Efficiently on Searching in DQN Using the Autonomous Internet of Things Hybridized by the Metaheuristic Techniques to Reduce the EVs’ Service Scheduling Time," Energies, MDPI, vol. 15(19), pages 1-25, September.
    11. Castillo, Victhalia Zapata & Boer, Harmen-Sytze de & Muñoz, Raúl Maícas & Gernaat, David E.H.J. & Benders, René & van Vuuren, Detlef, 2022. "Future global electricity demand load curves," Energy, Elsevier, vol. 258(C).
    12. Amjad, Muhammad & Farooq-i-Azam, Muhammad & Ni, Qiang & Dong, Mianxiong & Ansari, Ejaz Ahmad, 2022. "Wireless charging systems for electric vehicles," Renewable and Sustainable Energy Reviews, Elsevier, vol. 167(C).
    13. Tepe, Benedikt & Figgener, Jan & Englberger, Stefan & Sauer, Dirk Uwe & Jossen, Andreas & Hesse, Holger, 2022. "Optimal pool composition of commercial electric vehicles in V2G fleet operation of various electricity markets," Applied Energy, Elsevier, vol. 308(C).
    14. Wald, Dylan & King, Jennifer & Bay, Christopher J. & Chintala, Rohit & Johnson, Kathryn, 2022. "Integration of distributed controllers: Power reference tracking through charging station and building coordination," Applied Energy, Elsevier, vol. 314(C).
    15. Tao, Yuechuan & Qiu, Jing & Lai, Shuying & Sun, Xianzhuo & Zhao, Junhua & Zhou, Baorong & Cheng, Lanfen, 2022. "Data-driven on-demand energy supplement planning for electric vehicles considering multi-charging/swapping services," Applied Energy, Elsevier, vol. 311(C).
    16. Aree Wangsupphaphol & Surachai Chaitusaney, 2022. "Subsidizing Residential Low Priority Smart Charging: A Power Management Strategy for Electric Vehicle in Thailand," Sustainability, MDPI, vol. 14(10), pages 1-15, May.
    17. Zhao, Zhonghao & Lee, Carman K.M. & Ren, Jingzheng, 2024. "A two-level charging scheduling method for public electric vehicle charging stations considering heterogeneous demand and nonlinear charging profile," Applied Energy, Elsevier, vol. 355(C).
    18. Omar Al-Ani & Sanjoy Das, 2022. "Reinforcement Learning: Theory and Applications in HEMS," Energies, MDPI, vol. 15(17), pages 1-37, September.
    19. Nandan Gopinathan & Prabhakar Karthikeyan Shanmugam, 2022. "Energy Anxiety in Decentralized Electricity Markets: A Critical Review on EV Models," Energies, MDPI, vol. 15(14), pages 1-40, July.
    20. Pegah Alaee & Julius Bems & Amjad Anvari-Moghaddam, 2023. "A Review of the Latest Trends in Technical and Economic Aspects of EV Charging Management," Energies, MDPI, vol. 16(9), pages 1-28, April.
    21. Norouzi, Mohammadali & Aghaei, Jamshid & Pirouzi, Sasan & Niknam, Taher & Fotuhi-Firuzabad, Mahmud & Shafie-khah, Miadreza, 2021. "Hybrid stochastic/robust flexible and reliable scheduling of secure networked microgrids with electric springs and electric vehicles," Applied Energy, Elsevier, vol. 300(C).
    22. Manzolli, Jônatas Augusto & Trovão, João Pedro & Antunes, Carlos Henggeler, 2022. "A review of electric bus vehicles research topics – Methods and trends," Renewable and Sustainable Energy Reviews, Elsevier, vol. 159(C).
    23. Xianhao Shen & Yexin Zhang & Decheng Wang, 2022. "Online Charging Strategy for Electric Vehicle Clusters Based on Multi-Agent Reinforcement Learning and Long–Short Memory Networks," Energies, MDPI, vol. 15(13), pages 1-14, June.
    24. Dimitrios Vamvakas & Panagiotis Michailidis & Christos Korkas & Elias Kosmatopoulos, 2023. "Review and Evaluation of Reinforcement Learning Frameworks on Smart Grid Applications," Energies, MDPI, vol. 16(14), pages 1-38, July.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Sajjad Haider & Peter Schegner, 2020. "Heuristic Optimization of Overloading Due to Electric Vehicles in a Low Voltage Grid," Energies, MDPI, vol. 13(22), pages 1-19, November.
    2. Shi You & Junjie Hu & Charalampos Ziras, 2016. "An Overview of Modeling Approaches Applied to Aggregation-Based Fleet Management and Integration of Plug-in Electric Vehicles †," Energies, MDPI, vol. 9(11), pages 1-18, November.
    3. Nezamoddini, Nasim & Wang, Yong, 2016. "Risk management and participation planning of electric vehicles in smart grids for demand response," Energy, Elsevier, vol. 116(P1), pages 836-850.
    4. Barone, G. & Buonomano, A. & Calise, F. & Forzano, C. & Palombo, A., 2019. "Building to vehicle to building concept toward a novel zero energy paradigm: Modelling and case studies," Renewable and Sustainable Energy Reviews, Elsevier, vol. 101(C), pages 625-648.
    5. Vasileios Boglou & Christos-Spyridon Karavas & Konstantinos Arvanitis & Athanasios Karlis, 2020. "A Fuzzy Energy Management Strategy for the Coordination of Electric Vehicle Charging in Low Voltage Distribution Grids," Energies, MDPI, vol. 13(14), pages 1-34, July.
    6. Gaizka Saldaña & Jose Ignacio San Martin & Inmaculada Zamora & Francisco Javier Asensio & Oier Oñederra, 2019. "Electric Vehicle into the Grid: Charging Methodologies Aimed at Providing Ancillary Services Considering Battery Degradation," Energies, MDPI, vol. 12(12), pages 1-37, June.
    7. Weige Zhang & Di Zhang & Biqiang Mu & Le Yi Wang & Yan Bao & Jiuchun Jiang & Hugo Morais, 2017. "Decentralized Electric Vehicle Charging Strategies for Reduced Load Variation and Guaranteed Charge Completion in Regional Distribution Grids," Energies, MDPI, vol. 10(2), pages 1-19, January.
    8. Mahmoudzadeh Andwari, Amin & Pesiridis, Apostolos & Rajoo, Srithar & Martinez-Botas, Ricardo & Esfahanian, Vahid, 2017. "A review of Battery Electric Vehicle technology and readiness levels," Renewable and Sustainable Energy Reviews, Elsevier, vol. 78(C), pages 414-430.
    9. Popović Vlado & Jereb Borut & Kilibarda Milorad & Andrejić Milan & Keshavarzsaleh Abolfazl & Dragan Dejan, 2018. "Electric Vehicles as Electricity Storages in Electric Power Systems," Logistics, Supply Chain, Sustainability and Global Challenges, Sciendo, vol. 9(2), pages 57-72, October.
    10. Sarah Ouédraogo & Ghjuvan Antone Faggianelli & Guillaume Pigelet & Jean Laurent Duchaud & Gilles Notton, 2020. "Application of Optimal Energy Management Strategies for a Building Powered by PV/Battery System in Corsica Island," Energies, MDPI, vol. 13(17), pages 1-20, September.
    11. Adil Amin & Wajahat Ullah Khan Tareen & Muhammad Usman & Haider Ali & Inam Bari & Ben Horan & Saad Mekhilef & Muhammad Asif & Saeed Ahmed & Anzar Mahmood, 2020. "A Review of Optimal Charging Strategy for Electric Vehicles under Dynamic Pricing Schemes in the Distribution Charging Network," Sustainability, MDPI, vol. 12(23), pages 1-28, December.
    12. Essam H. Houssein & Sanchari Deb & Diego Oliva & Hegazy Rezk & Hesham Alhumade & Mokhtar Said, 2021. "Performance of Gradient-Based Optimizer on Charging Station Placement Problem," Mathematics, MDPI, vol. 9(21), pages 1-16, November.
    13. Wu, Juai & Zhang, Mengying & Xu, Tianheng & Gu, Duan & Xie, Dongliang & Zhang, Tengfei & Hu, Honglin & Zhou, Ting, 2023. "A review of key technologies in relation to large-scale clusters of electric vehicles supporting a new power system," Renewable and Sustainable Energy Reviews, Elsevier, vol. 182(C).
    14. Saleh Aghajan-Eshkevari & Sasan Azad & Morteza Nazari-Heris & Mohammad Taghi Ameli & Somayeh Asadi, 2022. "Charging and Discharging of Electric Vehicles in Power Systems: An Updated and Detailed Review of Methods, Control Structures, Objectives, and Optimization Methodologies," Sustainability, MDPI, vol. 14(4), pages 1-31, February.
    15. Vamsi Krishna Reddy, Aala Kalananda & Venkata Lakshmi Narayana, Komanapalli, 2022. "Meta-heuristics optimization in electric vehicles -an extensive review," Renewable and Sustainable Energy Reviews, Elsevier, vol. 160(C).
    16. Adnane Houbbadi & Rochdi Trigui & Serge Pelissier & Eduardo Redondo-Iglesias & Tanguy Bouton, 2019. "Optimal Scheduling to Manage an Electric Bus Fleet Overnight Charging," Energies, MDPI, vol. 12(14), pages 1-17, July.
    17. Haider, Sajjad & Rizvi, Rida e Zahra & Walewski, John & Schegner, Peter, 2022. "Investigating peer-to-peer power transactions for reducing EV induced network congestion," Energy, Elsevier, vol. 254(PB).
    18. He, Qiang & Yang, Yang & Luo, Chang & Zhai, Jun & Luo, Ronghua & Fu, Chunyun, 2022. "Energy recovery strategy optimization of dual-motor drive electric vehicle based on braking safety and efficient recovery," Energy, Elsevier, vol. 248(C).
    19. Li, Yi & Liu, Kailong & Foley, Aoife M. & Zülke, Alana & Berecibar, Maitane & Nanini-Maury, Elise & Van Mierlo, Joeri & Hoster, Harry E., 2019. "Data-driven health estimation and lifetime prediction of lithium-ion batteries: A review," Renewable and Sustainable Energy Reviews, Elsevier, vol. 113(C), pages 1-1.
    20. Takyi-Aninakwa, Paul & Wang, Shunli & Zhang, Hongying & Yang, Xiao & Fernandez, Carlos, 2023. "A hybrid probabilistic correction model for the state of charge estimation of lithium-ion batteries considering dynamic currents and temperatures," Energy, Elsevier, vol. 273(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:appene:v:285:y:2021:i:c:s0306261920317566. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/wps/find/journaldescription.cws_home/405891/description#description .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.