IDEAS home Printed from https://ideas.repec.org/a/eee/appene/v348y2023ics0306261923008541.html
   My bibliography  Save this article

A safe reinforcement learning-based charging strategy for electric vehicles in residential microgrid

Author

Listed:
  • Zhang, Shulei
  • Jia, Runda
  • Pan, Hengxin
  • Cao, Yankai

Abstract

With the growing popularity of electric vehicles (EVs), it is a new challenge for the residential microgrid system to conduct charging scheduling to meet the charging demands of EVs while maximizing its profit. In this work, a safe reinforcement learning (RL)-based charging scheduling strategy is proposed to meet this challenge. We construct a complete microgrid system equipped with a large charging station and consider different types of EVs, as well as the vehicle-to-grid (V2G) mode and nonlinear charging characteristics of EVs. Subsequently, the charging scheduling problem is formulated as a constrained Markov decision process (CMDP) due to the various limitations of power and demands. To effectively capture the uncertainties of the supply side and demand side of the microgrid, a model-free RL framework is employed. However, the curse of dimensionality of the action space is inevitable as EVs increase. To solve this problem, a charging and discharging strategy based on a general ladder electricity pricing scheme is designed. Different EVs are divided into different sets according to their states under this strategy, and the agent gives control signals of different sets instead of controlling each EV individually, which effectively reduces the dimension of the action space. Subsequently, a constrained soft actor-critic (CSAC) algorithm is designed to solve the established CMDP, and a safety filter is introduced to ensure safety. In the end, a numerical case is conducted to verify the effectiveness of the proposed method.

Suggested Citation

  • Zhang, Shulei & Jia, Runda & Pan, Hengxin & Cao, Yankai, 2023. "A safe reinforcement learning-based charging strategy for electric vehicles in residential microgrid," Applied Energy, Elsevier, vol. 348(C).
  • Handle: RePEc:eee:appene:v:348:y:2023:i:c:s0306261923008541
    DOI: 10.1016/j.apenergy.2023.121490
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0306261923008541
    Download Restriction: Full text for ScienceDirect subscribers only

    File URL: https://libkey.io/10.1016/j.apenergy.2023.121490?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Siobhan Powell & Gustavo Vianna Cezar & Liang Min & Inês M. L. Azevedo & Ram Rajagopal, 2022. "Charging infrastructure access and operation to reduce the grid impacts of deep electric vehicle adoption," Nature Energy, Nature, vol. 7(10), pages 932-945, October.
    2. Yuxin Wen & Peixiao Fan & Jia Hu & Song Ke & Fuzhang Wu & Xu Zhu, 2022. "An Optimal Scheduling Strategy of a Microgrid with V2G Based on Deep Q-Learning," Sustainability, MDPI, vol. 14(16), pages 1-18, August.
    3. Lyu, Yizheng & Gao, Hanbo & Yan, Kun & Liu, Yingjie & Tian, Jinping & Chen, Lyujun & Wan, Mei, 2022. "Carbon peaking strategies for industrial parks: Model development and applications in China," Applied Energy, Elsevier, vol. 322(C).
    4. Dong, Xiangxiang & Wu, Jiang & Xu, Zhanbo & Liu, Kun & Guan, Xiaohong, 2022. "Optimal coordination of hydrogen-based integrated energy systems with combination of hydrogen and water storage," Applied Energy, Elsevier, vol. 308(C).
    5. Athanasios Paraskevas & Dimitrios Aletras & Antonios Chrysopoulos & Antonios Marinopoulos & Dimitrios I. Doukas, 2022. "Optimal Management for EV Charging Stations: A Win–Win Strategy for Different Stakeholders Using Constrained Deep Q-Learning," Energies, MDPI, vol. 15(7), pages 1-24, March.
    6. Lai, Chun Sing & Chen, Dashen & Zhang, Jinning & Zhang, Xin & Xu, Xu & Taylor, Gareth A. & Lai, Loi Lei, 2022. "Profit maximization for large-scale energy storage systems to enable fast EV charging infrastructure in distribution networks," Energy, Elsevier, vol. 259(C).
    7. Lee, Sangyoon & Choi, Dae-Hyun, 2021. "Dynamic pricing and energy management for profit maximization in multiple smart electric vehicle charging stations: A privacy-preserving deep reinforcement learning approach," Applied Energy, Elsevier, vol. 304(C).
    8. Zhao, Shihao & Li, Kang & Yang, Zhile & Xu, Xinzhi & Zhang, Ning, 2022. "A new power system active rescheduling method considering the dispatchable plug-in electric vehicles and intermittent renewable energies," Applied Energy, Elsevier, vol. 314(C).
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Paesschesoone, Siebe & Kayedpour, Nezmin & Manna, Carlo & Crevecoeur, Guillaume, 2024. "Reinforcement learning for an enhanced energy flexibility controller incorporating predictive safety filter and adaptive policy updates," Applied Energy, Elsevier, vol. 368(C).
    2. Wang, Can & Zhang, Jiaheng & Wang, Aoqi & Wang, Zhen & Yang, Nan & Zhao, Zhuoli & Lai, Chun Sing & Lai, Loi Lei, 2024. "Prioritized sum-tree experience replay TD3 DRL-based online energy management of a residential microgrid," Applied Energy, Elsevier, vol. 368(C).
    3. Zhao, Zhonghao & Lee, Carman K.M. & Yan, Xiaoyuan & Wang, Haonan, 2024. "Reinforcement learning for electric vehicle charging scheduling: A systematic review," Transportation Research Part E: Logistics and Transportation Review, Elsevier, vol. 190(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Paudel, Diwas & Das, Tapas K., 2023. "A deep reinforcement learning approach for power management of battery-assisted fast-charging EV hubs participating in day-ahead and real-time electricity markets," Energy, Elsevier, vol. 283(C).
    2. Harasis, Salman & Khan, Irfan & Massoud, Ahmed, 2024. "Enabling large-scale integration of electric bus fleets in harsh environments: Possibilities, potentials, and challenges," Energy, Elsevier, vol. 300(C).
    3. Ma, Huan & Sun, Qinghan & Chen, Qun & Zhao, Tian & He, Kelun, 2023. "Exergy-based flexibility cost indicator and spatio-temporal coordination principle of distributed multi-energy systems," Energy, Elsevier, vol. 267(C).
    4. Yin, Rumeng & He, Jiang, 2023. "Design of a photovoltaic electric bike battery-sharing system in public transit stations," Applied Energy, Elsevier, vol. 332(C).
    5. Zhao, Yang & Jiang, Ziyue & Chen, Xinyu & Liu, Peng & Peng, Tianduo & Shu, Zhan, 2023. "Toward environmental sustainability: data-driven analysis of energy use patterns and load profiles for urban electric vehicle fleets," Energy, Elsevier, vol. 285(C).
    6. Alya AlHammadi & Nasser Al-Saif & Ameena Saad Al-Sumaiti & Mousa Marzband & Tareefa Alsumaiti & Ehsan Heydarian-Forushani, 2022. "Techno-Economic Analysis of Hybrid Renewable Energy Systems Designed for Electric Vehicle Charging: A Case Study from the United Arab Emirates," Energies, MDPI, vol. 15(18), pages 1-20, September.
    7. Xingyun Yan & Lingyu Wang & Mingzhu Fang & Jie Hu, 2022. "How Can Industrial Parks Achieve Carbon Neutrality? Literature Review and Research Prospect Based on the CiteSpace Knowledge Map," Sustainability, MDPI, vol. 15(1), pages 1-29, December.
    8. Zhao, Zhonghao & Lee, Carman K.M. & Huo, Jiage, 2023. "EV charging station deployment on coupled transportation and power distribution networks via reinforcement learning," Energy, Elsevier, vol. 267(C).
    9. Powell, Siobhan & Martin, Sonia & Rajagopal, Ram & Azevedo, Inês M.L. & de Chalendar, Jacques, 2024. "Future-proof rates for controlled electric vehicle charging: Comparing multi-year impacts of different emission factor signals," Energy Policy, Elsevier, vol. 190(C).
    10. Saeed Alyami, 2024. "Ensuring Sustainable Grid Stability through Effective EV Charging Management: A Time and Energy-Based Approach," Sustainability, MDPI, vol. 16(14), pages 1-15, July.
    11. Thi Ngoc Nguyen & Felix Muesgens, 2024. "Fuel tax loss in a world of electric mobility: A window of opportunity for congestion pricing," Papers 2409.20033, arXiv.org.
    12. Sohani, Ali & Cornaro, Cristina & Shahverdian, Mohammad Hassan & Moser, David & Pierro, Marco & Olabi, Abdul Ghani & Karimi, Nader & Nižetić, Sandro & Li, Larry K.B. & Doranehgard, Mohammad Hossein, 2023. "Techno-economic evaluation of a hybrid photovoltaic system with hot/cold water storage for poly-generation in a residential building," Applied Energy, Elsevier, vol. 331(C).
    13. Tian, Xueyu & You, Fengqi, 2024. "Broaden sustainable design and optimization of decarbonized campus Energy systems with scope 3 emissions accounting and social ramification analysis," Applied Energy, Elsevier, vol. 373(C).
    14. Cui, Li & Wang, Qingyuan & Qu, Hongquan & Wang, Mingshen & Wu, Yile & Ge, Le, 2023. "Dynamic pricing for fast charging stations with deep reinforcement learning," Applied Energy, Elsevier, vol. 346(C).
    15. Ibrahim Alsaidan & Mohd Bilal & Muhannad Alaraj & Mohammad Rizwan & Fahad M. Almasoudi, 2023. "A Novel EA-Based Techno–Economic Analysis of Charging System for Electric Vehicles: A Case Study of Qassim Region, Saudi Arabia," Mathematics, MDPI, vol. 11(9), pages 1-31, April.
    16. George-Williams, H. & Wade, N. & Carpenter, R.N., 2022. "A probabilistic framework for the techno-economic assessment of smart energy hubs for electric vehicle charging," Renewable and Sustainable Energy Reviews, Elsevier, vol. 162(C).
    17. Sauter, A.J. & Lara, José Daniel & Turk, Jennifer & Milford, Jana & Hodge, Bri-Mathias, 2024. "Power system operational impacts of electric vehicle dynamic wireless charging," Applied Energy, Elsevier, vol. 364(C).
    18. Md Sahabul Hossain & Craig Rodine & Eirini Eleni Tsiropoulou, 2024. "A Blockchain and PKI-Based Secure Vehicle-to-Vehicle Energy-Trading Protocol," Energies, MDPI, vol. 17(17), pages 1-52, August.
    19. Fengyuan Yan & Xiaolong Han & Qianwei Cheng & Yamin Yan & Qi Liao & Yongtu Liang, 2022. "Scenario-Based Comparative Analysis for Coupling Electricity and Hydrogen Storage in Clean Oilfield Energy Supply System," Energies, MDPI, vol. 15(6), pages 1-28, March.
    20. Truong, Van Binh & Le, Long Bao, 2024. "Electric vehicle charging design: The factored action based reinforcement learning approach," Applied Energy, Elsevier, vol. 359(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:appene:v:348:y:2023:i:c:s0306261923008541. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/wps/find/journaldescription.cws_home/405891/description#description .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.