IDEAS home Printed from https://ideas.repec.org/a/eee/appene/v335y2023ics0306261922018864.html
   My bibliography  Save this article

Deep reinforcement learning-based two-timescale Volt-VAR control with degradation-aware smart inverters in power distribution systems

Author

Listed:
  • Kabir, Farzana
  • Yu, Nanpeng
  • Gao, Yuanqi
  • Wang, Wenyu

Abstract

Higher penetration of intermittent solar photovoltaic (PV) systems in the distribution grid results in frequent voltage fluctuations. The conventional voltage regulating devices operating on a slow-timescale need to be supplemented with the fast-operating smart inverters with adjustable reactive power setpoints. Complete and accurate information about distribution network topology and line parameters is necessary for conventional model-based Volt-VAR control (VVC) methods. However, such information is often unavailable. To tackle these challenges, a reinforcement learning-based two-timescale VVC algorithm is proposed in this paper that jointly controls the conventional voltage regulating devices at the slow-timescale and the smart inverters at the fast-timescale. Our proposed VVC algorithm simultaneously minimizes voltage violation costs and system operation costs in a model-free manner utilizing historical operational data. Two hierarchically organized agents are set up for the slow-timescale and fast-timescale problems, which are coupled through a communication scheme. The two sets of control policies are learned concurrently by a deep deterministic policy gradient and multi-agent soft actor-critic algorithm respectively. Comprehensive numerical studies performed with the IEEE 123-bus distribution test feeder show that the proposed framework can identify near optimal control actions of voltage regulating devices and smart inverters in real-time operations.

Suggested Citation

  • Kabir, Farzana & Yu, Nanpeng & Gao, Yuanqi & Wang, Wenyu, 2023. "Deep reinforcement learning-based two-timescale Volt-VAR control with degradation-aware smart inverters in power distribution systems," Applied Energy, Elsevier, vol. 335(C).
  • Handle: RePEc:eee:appene:v:335:y:2023:i:c:s0306261922018864
    DOI: 10.1016/j.apenergy.2022.120629
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0306261922018864
    Download Restriction: Full text for ScienceDirect subscribers only

    File URL: https://libkey.io/10.1016/j.apenergy.2022.120629?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Jeon, Soi & Choi, Dae-Hyun, 2022. "Joint optimization of Volt/VAR control and mobile energy storage system scheduling in active power distribution networks under PV prediction uncertainty," Applied Energy, Elsevier, vol. 310(C).
    2. Zhang, Wenjie & Gandhi, Oktoviano & Quan, Hao & Rodríguez-Gallegos, Carlos D. & Srinivasan, Dipti, 2018. "A multi-agent based integrated volt-var optimization engine for fast vehicle-to-grid reactive power dispatch and electric vehicle coordination," Applied Energy, Elsevier, vol. 229(C), pages 96-110.
    3. Gao, Yuanqi & Yu, Nanpeng, 2022. "Model-augmented safe reinforcement learning for Volt-VAR control in power distribution networks," Applied Energy, Elsevier, vol. 313(C).
    4. Lee, Xian Yeow & Sarkar, Soumik & Wang, Yubo, 2022. "A graph policy network approach for Volt-Var Control in power distribution systems," Applied Energy, Elsevier, vol. 323(C).
    5. Mak, Davye & Choi, Dae-Hyun, 2020. "Optimization framework for coordinated operation of home energy management system and Volt-VAR optimization in unbalanced active distribution networks considering uncertainties," Applied Energy, Elsevier, vol. 276(C).
    6. Haider, Rabab & Annaswamy, Anuradha M., 2022. "A hybrid architecture for volt-var control in active distribution grids," Applied Energy, Elsevier, vol. 312(C).
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Boscaino, Valeria & Ditta, Vito & Marsala, Giuseppe & Panzavecchia, Nicola & Tinè, Giovanni & Cosentino, Valentina & Cataliotti, Antonio & Di Cara, Dario, 2024. "Grid-connected photovoltaic inverters: Grid codes, topologies and control techniques," Renewable and Sustainable Energy Reviews, Elsevier, vol. 189(PA).
    2. Zhao, Yincheng & Zhang, Guozhou & Hu, Weihao & Huang, Qi & Chen, Zhe & Blaabjerg, Frede, 2023. "Meta-learning based voltage control strategy for emergency faults of active distribution networks," Applied Energy, Elsevier, vol. 349(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Jeon, Soi & Choi, Dae-Hyun, 2022. "Joint optimization of Volt/VAR control and mobile energy storage system scheduling in active power distribution networks under PV prediction uncertainty," Applied Energy, Elsevier, vol. 310(C).
    2. Guo, Guodong & Zhang, Mengfan & Gong, Yanfeng & Xu, Qianwen, 2023. "Safe multi-agent deep reinforcement learning for real-time decentralized control of inverter based renewable energy resources considering communication delay," Applied Energy, Elsevier, vol. 349(C).
    3. Edward J. Smith & Duane A. Robinson & Sean Elphick, 2024. "DER Control and Management Strategies for Distribution Networks: A Review of Current Practices and Future Directions," Energies, MDPI, vol. 17(11), pages 1-40, May.
    4. Mak, Davye & Choeum, Daranith & Choi, Dae-Hyun, 2020. "Sensitivity analysis of volt-VAR optimization to data changes in distribution networks with distributed energy resources," Applied Energy, Elsevier, vol. 261(C).
    5. Jeddi, Babak & Mishra, Yateendra & Ledwich, Gerard, 2021. "Distributed load scheduling in residential neighborhoods for coordinated operation of multiple home energy management systems," Applied Energy, Elsevier, vol. 300(C).
    6. Zhou, Yu & Li, Zhengshuo & Wang, Guangrui, 2021. "Study on leveraging wind farms' robust reactive power range for uncertain power system reactive power optimization," Applied Energy, Elsevier, vol. 298(C).
    7. Wang, Yi & Qiu, Dawei & Sun, Mingyang & Strbac, Goran & Gao, Zhiwei, 2023. "Secure energy management of multi-energy microgrid: A physical-informed safe reinforcement learning approach," Applied Energy, Elsevier, vol. 335(C).
    8. Zhao, Yincheng & Zhang, Guozhou & Hu, Weihao & Huang, Qi & Chen, Zhe & Blaabjerg, Frede, 2023. "Meta-learning based voltage control strategy for emergency faults of active distribution networks," Applied Energy, Elsevier, vol. 349(C).
    9. Zhuoxin Lu & Xiaoyuan Xu & Zheng Yan & Dong Han & Shiwei Xia, 2024. "Mobile Energy-Storage Technology in Power Grid: A Review of Models and Applications," Sustainability, MDPI, vol. 16(16), pages 1-19, August.
    10. Das, H.S. & Rahman, M.M. & Li, S. & Tan, C.W., 2020. "Electric vehicles standards, charging infrastructure, and impact on grid integration: A technological review," Renewable and Sustainable Energy Reviews, Elsevier, vol. 120(C).
    11. Zhang, Ziqi & Li, Peng & Ji, Haoran & Zhao, Jinli & Xi, Wei & Wu, Jianzhong & Wang, Chengshan, 2024. "Combined central-local voltage control of inverter-based DG in active distribution networks11The short version of the paper was presented at CUE2023. This paper is a substantial extension of the short," Applied Energy, Elsevier, vol. 372(C).
    12. Jude Suchithra & Amin Rajabi & Duane A. Robinson, 2024. "Enhancing PV Hosting Capacity of Electricity Distribution Networks Using Deep Reinforcement Learning-Based Coordinated Voltage Control," Energies, MDPI, vol. 17(20), pages 1-27, October.
    13. Vijayan, Vineeth & Mohapatra, Abheejeet & Singh, S.N., 2021. "Demand Response with Volt/Var Optimization for unbalanced active distribution systems," Applied Energy, Elsevier, vol. 300(C).
    14. Buonomano, Annamaria, 2020. "Building to Vehicle to Building concept: A comprehensive parametric and sensitivity analysis for decision making aims," Applied Energy, Elsevier, vol. 261(C).
    15. Chen, Yongdong & Liu, Youbo & Zhao, Junbo & Qiu, Gao & Yin, Hang & Li, Zhengbo, 2023. "Physical-assisted multi-agent graph reinforcement learning enabled fast voltage regulation for PV-rich active distribution network," Applied Energy, Elsevier, vol. 351(C).
    16. Buonomano, A. & Calise, F. & Cappiello, F.L. & Palombo, A. & Vicidomini, M., 2019. "Dynamic analysis of the integration of electric vehicles in efficient buildings fed by renewables," Applied Energy, Elsevier, vol. 245(C), pages 31-50.
    17. Alex Chamba & Carlos Barrera-Singaña & Hugo Arcos, 2023. "Optimal Reactive Power Dispatch in Electric Transmission Systems Using the Multi-Agent Model with Volt-VAR Control," Energies, MDPI, vol. 16(13), pages 1-25, June.
    18. Gandhi, Oktoviano & Zhang, Wenjie & Rodríguez-Gallegos, Carlos D. & Verbois, Hadrien & Sun, Hongbin & Reindl, Thomas & Srinivasan, Dipti, 2020. "Local reactive power dispatch optimisation minimising global objectives," Applied Energy, Elsevier, vol. 262(C).
    19. Prabawa, Panggah & Choi, Dae-Hyun, 2024. "Safe deep reinforcement learning-assisted two-stage energy management for active power distribution networks with hydrogen fueling stations," Applied Energy, Elsevier, vol. 375(C).
    20. Barone, Giovanni & Buonomano, Annamaria & Forzano, Cesare & Giuzio, Giovanni Francesco & Palombo, Adolfo, 2020. "Increasing self-consumption of renewable energy through the Building to Vehicle to Building approach applied to multiple users connected in a virtual micro-grid," Renewable Energy, Elsevier, vol. 159(C), pages 1165-1176.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:appene:v:335:y:2023:i:c:s0306261922018864. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/wps/find/journaldescription.cws_home/405891/description#description .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.