IDEAS home Printed from https://ideas.repec.org/a/eee/appene/v371y2024ics0306261924010377.html
   My bibliography  Save this article

Privacy-preserving multi-level co-regulation of VPPs via hierarchical safe deep reinforcement learning

Author

Listed:
  • Xue, Lin
  • Zhang, Yao
  • Wang, Jianxue
  • Li, Haotian
  • Li, Fangshi

Abstract

Large amounts of distributed energy sources have brought challenges in terms of the safe and stable operation of the power grid. As a key technology between user-side energy resources and the distribution network (DN), how to realize the online coordinated scheduling of VPP and DN as well as the real-time response strategy of distributed equipment (DE) within VPP is the focus of this study. Thus, the hierarchical deep reinforcement learning (DRL) Hierarchical-TD3 algorithm is designed based on the unified modeling of adjustable space to achieve the real-time economic scheduling of VPPs. The upper layer DN considers the network security constraints and solves the economic scheduling model of VPPs based on the single-agent TD3 algorithm. Based on the scheduling instructions from the upper layer, the lower layer VPPs consider the requirements of privacy protection and control autonomy and realize real-time response of the DE within VPP via multi-agent MATD3 algorithm. Numerical results in the modified 33-nodes system show that the proposed Hierarchical-TD3 algorithm can achieve privacy protection and the coordinated scheduling of VPP and DE to reduce the operating cost. It differs from the optimal value by only 1.46% but can achieve online decision-making on the millisecond scale. Compared with the traditional centralized and decentralized DRL algorithms, the total cost is reduced by 10.15% and 5.52% respectively. Compared with the traditional soft-constraint method, there is no constraint violation during the training and testing phases. Finally, the actual 116-nodes testing system validates the scalability of the proposed Hierarchical-TD3 algorithm.

Suggested Citation

  • Xue, Lin & Zhang, Yao & Wang, Jianxue & Li, Haotian & Li, Fangshi, 2024. "Privacy-preserving multi-level co-regulation of VPPs via hierarchical safe deep reinforcement learning," Applied Energy, Elsevier, vol. 371(C).
    • Handle: RePEc:eee:appene:v:371:y:2024:i:c:s0306261924010377
    DOI: 10.1016/j.apenergy.2024.123654
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0306261924010377
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.apenergy.2024.123654?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Xue, Lin & Wang, Jianxue & Zhang, Yao & Yong, Weizhen & Qi, Jie & Li, Haotian, 2023. "Model-data-event based community integrated energy system low-carbon economic scheduling," Renewable and Sustainable Energy Reviews, Elsevier, vol. 182(C).
    2. Zamani, Ali Ghahgharaee & Zakariazadeh, Alireza & Jadid, Shahram, 2016. "Day-ahead resource scheduling of a renewable energy based virtual power plant," Applied Energy, Elsevier, vol. 169(C), pages 324-340.
    3. Li, Qiang & Wei, Fanchao & Zhou, Yongcheng & Li, Jiajia & Zhou, Guowen & Wang, Zhonghao & Liu, Jinfu & Yan, Peigang & Yu, Daren, 2023. "A scheduling framework for VPP considering multiple uncertainties and flexible resources," Energy, Elsevier, vol. 282(C).
    4. Xiang, Yue & Lu, Yu & Liu, Junyong, 2023. "Deep reinforcement learning based topology-aware voltage regulation of distribution networks with distributed energy storage," Applied Energy, Elsevier, vol. 332(C).
    5. Chang, Weiguang & Yang, Qiang, 2023. "Low carbon oriented collaborative energy management framework for multi-microgrid aggregated virtual power plant considering electricity trading," Applied Energy, Elsevier, vol. 351(C).
    6. Kofinas, P. & Dounis, A.I. & Vouros, G.A., 2018. "Fuzzy Q-Learning for multi-agent decentralized energy management in microgrids," Applied Energy, Elsevier, vol. 219(C), pages 53-67.
    7. Zhu, Ziqing & Wing Chan, Ka & Bu, Siqi & Zhou, Bin & Xia, Shiwei, 2021. "Real-Time interaction of active distribution network and virtual microgrids: Market paradigm and data-driven stakeholder behavior analysis," Applied Energy, Elsevier, vol. 297(C).
    8. Zhu, Xingxu & Hou, Xiangchen & Li, Junhui & Yan, Gangui & Li, Cuiping & Wang, Dongbo, 2023. "Distributed online prediction optimization algorithm for distributed energy resources considering the multi-periods optimal operation," Applied Energy, Elsevier, vol. 348(C).
    9. Guo, Guodong & Zhang, Mengfan & Gong, Yanfeng & Xu, Qianwen, 2023. "Safe multi-agent deep reinforcement learning for real-time decentralized control of inverter based renewable energy resources considering communication delay," Applied Energy, Elsevier, vol. 349(C).
    10. Guo, Chenyu & Wang, Xin & Zheng, Yihui & Zhang, Feng, 2022. "Real-time optimal energy management of microgrid with uncertainties based on deep reinforcement learning," Energy, Elsevier, vol. 238(PC).
    11. Kou, Peng & Liang, Deliang & Wang, Chen & Wu, Zihao & Gao, Lin, 2020. "Safe deep reinforcement learning-based constrained optimal control scheme for active distribution networks," Applied Energy, Elsevier, vol. 264(C).
    12. Fan, Shuai & Liu, Jiang & Wu, Qing & Cui, Mingjian & Zhou, Huan & He, Guangyu, 2020. "Optimal coordination of virtual power plant with photovoltaics and electric vehicles: A temporally coupled distributed online algorithm," Applied Energy, Elsevier, vol. 277(C).
    13. Wang, Xuejie & zhao, Huiru & Lu, Hao & Zhang, Yuanyuan & Wang, Yuwei & Wang, Jingbo, 2022. "Decentralized coordinated operation model of VPP and P2H systems based on stochastic-bargaining game considering multiple uncertainties and carbon cost," Applied Energy, Elsevier, vol. 312(C).
    14. Song, Jiancai & Bian, Tianxiang & Xue, Guixiang & Wang, Hanyu & Shen, Xingliang & Wu, Xiangdong, 2023. "Short-term forecasting model for residential indoor temperature in DHS based on sequence generative adversarial network," Applied Energy, Elsevier, vol. 348(C).
    15. Lu, Renzhi & Hong, Seung Ho, 2019. "Incentive-based demand response for smart grid with reinforcement learning and deep neural network," Applied Energy, Elsevier, vol. 236(C), pages 937-949.
    16. Liu, Chunming & Wang, Chunling & Yin, Yujun & Yang, Peihong & Jiang, Hui, 2022. "Bi-level dispatch and control strategy based on model predictive control for community integrated energy system considering dynamic response performance," Applied Energy, Elsevier, vol. 310(C).
    17. Huang, Ruchen & He, Hongwen & Zhao, Xuyang & Wang, Yunlong & Li, Menglin, 2022. "Battery health-aware and naturalistic data-driven energy management for hybrid electric bus based on TD3 deep reinforcement learning algorithm," Applied Energy, Elsevier, vol. 321(C).
    18. Zhou, Huan & Fan, Shuai & Wu, Qing & Dong, Lianxin & Li, Zuyi & He, Guangyu, 2021. "Stimulus-response control strategy based on autonomous decentralized system theory for exploitation of flexibility by virtual power plant," Applied Energy, Elsevier, vol. 285(C).
    19. Pinciroli, Luca & Baraldi, Piero & Compare, Michele & Zio, Enrico, 2023. "Optimal operation and maintenance of energy storage systems in grid-connected microgrids by deep reinforcement learning," Applied Energy, Elsevier, vol. 352(C).
    20. Wang, Jiewei & Wei, Ziqing & Zhu, Yikang & Zheng, Chunyuan & Li, Bin & Zhai, Xiaoqiang, 2023. "Demand response via optimal pre-cooling combined with temperature reset strategy for air conditioning system: A case study of office building," Energy, Elsevier, vol. 282(C).
    21. Park, Sung-Won & Son, Sung-Yong, 2020. "Interaction-based virtual power plant operation methodology for distribution system operator’s voltage management," Applied Energy, Elsevier, vol. 271(C).
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Omar Al-Ani & Sanjoy Das, 2022. "Reinforcement Learning: Theory and Applications in HEMS," Energies, MDPI, vol. 15(17), pages 1-37, September.
    2. Ju, Liwei & Yin, Zhe & Lu, Xiaolong & Yang, Shenbo & Li, Peng & Rao, Rao & Tan, Zhongfu, 2022. "A Tri-dimensional Equilibrium-based stochastic optimal dispatching model for a novel virtual power plant incorporating carbon Capture, Power-to-Gas and electric vehicle aggregator," Applied Energy, Elsevier, vol. 324(C).
    3. Yi Kuang & Xiuli Wang & Hongyang Zhao & Yijun Huang & Xianlong Chen & Xifan Wang, 2020. "Agent-Based Energy Sharing Mechanism Using Deep Deterministic Policy Gradient Algorithm," Energies, MDPI, vol. 13(19), pages 1-20, September.
    4. Cao, Jinye & Yang, Dechang & Dehghanian, Payman, 2024. "Cooperative operation for multiple virtual power plants considering energy-carbon trading: A Nash bargaining model," Energy, Elsevier, vol. 307(C).
    5. Hou, Guolian & Huang, Ting & Zheng, Fumeng & Huang, Congzhi, 2024. "A hierarchical reinforcement learning GPC for flexible operation of ultra-supercritical unit considering economy," Energy, Elsevier, vol. 289(C).
    6. Francesco Gulotta & Edoardo Daccò & Alessandro Bosisio & Davide Falabretti, 2023. "Opening of Ancillary Service Markets to Distributed Energy Resources: A Review," Energies, MDPI, vol. 16(6), pages 1-25, March.
    7. Wang, Yi & Qiu, Dawei & Sun, Mingyang & Strbac, Goran & Gao, Zhiwei, 2023. "Secure energy management of multi-energy microgrid: A physical-informed safe reinforcement learning approach," Applied Energy, Elsevier, vol. 335(C).
    8. Zhao, Yincheng & Zhang, Guozhou & Hu, Weihao & Huang, Qi & Chen, Zhe & Blaabjerg, Frede, 2023. "Meta-learning based voltage control strategy for emergency faults of active distribution networks," Applied Energy, Elsevier, vol. 349(C).
    9. Pinto, Giuseppe & Piscitelli, Marco Savino & Vázquez-Canteli, José Ramón & Nagy, Zoltán & Capozzoli, Alfonso, 2021. "Coordinated energy management for a cluster of buildings through deep reinforcement learning," Energy, Elsevier, vol. 229(C).
    10. Ren, Junzhi & Zeng, Yuan & Qin, Chao & Li, Bao & Wang, Ziqiang & Yuan, Quan & Zhai, Hefeng & Li, Peng, 2024. "Characterization and application of flexible operation region of virtual power plant," Applied Energy, Elsevier, vol. 371(C).
    11. Zhu, Ziqing & Hu, Ze & Chan, Ka Wing & Bu, Siqi & Zhou, Bin & Xia, Shiwei, 2023. "Reinforcement learning in deregulated energy market: A comprehensive review," Applied Energy, Elsevier, vol. 329(C).
    12. Dong, Lianxin & Fan, Shuai & Wang, Zhihua & Xiao, Jucheng & Zhou, Huan & Li, Zuyi & He, Guangyu, 2021. "An adaptive decentralized economic dispatch method for virtual power plant," Applied Energy, Elsevier, vol. 300(C).
    13. Pinto, Giuseppe & Deltetto, Davide & Capozzoli, Alfonso, 2021. "Data-driven district energy management with surrogate models and deep reinforcement learning," Applied Energy, Elsevier, vol. 304(C).
    14. Zhu, Dafeng & Yang, Bo & Liu, Yuxiang & Wang, Zhaojian & Ma, Kai & Guan, Xinping, 2022. "Energy management based on multi-agent deep reinforcement learning for a multi-energy industrial park," Applied Energy, Elsevier, vol. 311(C).
    15. Dong, Lianxin & Wu, Qing & Hong, Juhua & Wang, Zhihua & Fan, Shuai & He, Guangyu, 2023. "An adaptive decentralized regulation strategy for the cluster with massive inverter air conditionings," Applied Energy, Elsevier, vol. 330(PA).
    16. Jiang, Yuzheng & Dong, Jun & Huang, Hexiang, 2024. "Optimal bidding strategy for the price-maker virtual power plant in the day-ahead market based on multi-agent twin delayed deep deterministic policy gradient algorithm," Energy, Elsevier, vol. 306(C).
    17. Kong, Xiangyu & Kong, Deqian & Yao, Jingtao & Bai, Linquan & Xiao, Jie, 2020. "Online pricing of demand response based on long short-term memory and reinforcement learning," Applied Energy, Elsevier, vol. 271(C).
    18. Chen, Yongdong & Liu, Youbo & Zhao, Junbo & Qiu, Gao & Yin, Hang & Li, Zhengbo, 2023. "Physical-assisted multi-agent graph reinforcement learning enabled fast voltage regulation for PV-rich active distribution network," Applied Energy, Elsevier, vol. 351(C).
    19. Wafa Nafkha-Tayari & Seifeddine Ben Elghali & Ehsan Heydarian-Forushani & Mohamed Benbouzid, 2022. "Virtual Power Plants Optimization Issue: A Comprehensive Review on Methods, Solutions, and Prospects," Energies, MDPI, vol. 15(10), pages 1-20, May.
    20. Guixing Yang & Haoran Liu & Weiqing Wang & Junru Chen & Shunbo Lei, 2023. "Distributed Optimal Coordination of a Virtual Power Plant with Residential Regenerative Electric Heating Systems," Energies, MDPI, vol. 16(11), pages 1-15, May.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:appene:v:371:y:2024:i:c:s0306261924010377. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/wps/find/journaldescription.cws_home/405891/description#description .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.