IDEAS home Printed from https://ideas.repec.org/a/gam/jeners/v18y2025i5p1020-d1595435.html
   My bibliography  Save this article

Enhanced Dynamic Expansion Planning Model Incorporating Q-Learning and Distributionally Robust Optimization for Resilient and Cost-Efficient Distribution Networks

Author

Listed:
  • Gang Lu

    (State Grid Energy Research Institute Co., Ltd., Beijing 102209, China)

  • Bo Yuan

    (State Grid Energy Research Institute Co., Ltd., Beijing 102209, China)

  • Baorui Nie

    (School of Economics and Management, North China Electric Power University, Beijing 102206, China)

  • Peng Xia

    (State Grid Energy Research Institute Co., Ltd., Beijing 102209, China)

  • Cong Wu

    (State Grid Energy Research Institute Co., Ltd., Beijing 102209, China)

  • Guangzeng Sun

    (State Grid Energy Research Institute Co., Ltd., Beijing 102209, China)

Abstract

The increasing integration of renewable energy-based distributed generation (DG) in modern distribution networks is essential for reducing reliance on fossil fuels. However, the unpredictability and intermittency of renewable sources such as wind and photovoltaic (PV) systems introduce significant challenges for distribution network planning. To address these challenges, this paper proposes a Q-learning-based Distributionally Robust Optimization (DRO) model for expansion planning of distribution networks and generation units. The proposed model incorporates energy storage systems (ESSs), renewable DG, substations, and distribution lines while considering uncertainties such as renewable generation variability, load fluctuations, and system contingencies. Through a dynamic decision-making process using Q-learning, the model adapts to changing network conditions to minimize the total system cost while maintaining reliability. The Latin Hypercube Sampling (LHS) method is employed to generate multi-scenario data, and piecewise linearization is used to reduce the computational complexity of the AC power flow equations. Numerical results demonstrate that the model significantly improves system reliability and economic efficiency under multiple uncertainty scenarios. The results also highlight the crucial role of the ESS in mitigating the variability of renewable energy and reducing the expected energy not supplied (EENS).

Suggested Citation

  • Gang Lu & Bo Yuan & Baorui Nie & Peng Xia & Cong Wu & Guangzeng Sun, 2025. "Enhanced Dynamic Expansion Planning Model Incorporating Q-Learning and Distributionally Robust Optimization for Resilient and Cost-Efficient Distribution Networks," Energies, MDPI, vol. 18(5), pages 1-25, February.
    • Handle: RePEc:gam:jeners:v:18:y:2025:i:5:p:1020-:d:1595435
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/1996-1073/18/5/1020/pdf
    Download Restriction: no
    File URL: https://www.mdpi.com/1996-1073/18/5/1020/
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Canizes, Bruno & Soares, João & Lezama, Fernando & Silva, Cátia & Vale, Zita & Corchado, Juan M., 2019. "Optimal expansion planning considering storage investment and seasonal effect of demand and renewable generation," Renewable Energy, Elsevier, vol. 138(C), pages 937-954.
    2. Li, Rui & Wang, Wei & Wu, Xuezhi & Tang, Fen & Chen, Zhe, 2019. "Cooperative planning model of renewable energy sources and energy storage units in active distribution systems: A bi-level model and Pareto analysis," Energy, Elsevier, vol. 168(C), pages 30-42.
    3. Yang, Zhixue & Ren, Zhouyang & Li, Hui & Sun, Zhiyuan & Feng, Jianbing & Xia, Weiyi, 2024. "A multi-stage stochastic dispatching method for electricity‑hydrogen integrated energy systems driven by model and data," Applied Energy, Elsevier, vol. 371(C).
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Zhou, Siyu & Han, Yang & Chen, Shuheng & Yang, Ping & Mahmoud, Karar & Darwish, Mohamed M.F. & Matti, Lehtonen & Zalhaf, Amr S., 2023. "A multiple uncertainty-based Bi-level expansion planning paradigm for distribution networks complying with energy storage system functionalities," Energy, Elsevier, vol. 275(C).
    2. Wang, Bangyan & Wang, Xiuli & Zhu, Zongyao & Wu, Xiong, 2023. "Siting and sizing of energy storage for renewable generation utilization with multi-stage dispatch under uncertainty: A tri-level model and decomposition approach," Applied Energy, Elsevier, vol. 344(C).
    3. Li, Jinghua & Lu, Bo & Wang, Zhibang & Zhu, Mengshu, 2021. "Bi-level optimal planning model for energy storage systems in a virtual power plant," Renewable Energy, Elsevier, vol. 165(P2), pages 77-95.
    4. Ghahramani, Mehrdad & Nazari-Heris, Morteza & Zare, Kazem & Mohammadi-Ivatloo, Behnam, 2019. "Energy and reserve management of a smart distribution system by incorporating responsive-loads /battery/wind turbines considering uncertain parameters," Energy, Elsevier, vol. 183(C), pages 205-219.
    5. Mohammed, Nooriya A. & Al-Bazi, Ammar, 2021. "Management of renewable energy production and distribution planning using agent-based modelling," Renewable Energy, Elsevier, vol. 164(C), pages 509-520.
    6. Mateusz Andrychowicz, 2021. "RES and ES Integration in Combination with Distribution Grid Development Using MILP," Energies, MDPI, vol. 14(2), pages 1-19, January.
    7. Wang, Ying & Chen, Ying & Xiao, Xianyong & Chen, Yunzhu & Li, Qilin, 2025. "Bi-level planning-operation model of PV considering reactive power capability," Applied Energy, Elsevier, vol. 378(PA).
    8. Huang, Congzhi & Sheng, Xinxin, 2020. "Data-driven model identification of boiler-turbine coupled process in 1000 MW ultra-supercritical unit by improved bird swarm algorithm," Energy, Elsevier, vol. 205(C).
    9. Yamchi, Hamid Bakhshi & Safari, Amin & Guerrero, Josep M., 2021. "A multi-objective mixed integer linear programming model for integrated electricity-gas network expansion planning considering the impact of photovoltaic generation," Energy, Elsevier, vol. 222(C).
    10. Gu, Chenjia & Wang, Jianxue & Zhang, Yao & Li, Qingtao & Chen, Yang, 2022. "Optimal energy storage planning for stacked benefits in power distribution network," Renewable Energy, Elsevier, vol. 195(C), pages 366-380.
    11. Xie, Shiwei & Hu, Zhijian & Wang, Jueying, 2020. "Two-stage robust optimization for expansion planning of active distribution systems coupled with urban transportation networks," Applied Energy, Elsevier, vol. 261(C).
    12. Li, Xiangrong & Zhu, Shaoying & Yüksel, Serhat & Dinçer, Hasan & Ubay, Gözde Gülseven, 2020. "Kano-based mapping of innovation strategies for renewable energy alternatives using hybrid interval type-2 fuzzy decision-making approach," Energy, Elsevier, vol. 211(C).
    13. Xia, Weiyi & Ren, Zhouyang & Li, Hui & Pan, Zhen, 2024. "A data-driven probabilistic evaluation method of hydrogen fuel cell vehicles hosting capacity for integrated hydrogen-electricity network," Applied Energy, Elsevier, vol. 376(PB).
    14. Wang, Weijun & Li, Chen & He, Yan & Bai, Haining & Jia, Kaiqing & Kong, Zhe, 2024. "Enhancement of household photovoltaic consumption potential in village microgrid considering electric vehicles scheduling and energy storage system configuration," Energy, Elsevier, vol. 311(C).
    15. Hu, Xiao & Zhang, Heng & Chen, Dongwen & Li, Yong & Wang, Li & Zhang, Feng & Cheng, Haozhong, 2020. "Multi-objective planning for integrated energy systems considering both exergy efficiency and economy," Energy, Elsevier, vol. 197(C).
    16. Gao, Yuan & Tahir, Mustafa & Siano, Pierluigi & Bi, Yue & Hu, Sile & Yang, Jiaqiang, 2025. "Optimization of renewable energy-based integrated energy systems: A three-stage stochastic robust model," Applied Energy, Elsevier, vol. 377(PD).
    17. Xuehan Zhang & Yongju Son & Sungyun Choi, 2022. "Optimal Scheduling of Battery Energy Storage Systems and Demand Response for Distribution Systems with High Penetration of Renewable Energy Sources," Energies, MDPI, vol. 15(6), pages 1-18, March.
    18. Zhang, Jingrui & Zhou, Yulu & Li, Zhuoyun & Cai, Junfeng, 2021. "Three-level day-ahead optimal scheduling framework considering multi-stakeholders in active distribution networks: Up-to-down approach," Energy, Elsevier, vol. 219(C).
    19. Lazo, Joaquín & Watts, David, 2024. "Stochastic model for active distribution networks planning: An analysis of the combination of active network management schemes," Renewable and Sustainable Energy Reviews, Elsevier, vol. 191(C).
    20. Zhou, Siyu & Han, Yang & Mahmoud, Karar & Darwish, Mohamed M.F. & Lehtonen, Matti & Yang, Ping & Zalhaf, Amr S., 2023. "A novel unified planning model for distributed generation and electric vehicle charging station considering multi-uncertainties and battery degradation," Applied Energy, Elsevier, vol. 348(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jeners:v:18:y:2025:i:5:p:1020-:d:1595435. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager (email available below). General contact details of provider: https://www.mdpi.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.