IDEAS home Printed from https://ideas.repec.org/a/eee/energy/v289y2024ics0360544223033303.html
   My bibliography  Save this article

A hierarchical reinforcement learning GPC for flexible operation of ultra-supercritical unit considering economy

Author

Listed:
  • Hou, Guolian
  • Huang, Ting
  • Zheng, Fumeng
  • Huang, Congzhi

Abstract

With the growing proportion of renewable energy sources in power grid, the demands for deep peak shaving and rapid load regulation of ultra-supercritical (USC) unit to provide more flexible and stable power services is further enhanced. Simultaneously, the load tracking speed is no longer the only significant task in the flexible operation of USC unit, the operational economy is more and more important. To this end, a hierarchical reinforcement learning GPC control scheme with a better economy is proposed and apply to USC unit. Firstly, the model mismatch problem in control process is taken into account, and an integral mode is added into the GPC to eliminate the steady state error during the load regulation. Secondly, the hierarchical reinforcement learning GPC scheme integrated the GPC and twin delayed deep deterministic policy gradient algorithms is presented. The complex flexibility demand problem is decomposed into rapid load regulation and economic operation and then respectively solved in the GPC-based upper layer and reinforcement learning agent-based lower layer. Thirdly, under the flexibility demand, a multi-criteria optimization function including the load tracking cost, main steam throttling loss and coal consumption rate of unit is constructed. Then, via the accurate trajectory tracking by GPC and on-line control sequence optimization by reinforcement learning agent, the operational flexibility and economy of USC unit are simultaneously improved within safe boundaries. Finally, to verify the availability of the proposed hierarchical reinforcement learning GPC strategy, large-scale load variations and disturbance rejection ability tests cover the 30%–100 % rated load of a 1000 MW USC unit are carried out, and the load regulation performance is compared to other strategies. Results confirm that the designed hierarchical reinforcement learning GPC control scheme is profit for the flexible and economic operation capability promotion of USC unit.

Suggested Citation

  • Hou, Guolian & Huang, Ting & Zheng, Fumeng & Huang, Congzhi, 2024. "A hierarchical reinforcement learning GPC for flexible operation of ultra-supercritical unit considering economy," Energy, Elsevier, vol. 289(C).
    • Handle: RePEc:eee:energy:v:289:y:2024:i:c:s0360544223033303
    DOI: 10.1016/j.energy.2023.129936
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0360544223033303
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.energy.2023.129936?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Li, Qiang & Wei, Fanchao & Zhou, Yongcheng & Li, Jiajia & Zhou, Guowen & Wang, Zhonghao & Liu, Jinfu & Yan, Peigang & Yu, Daren, 2023. "A scheduling framework for VPP considering multiple uncertainties and flexible resources," Energy, Elsevier, vol. 282(C).
    2. Yong, Qingqing & Jin, Kaiyuan & Li, Xiaobo & Yang, Ronggui, 2023. "Thermo-economic analysis for a novel grid-scale pumped thermal electricity storage system coupled with a coal-fired power plant," Energy, Elsevier, vol. 280(C).
    3. Yan, Rujing & Wang, Jiangjiang & Huo, Shuojie & Zhang, Jing & Tang, Saiqiu & Yang, Mei, 2023. "Comparative study for four technologies on flexibility improvement and renewable energy accommodation of combined heat and power system," Energy, Elsevier, vol. 263(PE).
    4. Fu, Yue & Wang, Liyuan & Liu, Ming & Wang, Jinshi & Yan, Junjie, 2023. "Performance analysis of coal-fired power plants integrated with carbon capture system under load-cycling operation conditions," Energy, Elsevier, vol. 276(C).
    5. Li, Jianlan & Xin, Yu & Hu, Bo & Zeng, Kuo & Wu, Zhiyi & Fan, Shiwang & Li, Yuanyuan & Chen, Yongzhao & Wang, Shunjiang & Wang, Jizhou & Min, Yong & Li, Jun & Flamant, Gilles, 2021. "Safety and thermal efficiency performance assessment of solar aided coal-fired power plant based on turbine steam double reheat," Energy, Elsevier, vol. 226(C).
    6. Hou, Guolian & Xiong, Jian & Zhou, Guiping & Gong, Linjuan & Huang, Congzhi & Wang, Shunjiang, 2021. "Coordinated control system modeling of ultra-supercritical unit based on a new fuzzy neural network," Energy, Elsevier, vol. 234(C).
    7. Wang, Xuan & Wang, Rui & Jin, Ming & Shu, Gequn & Tian, Hua & Pan, Jiaying, 2020. "Control of superheat of organic Rankine cycle under transient heat source based on deep reinforcement learning," Applied Energy, Elsevier, vol. 278(C).
    8. Liu, Chunming & Wang, Chunling & Yin, Yujun & Yang, Peihong & Jiang, Hui, 2022. "Bi-level dispatch and control strategy based on model predictive control for community integrated energy system considering dynamic response performance," Applied Energy, Elsevier, vol. 310(C).
    9. Bravo, Julio & Abdulridha, Ahmed & Wang, Shuoyu & Matrone, Dominic & Yao, Zheng & Neti, Sudhakar & Naito, Clay & Quiel, Spencer & Suleiman, Muhannad & Romero, Carlos, 2023. "Design and experimental testing of a 150 kWh thermal battery using thermosiphons embedded in a concrete matrix for power plant flexible operation," Energy, Elsevier, vol. 277(C).
    10. Zhang, Guangming & Zhang, Chao & Wang, Wei & Cao, Huan & Chen, Zhenyu & Niu, Yuguang, 2023. "Offline reinforcement learning control for electricity and heat coordination in a supercritical CHP unit," Energy, Elsevier, vol. 266(C).
    11. De Lorenzi, Andrea & Gambarotta, Agostino & Marzi, Emanuela & Morini, Mirko & Saletti, Costanza, 2022. "Predictive control of a combined heat and power plant for grid flexibility under demand uncertainty," Applied Energy, Elsevier, vol. 314(C).
    12. Hong, Feng & Ji, Weiming & Pang, Yalei & Hao, Junhong & Du, Ming & Fang, Fang & Liu, Jizhen, 2023. "A new energy state-based modeling and performance assessment method for primary frequency control of thermal power plants," Energy, Elsevier, vol. 276(C).
    13. Yang, Tingting & Liu, Ziyuan & Zeng, Deliang & Zhu, Yansong, 2023. "Simulation and evaluation of flexible enhancement of thermal power unit coupled with flywheel energy storage array," Energy, Elsevier, vol. 281(C).
    14. Ibrahem, Ibrahem M.A. & Akhrif, Ouassima & Moustapha, Hany & Staniszewski, Martin, 2021. "Nonlinear generalized predictive controller based on ensemble of NARX models for industrial gas turbine engine," Energy, Elsevier, vol. 230(C).
    15. Lin, Xiaojie & Lin, Xueru & Zhong, Wei & Zhou, Yi, 2023. "Predictive operation optimization of multi-energy virtual power plant considering behavior uncertainty of diverse stakeholders," Energy, Elsevier, vol. 280(C).
    16. Wang, Zhu & Liu, Ming & Yan, Junjie, 2021. "Flexibility and efficiency co-enhancement of thermal power plant by control strategy improvement considering time varying and detailed boiler heat storage characteristics," Energy, Elsevier, vol. 232(C).
    17. Kong, Xiaobing & Abdelbaky, Mohamed Abdelkarim & Liu, Xiangjie & Lee, Kwang Y., 2023. "Stable feedback linearization-based economic MPC scheme for thermal power plant," Energy, Elsevier, vol. 268(C).
    18. Ceusters, Glenn & Rodríguez, Román Cantú & García, Alberte Bouso & Franke, Rüdiger & Deconinck, Geert & Helsen, Lieve & Nowé, Ann & Messagie, Maarten & Camargo, Luis Ramirez, 2021. "Model-predictive control and reinforcement learning in multi-energy system case studies," Applied Energy, Elsevier, vol. 303(C).
    19. Taler, Jan & Trojan, Marcin & Dzierwa, Piotr & Kaczmarski, Karol & Węglowski, Bohdan & Taler, Dawid & Zima, Wiesław & Grądziel, Sławomir & Ocłoń, Paweł & Sobota, Tomasz & Rerak, Monika & Jaremkiewicz,, 2023. "The flexible boiler operation in a wide range of load changes with considering the strength and environmental restrictions," Energy, Elsevier, vol. 263(PB).
    20. Tang, Zihan & Wu, Xiao, 2023. "Distributed predictive control guided by intelligent reboiler steam feedforward for the coordinated operation of power plant-carbon capture system," Energy, Elsevier, vol. 267(C).
    21. Skjervold, Vidar T. & Mondino, Giorgia & Riboldi, Luca & Nord, Lars O., 2023. "Investigation of control strategies for adsorption-based CO2 capture from a thermal power plant under variable load operation," Energy, Elsevier, vol. 268(C).
    22. Wang, Zhu & Liu, Ming & Yan, Hui & Yan, Junjie, 2022. "Optimization on coordinate control strategy assisted by high-pressure extraction steam throttling to achieve flexible and efficient operation of thermal power plants," Energy, Elsevier, vol. 244(PA).
    23. Wang, Congyu & Song, Jiwei, 2023. "Performance assessment of the novel coal-fired combined heat and power plant integrating with flexibility renovations," Energy, Elsevier, vol. 263(PC).
    24. Zhu, Hengyi & Tan, Peng & He, Ziqian & Zhang, Cheng & Fang, Qingyan & Chen, Gang, 2022. "Nonlinear model predictive control of USC boiler-turbine power units in flexible operations via input convex neural network," Energy, Elsevier, vol. 255(C).
    25. Hou, Guolian & Huang, Ting & Huang, Congzhi, 2023. "Flexibility improvement of 1000 MW ultra-supercritical unit under full operating conditions by error-based ADRC and fast pigeon-inspired optimizer," Energy, Elsevier, vol. 270(C).
    26. Jiang, Yue & Duan, Liqiang & Pang, Liping & Song, Jifeng, 2021. "Thermal performance study of tower solar aided double reheat coal-fired power generation system," Energy, Elsevier, vol. 230(C).
    27. Roy, Kallol & Mandal, Kamal Krishna & Mandal, Atis Chandra, 2019. "Ant-Lion Optimizer algorithm and recurrent neural network for energy management of micro grid connected system," Energy, Elsevier, vol. 167(C), pages 402-416.
    28. Zhou, Hong & Chen, Cheng & Lai, Jingang & Lu, Xiaoqing & Deng, Qijun & Gao, Xingran & Lei, Zhongcheng, 2018. "Affine nonlinear control for an ultra-supercritical coal fired once-through boiler-turbine unit," Energy, Elsevier, vol. 153(C), pages 638-649.
    29. Hou, Guolian & Huang, Ting & Zheng, Fumeng & Gong, Linjuan & Huang, Congzhi & Zhang, Jianhua, 2023. "Application of multi-agent EADRC in flexible operation of combined heat and power plant considering carbon emission and economy," Energy, Elsevier, vol. 263(PB).
    30. Huang, Ruchen & He, Hongwen & Zhao, Xuyang & Wang, Yunlong & Li, Menglin, 2022. "Battery health-aware and naturalistic data-driven energy management for hybrid electric bus based on TD3 deep reinforcement learning algorithm," Applied Energy, Elsevier, vol. 321(C).
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Hou, Guolian & Huang, Ting & Huang, Congzhi, 2023. "Flexibility improvement of 1000 MW ultra-supercritical unit under full operating conditions by error-based ADRC and fast pigeon-inspired optimizer," Energy, Elsevier, vol. 270(C).
    2. Li, Xuehan & Wang, Wei & Ye, Lingling & Ren, Guorui & Fang, Fang & Liu, Jizhen & Chen, Zhe & Zhou, Qiang, 2024. "Improving frequency regulation ability for a wind-thermal power system by multi-objective optimized sliding mode control design," Energy, Elsevier, vol. 300(C).
    3. Dong, Zhe & Cheng, Zhonghua & Zhu, Yunlong & Huang, Xiaojin & Dong, Yujie & Zhang, Zuoyi, 2023. "Coordinated control of mHTGR-based nuclear steam supply systems considering cold helium temperature," Energy, Elsevier, vol. 284(C).
    4. Wu, Long & Yin, Xunyuan & Pan, Lei & Liu, Jinfeng, 2023. "Distributed economic predictive control of integrated energy systems for enhanced synergy and grid response: A decomposition and cooperation strategy," Applied Energy, Elsevier, vol. 349(C).
    5. Yan, Hui & Liu, Ming & Wang, Zhu & Zhang, Kezhen & Chong, Daotong & Yan, Junjie, 2023. "Flexibility enhancement of solar-aided coal-fired power plant under different direct normal irradiance conditions," Energy, Elsevier, vol. 262(PA).
    6. Liu, Zefeng & Wang, Chaoyang & Fan, Jianlin & Liu, Ming & Xing, Yong & Yan, Junjie, 2024. "Enhancing the flexibility and stability of coal-fired power plants by optimizing control schemes of throttling high-pressure extraction steam," Energy, Elsevier, vol. 288(C).
    7. Yin, Linfei & Zhou, Hang, 2024. "Modal decomposition integrated model for ultra-supercritical coal-fired power plant reheater tube temperature multi-step prediction," Energy, Elsevier, vol. 292(C).
    8. Taler, Dawid & Kaczmarski, Karol & Dzierwa, Piotr & Taler, Jan & Trojan, Marcin, 2024. "Optimisation of the cooling of pressurised thick-walled components operating with fluid at saturation temperature," Energy, Elsevier, vol. 290(C).
    9. Li, Chao & Sun, Yang & Bi, Tianjiao & Zhai, Rongrong, 2023. "Performance enhancement of a solar-assisted pulverized coal power system by integrating a supercritical CO2 cycle," Renewable Energy, Elsevier, vol. 219(P1).
    10. Sterkhov, K.V. & Khokhlov, D.A. & Zaichenko, M.N., 2024. "Zero carbon emission CCGT power plant with integrated solid fuel gasification," Energy, Elsevier, vol. 294(C).
    11. Esmaeili, Mohammad & Moradi, Hamed, 2023. "Robust & nonlinear control of an ultra-supercritical coal fired once-through boiler-turbine unit in order to optimize the uncertain problem," Energy, Elsevier, vol. 282(C).
    12. Hou, Guolian & Gong, Linjuan & Hu, Bo & Su, Huilin & Huang, Ting & Huang, Congzhi & Fan, Wei & Zhao, Yuanzhu, 2022. "Application of fast adaptive moth-flame optimization in flexible operation modeling for supercritical unit," Energy, Elsevier, vol. 239(PA).
    13. Hou, Guolian & Huang, Ting & Jiang, Hao & Cao, Huan & Zhang, Tianhao & Zhang, Jianhua & Gao, He & Liu, Yong & Zhou, Zhenhua & An, Zhenyi, 2024. "A flexible and deep peak shaving scheme for combined heat and power plant under full operating conditions," Energy, Elsevier, vol. 299(C).
    14. Wang, Jian & Ilea, Valentin & Bovo, Cristian & Xie, Ning & Wang, Yong, 2023. "Optimal self-scheduling for a multi-energy virtual power plant providing energy and reserve services under a holistic market framework," Energy, Elsevier, vol. 278(PB).
    15. Wu, Long & Yin, Xunyuan & Pan, Lei & Liu, Jinfeng, 2022. "Economic model predictive control of integrated energy systems: A multi-time-scale framework," Applied Energy, Elsevier, vol. 328(C).
    16. Jiakui Shi & Shuangshuang Fan & Jiajia Li & Jiangnan Cheng & Jie Wan & Peng E, 2023. "An Optimization Method of Steam Turbine Load Resilient Adjustment by Characterizing Dynamic Changes in Superheated Steam Energy," Energies, MDPI, vol. 16(8), pages 1-15, April.
    17. Tang, Zhenhao & Sui, Mengxuan & Wang, Xu & Xue, Wenyuan & Yang, Yuan & Wang, Zhi & Ouyang, Tinghui, 2024. "Theory-guided deep neural network for boiler 3-D NOx concentration distribution prediction," Energy, Elsevier, vol. 299(C).
    18. Yanfeng Li & Jingru Liu & Guohe Huang, 2022. "Pressure Drop Optimization of the Main Steam and Reheat Steam System of a 1000 MW Secondary Reheat Unit," Energies, MDPI, vol. 15(9), pages 1-15, April.
    19. Barik, Soumyabrata & Das, Debapriya, 2020. "A novel Q−PQV bus pair method of biomass DGs placement in distribution networks to maintain the voltage of remotely located buses," Energy, Elsevier, vol. 194(C).
    20. Gokhale, Gargya & Claessens, Bert & Develder, Chris, 2022. "Physics informed neural networks for control oriented thermal modeling of buildings," Applied Energy, Elsevier, vol. 314(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:energy:v:289:y:2024:i:c:s0360544223033303. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.journals.elsevier.com/energy .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.