IDEAS home Printed from https://ideas.repec.org/a/eee/appene/v286y2021ics0306261921000878.html
   My bibliography  Save this article

A Q-learning based transient power optimization method for organic Rankine cycle waste heat recovery system in heavy duty diesel engine applications

Author

Listed:
  • Xu, Bin
  • Li, Xiaoya

Abstract

In recent years, the organic Rankine cycle waste heat recovery (ORC-WHR) technology gains popularity in heavy-duty diesel engine applications. Drastic fluctuations of the waste heat caused by variable daily operation of mobile heavy-duty trucks bring an extreme transient power optimization challenge to ORC-WHR systems. Existing power optimization methods either neglect transient behavior of the Rankine cycle system or compromise model accuracy for computation efficiency. Different from literature, this study first time proposes a model-free reinforcement learning method to achieve online transient power optimization for the ORC-WHR system and explains the benefits of learning method in this application. A tabular Q-learning is formulated to optimize the net power on an experimentally validated ORC-WHR system. Q-learning is explained in detail using states, action, and policy information. To quantify the power optimization of the proposed method, Proper-Integral-Derivative method, state-of-art offline and online Dynamic Programming methods are implemented. The results showed that Q-learning generated 22% more cumulative energy than the energy Proper-Integral-Derivative method generated. Furthermore, Q-learning produces 96.6% of cumulative energy that the offline Dynamic Programming generates over a transient engine condition, while it requires less computation cost and is executed online. Additionally, the Q-learning produces 0.5% more cumulative energy than the machine learning-based online Dynamic Programming results and exhibits better vapor temperature robustness than the online Dynamic Programming method (4 °C-28 °C superheat by Q-learning vs. 5 °C-94 °C superheat by online Dynamic Programming). Given the excellent power production performance, low computation cost requirement and high robustness, the proposed Q-learning method has the potential to improve the power production of the ORC-WHR system with different configurations.

Suggested Citation

  • Xu, Bin & Li, Xiaoya, 2021. "A Q-learning based transient power optimization method for organic Rankine cycle waste heat recovery system in heavy duty diesel engine applications," Applied Energy, Elsevier, vol. 286(C).
    • Handle: RePEc:eee:appene:v:286:y:2021:i:c:s0306261921000878
    DOI: 10.1016/j.apenergy.2021.116532
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0306261921000878
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.apenergy.2021.116532?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Xu, Bin & Rathod, Dhruvang & Zhang, Darui & Yebi, Adamu & Zhang, Xueyu & Li, Xiaoya & Filipi, Zoran, 2020. "Parametric study on reinforcement learning optimized energy management strategy for a hybrid electric vehicle," Applied Energy, Elsevier, vol. 259(C).
    2. Xu, Bin & Rathod, Dhruvang & Kulkarni, Shreyas & Yebi, Adamu & Filipi, Zoran & Onori, Simona & Hoffman, Mark, 2017. "Transient dynamic modeling and validation of an organic Rankine cycle waste heat recovery system for heavy duty diesel engine applications," Applied Energy, Elsevier, vol. 205(C), pages 260-279.
    3. Wang, E.H. & Zhang, H.G. & Fan, B.Y. & Ouyang, M.G. & Zhao, Y. & Mu, Q.H., 2011. "Study of working fluid selection of organic Rankine cycle (ORC) for engine waste heat recovery," Energy, Elsevier, vol. 36(5), pages 3406-3418.
    4. Quoilin, Sylvain & Aumann, Richard & Grill, Andreas & Schuster, Andreas & Lemort, Vincent & Spliethoff, Hartmut, 2011. "Dynamic modeling and optimal control strategy of waste heat recovery Organic Rankine Cycles," Applied Energy, Elsevier, vol. 88(6), pages 2183-2190, June.
    5. Hernandez, Andres & Desideri, Adriano & Gusev, Sergei & Ionescu, Clara M. & Den Broek, Martijn Van & Quoilin, Sylvain & Lemort, Vincent & De Keyser, Robin, 2017. "Design and experimental validation of an adaptive control law to maximize the power generation of a small-scale waste heat recovery system," Applied Energy, Elsevier, vol. 203(C), pages 549-559.
    6. Xu, Bin & Rathod, Dhruvang & Yebi, Adamu & Filipi, Zoran & Onori, Simona & Hoffman, Mark, 2019. "A comprehensive review of organic rankine cycle waste heat recovery systems in heavy-duty diesel engine applications," Renewable and Sustainable Energy Reviews, Elsevier, vol. 107(C), pages 145-170.
    7. Quoilin, Sylvain & Broek, Martijn Van Den & Declaye, Sébastien & Dewallef, Pierre & Lemort, Vincent, 2013. "Techno-economic survey of Organic Rankine Cycle (ORC) systems," Renewable and Sustainable Energy Reviews, Elsevier, vol. 22(C), pages 168-186.
    8. Li, Xiaoya & Tian, Hua & Shu, Gequn & Zhao, Mingru & Markides, Christos N. & Hu, Chen, 2019. "Potential of carbon dioxide transcritical power cycle waste-heat recovery systems for heavy-duty truck engines," Applied Energy, Elsevier, vol. 250(C), pages 1581-1599.
    9. Volodymyr Mnih & Koray Kavukcuoglu & David Silver & Andrei A. Rusu & Joel Veness & Marc G. Bellemare & Alex Graves & Martin Riedmiller & Andreas K. Fidjeland & Georg Ostrovski & Stig Petersen & Charle, 2015. "Human-level control through deep reinforcement learning," Nature, Nature, vol. 518(7540), pages 529-533, February.
    10. Emanuel Feru & Frank Willems & Bram De Jager & Maarten Steinbuch, 2014. "Modeling and Control of a Parallel Waste Heat Recovery System for Euro-VI Heavy-Duty Diesel Engines," Energies, MDPI, vol. 7(10), pages 1-22, October.
    11. Xu, Bin & Rathod, Dhruvang & Yebi, Adamu & Filipi, Zoran, 2020. "Real-time realization of Dynamic Programming using machine learning methods for IC engine waste heat recovery system power optimization," Applied Energy, Elsevier, vol. 262(C).
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Lu, Bowen & Zhang, Zhifu & Cai, Jinwen & Wang, Wei & Ju, Xueming & Xu, Yao & Lu, Xun & Tian, Hua & Shi, Lingfeng & Shu, Gequn, 2023. "Integrating engine thermal management into waste heat recovery under steady-state design and dynamic off-design conditions," Energy, Elsevier, vol. 272(C).
    2. Lin, Runze & Luo, Yangyang & Wu, Xialai & Chen, Junghui & Huang, Biao & Su, Hongye & Xie, Lei, 2024. "Surrogate empowered Sim2Real transfer of deep reinforcement learning for ORC superheat control," Applied Energy, Elsevier, vol. 356(C).
    3. Zhang, Hongsheng & Hao, Ruijun & Liu, Xingang & Zhang, Ning & Guo, Wenli & Zhang, Zhenghui & Liu, Chengjun & Liu, Yifeng & Duan, Chenghong & Qin, Jiyun, 2022. "Thermodynamic performance analysis of an improved coal-fired power generation system coupled with geothermal energy based on organic Rankine cycle," Renewable Energy, Elsevier, vol. 201(P1), pages 273-290.
    4. Xinxing Lin & Chonghui Chen & Aofang Yu & Likun Yin & Wen Su, 2021. "Performance Comparison of Advanced Transcritical Power Cycles with High-Temperature Working Fluids for the Engine Waste Heat Recovery," Energies, MDPI, vol. 14(18), pages 1-32, September.
    5. Ping, Xu & Yang, Fubin & Zhang, Hongguang & Xing, Chengda & Yu, Mingzhe & Wang, Yan, 2023. "Investigation and multi-objective optimization of vehicle engine-organic Rankine cycle (ORC) combined system in different driving conditions," Energy, Elsevier, vol. 263(PB).
    6. Ping, Xu & Yang, Fubin & Zhang, Hongguang & Xing, Chengda & Pan, Yachao & Zhang, Wujie & Wang, Yan, 2023. "Nonlinear modeling and multi-scale influence characteristics analysis of organic Rankine cycle (ORC) system considering variable driving cycles," Energy, Elsevier, vol. 265(C).
    7. Abid, Md. Shadman & Apon, Hasan Jamil & Hossain, Salman & Ahmed, Ashik & Ahshan, Razzaqul & Lipu, M.S. Hossain, 2024. "A novel multi-objective optimization based multi-agent deep reinforcement learning approach for microgrid resources planning," Applied Energy, Elsevier, vol. 353(PA).
    8. Li, Xiaoya & Xu, Bin & Tian, Hua & Shu, Gequn, 2021. "Towards a novel holistic design of organic Rankine cycle (ORC) systems operating under heat source fluctuations and intermittency," Renewable and Sustainable Energy Reviews, Elsevier, vol. 147(C).
    9. Ping, Xu & Yang, Fubin & Zhang, Hongguang & Zhang, Jian & Xing, Chengda & Yan, Yinlian & Yang, Anren & Wang, Yan, 2023. "Information theory-based dynamic feature capture and global multi-objective optimization approach for organic Rankine cycle (ORC) considering road environment," Applied Energy, Elsevier, vol. 348(C).
    10. Ping, Xu & Yang, Fubin & Zhang, Hongguang & Xing, Chengda & Yao, Baofeng & Wang, Yan, 2022. "An outlier removal and feature dimensionality reduction framework with unsupervised learning and information theory intervention for organic Rankine cycle (ORC)," Energy, Elsevier, vol. 254(PB).
    11. Daniel Egan & Qilun Zhu & Robert Prucka, 2023. "A Review of Reinforcement Learning-Based Powertrain Controllers: Effects of Agent Selection for Mixed-Continuity Control and Reward Formulation," Energies, MDPI, vol. 16(8), pages 1-31, April.
    12. Ping, Xu & Yang, Fubin & Zhang, Hongguang & Xing, Chengda & Yang, Anren & Yan, Yinlian & Pan, Yachao & Wang, Yan, 2023. "Ensemble of self-organizing adaptive maps and dynamic multi-objective optimization for organic Rankine cycle (ORC) under transportation and driving environment," Energy, Elsevier, vol. 275(C).
    13. Xu Ping & Baofeng Yao & Hongguang Zhang & Hongzhi Zhang & Jia Liang & Meng Yuan & Kai Niu & Yan Wang, 2022. "Comprehensive Performance Assessment of Dual Loop Organic Rankine Cycle (DORC) for CNG Engine: Energy, Thermoeconomic and Environment," Energies, MDPI, vol. 15(21), pages 1-28, October.
    14. Huiru Zhao & Chao Zhang & Yihang Zhao & Xuejie Wang, 2022. "Low-Carbon Economic Dispatching of Multi-Energy Virtual Power Plant with Carbon Capture Unit Considering Uncertainty and Carbon Market," Energies, MDPI, vol. 15(19), pages 1-25, October.
    15. Yang, Liu & Su, Zixiang, 2022. "An eco-friendly and efficient trigeneration system for dual-fuel marine engine considering heat storage and energy deployment," Energy, Elsevier, vol. 239(PA).
    16. Johannes Petrus Bester & Martin Van Eldik & Philip van Zyl Venter, 2023. "Energy Recovery Maximisation Modelling Subject to Constrained Cooling," Energies, MDPI, vol. 17(1), pages 1-23, December.
    17. Ifaei, Pouya & Nazari-Heris, Morteza & Tayerani Charmchi, Amir Saman & Asadi, Somayeh & Yoo, ChangKyoo, 2023. "Sustainable energies and machine learning: An organized review of recent applications and challenges," Energy, Elsevier, vol. 266(C).
    18. Feng, Yong-Qiang & Zhang, Qiang & Xu, Kang-Jing & Wang, Chun-Ming & He, Zhi-Xia & Hung, Tzu-Chen, 2023. "Operation characteristics and performance prediction of a 3 kW organic Rankine cycle (ORC) with automatic control system based on machine learning methodology," Energy, Elsevier, vol. 263(PC).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Li, Xiaoya & Xu, Bin & Tian, Hua & Shu, Gequn, 2021. "Towards a novel holistic design of organic Rankine cycle (ORC) systems operating under heat source fluctuations and intermittency," Renewable and Sustainable Energy Reviews, Elsevier, vol. 147(C).
    2. Xu, Bin & Rathod, Dhruvang & Yebi, Adamu & Filipi, Zoran & Onori, Simona & Hoffman, Mark, 2019. "A comprehensive review of organic rankine cycle waste heat recovery systems in heavy-duty diesel engine applications," Renewable and Sustainable Energy Reviews, Elsevier, vol. 107(C), pages 145-170.
    3. Imran, Muhammad & Pili, Roberto & Usman, Muhammad & Haglind, Fredrik, 2020. "Dynamic modeling and control strategies of organic Rankine cycle systems: Methods and challenges," Applied Energy, Elsevier, vol. 276(C).
    4. Rijpkema, Jelmer & Erlandsson, Olof & Andersson, Sven B. & Munch, Karin, 2022. "Exhaust waste heat recovery from a heavy-duty truck engine: Experiments and simulations," Energy, Elsevier, vol. 238(PB).
    5. Wen Zhang & Enhua Wang & Fanxiao Meng & Fujun Zhang & Changlu Zhao, 2020. "Closed-Loop PI Control of an Organic Rankine Cycle for Engine Exhaust Heat Recovery," Energies, MDPI, vol. 13(15), pages 1-20, July.
    6. Vaupel, Yannic & Huster, Wolfgang R. & Mhamdi, Adel & Mitsos, Alexander, 2021. "Optimal operating policies for organic Rankine cycles for waste heat recovery under transient conditions," Energy, Elsevier, vol. 224(C).
    7. Daniel Egan & Qilun Zhu & Robert Prucka, 2023. "A Review of Reinforcement Learning-Based Powertrain Controllers: Effects of Agent Selection for Mixed-Continuity Control and Reward Formulation," Energies, MDPI, vol. 16(8), pages 1-31, April.
    8. Cioccolanti, Luca & Tascioni, Roberto & Arteconi, Alessia, 2018. "Mathematical modelling of operation modes and performance evaluation of an innovative small-scale concentrated solar organic Rankine cycle plant," Applied Energy, Elsevier, vol. 221(C), pages 464-476.
    9. Andres Hernandez & Adriano Desideri & Clara Ionescu & Robin De Keyser & Vincent Lemort & Sylvain Quoilin, 2016. "Real-Time Optimization of Organic Rankine Cycle Systems by Extremum-Seeking Control," Energies, MDPI, vol. 9(5), pages 1-18, May.
    10. Chintala, Venkateswarlu & Kumar, Suresh & Pandey, Jitendra K., 2018. "A technical review on waste heat recovery from compression ignition engines using organic Rankine cycle," Renewable and Sustainable Energy Reviews, Elsevier, vol. 81(P1), pages 493-509.
    11. Wang, Xuan & Wang, Rui & Jin, Ming & Shu, Gequn & Tian, Hua & Pan, Jiaying, 2020. "Control of superheat of organic Rankine cycle under transient heat source based on deep reinforcement learning," Applied Energy, Elsevier, vol. 278(C).
    12. Hernandez, Andres & Desideri, Adriano & Gusev, Sergei & Ionescu, Clara M. & Den Broek, Martijn Van & Quoilin, Sylvain & Lemort, Vincent & De Keyser, Robin, 2017. "Design and experimental validation of an adaptive control law to maximize the power generation of a small-scale waste heat recovery system," Applied Energy, Elsevier, vol. 203(C), pages 549-559.
    13. Cai, Jinwen & Shu, Gequn & Tian, Hua & Wang, Xuan & Wang, Rui & Shi, Xiaolei, 2020. "Validation and analysis of organic Rankine cycle dynamic model using zeotropic mixture," Energy, Elsevier, vol. 197(C).
    14. Roberto Pili & Hartmut Spliethoff & Christoph Wieland, 2017. "Dynamic Simulation of an Organic Rankine Cycle—Detailed Model of a Kettle Boiler," Energies, MDPI, vol. 10(4), pages 1-28, April.
    15. Jiménez-Arreola, Manuel & Pili, Roberto & Wieland, Christoph & Romagnoli, Alessandro, 2018. "Analysis and comparison of dynamic behavior of heat exchangers for direct evaporation in ORC waste heat recovery applications from fluctuating sources," Applied Energy, Elsevier, vol. 216(C), pages 724-740.
    16. Cavazzini, G. & Bari, S. & Pavesi, G. & Ardizzon, G., 2017. "A multi-fluid PSO-based algorithm for the search of the best performance of sub-critical Organic Rankine Cycles," Energy, Elsevier, vol. 129(C), pages 42-58.
    17. Huster, Wolfgang R. & Vaupel, Yannic & Mhamdi, Adel & Mitsos, Alexander, 2018. "Validated dynamic model of an organic Rankine cycle (ORC) for waste heat recovery in a diesel truck," Energy, Elsevier, vol. 151(C), pages 647-661.
    18. Ni, Jiaxin & Zhao, Li & Zhang, Zhengtao & Zhang, Ying & Zhang, Jianyuan & Deng, Shuai & Ma, Minglu, 2018. "Dynamic performance investigation of organic Rankine cycle driven by solar energy under cloudy condition," Energy, Elsevier, vol. 147(C), pages 122-141.
    19. Cheng, Ziyang & Wang, Jiangfeng & Yang, Peijun & Wang, Yaxiong & Chen, Gang & Zhao, Pan & Dai, Yiping, 2022. "Comparison of control strategies and dynamic behaviour analysis of a Kalina cycle driven by a low-grade heat source," Energy, Elsevier, vol. 242(C).
    20. Wu, Xialai & Chen, Junghui & Xie, Lei, 2018. "Integrated operation design and control of Organic Rankine Cycle systems with disturbances," Energy, Elsevier, vol. 163(C), pages 115-129.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:appene:v:286:y:2021:i:c:s0306261921000878. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/wps/find/journaldescription.cws_home/405891/description#description .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.