IDEAS home Printed from https://ideas.repec.org/a/gam/jeners/v12y2019i18p3461-d265235.html
   My bibliography  Save this article

Application of a Deep Deterministic Policy Gradient Algorithm for Energy-Aimed Timetable Rescheduling Problem

Author

Listed:
  • Guang Yang

    (School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University, 800 Dongchuan RD, Shanghai 200240, China)

  • Feng Zhang

    (School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University, 800 Dongchuan RD, Shanghai 200240, China)

  • Cheng Gong

    (School of Electrical and Computer Engineering, Georgia Institute of Technology, 85 5th Street NW, Atlanta, GA 30308, USA)

  • Shiwen Zhang

    (School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University, 800 Dongchuan RD, Shanghai 200240, China)

Abstract

Reinforcement learning has potential in the area of intelligent transportation due to its generality and real-time feature. The Q-learning algorithm, which is an early proposed algorithm, has its own merits to solve the train timetable rescheduling (TTR) problem. However, it has shortage in two aspects: Dimensional limits of action and a slow convergence rate. In this paper, a deep deterministic policy gradient (DDPG) algorithm is applied to solve the energy-aimed train timetable rescheduling (ETTR) problem. This algorithm belongs to reinforcement learning, which fulfills real-time requirements of the ETTR problem, and has adaptability on random disturbances. Superior to the Q-learning, DDPG has a continuous state space and action space. After enough training, the learning agent based on DDPG takes proper action by adjusting the cruising speed and the dwelling time continuously for each train in a metro network when random disturbances happen. Although training needs an iteration for thousands of episodes, the policy decision during each testing episode takes a very short time. Models for the metro network, based on a real case of the Shanghai Metro Line 1, are established as a training and testing environment. To validate the energy-saving effect and the real-time feature of the proposed algorithm, four experiments are designed and conducted. Compared with the no action strategy, results show that the proposed algorithm has real-time performance, and saves a significant percentage of energy under random disturbances.

Suggested Citation

  • Guang Yang & Feng Zhang & Cheng Gong & Shiwen Zhang, 2019. "Application of a Deep Deterministic Policy Gradient Algorithm for Energy-Aimed Timetable Rescheduling Problem," Energies, MDPI, vol. 12(18), pages 1-19, September.
    • Handle: RePEc:gam:jeners:v:12:y:2019:i:18:p:3461-:d:265235
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/1996-1073/12/18/3461/pdf
    Download Restriction: no
    File URL: https://www.mdpi.com/1996-1073/12/18/3461/
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Francisco A. Ortega & Miguel A. Pozo & Justo Puerto, 2018. "On-Line Timetable Rescheduling in a Transit Line," Transportation Science, INFORMS, vol. 52(5), pages 1106-1121, October.
    2. Li Wang & Wenting Mo & Yong Qin & Fei Dou & Limin Jia, 2014. "Optimization Based High-Speed Railway Train Rescheduling with Speed Restriction," Discrete Dynamics in Nature and Society, Hindawi, vol. 2014, pages 1-14, January.
    3. Cheng Gong & Shiwen Zhang & Feng Zhang & Jianguo Jiang & Xinheng Wang, 2014. "An Integrated Energy-Efficient Operation Methodology for Metro Systems Based on a Real Case of Shanghai Metro Line One," Energies, MDPI, vol. 7(11), pages 1-25, November.
    4. Šemrov, D. & Marsetič, R. & Žura, M. & Todorovski, L. & Srdic, A., 2016. "Reinforcement learning approach for train rescheduling on a single-track railway," Transportation Research Part B: Methodological, Elsevier, vol. 86(C), pages 250-267.
    5. Albrecht, Amie & Howlett, Phil & Pudney, Peter & Vu, Xuan & Zhou, Peng, 2016. "The key principles of optimal train control—Part 1: Formulation of the model, strategies of optimal type, evolutionary lines, location of optimal switching points," Transportation Research Part B: Methodological, Elsevier, vol. 94(C), pages 482-508.
    6. Petru Valentin Radu & Adam Szelag & Marcin Steczek, 2019. "On-Board Energy Storage Devices with Supercapacitors for Metro Trains—Case Study Analysis of Application Effectiveness," Energies, MDPI, vol. 12(7), pages 1-22, April.
    7. Adrián Fernández-Rodríguez & Antonio Fernández-Cardador & Asunción P. Cucala & Maria Carmen Falvo, 2019. "Energy Efficiency and Integration of Urban Electrical Transport Systems: EVs and Metro-Trains of Two Real European Lines," Energies, MDPI, vol. 12(3), pages 1-20, January.
    8. Agostinho Rocha & Armando Araújo & Adriano Carvalho & João Sepulveda, 2018. "A New Approach for Real Time Train Energy Efficiency Optimization," Energies, MDPI, vol. 11(10), pages 1-21, October.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Bo Hu & Jiaxi Li & Shuang Li & Jie Yang, 2019. "A Hybrid End-to-End Control Strategy Combining Dueling Deep Q-network and PID for Transient Boost Control of a Diesel Engine with Variable Geometry Turbocharger and Cooled EGR," Energies, MDPI, vol. 12(19), pages 1-15, September.
    2. Zhang, Lang & He, Deqiang & He, Yan & Liu, Bin & Chen, Yanjun & Shan, Sheng, 2022. "Real-time energy saving optimization method for urban rail transit train timetable under delay condition," Energy, Elsevier, vol. 258(C).
    3. Jiang Liu & Tian-tian Li & Bai-gen Cai & Jiao Zhang, 2020. "Boundary Identification for Traction Energy Conservation Capability of Urban Rail Timetables: A Case Study of the Beijing Batong Line," Energies, MDPI, vol. 13(8), pages 1-25, April.
    4. Zhan, Shuguang & Wang, Pengling & Wong, S.C. & Lo, S.M., 2022. "Energy-efficient high-speed train rescheduling during a major disruption," Transportation Research Part E: Logistics and Transportation Review, Elsevier, vol. 157(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Artur Kierzkowski & Szymon Haładyn, 2022. "Method for Reconfiguring Train Schedules Taking into Account the Global Reduction of Railway Energy Consumption," Energies, MDPI, vol. 15(5), pages 1-18, March.
    2. Franciszek Restel & Szymon Mateusz Haładyn, 2022. "The Railway Timetable Evaluation Method in Terms of Operational Robustness against Overloads of the Power Supply System," Energies, MDPI, vol. 15(17), pages 1-17, September.
    3. Wang, Xuekai & D’Ariano, Andrea & Su, Shuai & Tang, Tao, 2023. "Cooperative train control during the power supply shortage in metro system: A multi-agent reinforcement learning approach," Transportation Research Part B: Methodological, Elsevier, vol. 170(C), pages 244-278.
    4. Zhang, Lang & He, Deqiang & He, Yan & Liu, Bin & Chen, Yanjun & Shan, Sheng, 2022. "Real-time energy saving optimization method for urban rail transit train timetable under delay condition," Energy, Elsevier, vol. 258(C).
    5. M. Shakibayifar & A. Sheikholeslami & F. Corman & E. Hassannayebi, 2020. "An integrated rescheduling model for minimizing train delays in the case of line blockage," Operational Research, Springer, vol. 20(1), pages 59-87, March.
    6. Boud Verbrugge & Mohammed Mahedi Hasan & Haaris Rasool & Thomas Geury & Mohamed El Baghdadi & Omar Hegazy, 2021. "Smart Integration of Electric Buses in Cities: A Technological Review," Sustainability, MDPI, vol. 13(21), pages 1-23, November.
    7. Zhan, Shuguang & Wang, Pengling & Wong, S.C. & Lo, S.M., 2022. "Energy-efficient high-speed train rescheduling during a major disruption," Transportation Research Part E: Logistics and Transportation Review, Elsevier, vol. 157(C).
    8. Andreas Bärmann & Alexander Martin & Oskar Schneider, 2017. "A comparison of performance metrics for balancing the power consumption of trains in a railway network by slight timetable adaptation," Public Transport, Springer, vol. 9(1), pages 95-113, July.
    9. Jefimowski, Włodzimierz & Szeląg, Adam & Steczek, Marcin & Nikitenko, Anatolii, 2020. "Vanadium redox flow battery parameters optimization in a transportation microgrid: A case study," Energy, Elsevier, vol. 195(C).
    10. Xiaowen Wang & Zhuang Xiao & Mo Chen & Pengfei Sun & Qingyuan Wang & Xiaoyun Feng, 2020. "Energy-Efficient Speed Profile Optimization and Sliding Mode Speed Tracking for Metros," Energies, MDPI, vol. 13(22), pages 1-29, November.
    11. Luan, Xiaojie & Wang, Yihui & De Schutter, Bart & Meng, Lingyun & Lodewijks, Gabriel & Corman, Francesco, 2018. "Integration of real-time traffic management and train control for rail networks - Part 2: Extensions towards energy-efficient train operations," Transportation Research Part B: Methodological, Elsevier, vol. 115(C), pages 72-94.
    12. Iosvany López-Sandin & Guadalupe Gutiérrez-Soto & Adriana Gutiérrez-Díez & Nancy Medina-Herrera & Edgar Gutiérrez-Castorena & Francisco Zavala-García, 2019. "Evaluation of the Use of Energy in the Production of Sweet Sorghum ( Sorghum Bicolor (L.) Moench) under Different Production Systems," Energies, MDPI, vol. 12(9), pages 1-13, May.
    13. Ding, Yida & Wandelt, Sebastian & Wu, Guohua & Xu, Yifan & Sun, Xiaoqian, 2023. "Towards efficient airline disruption recovery with reinforcement learning," Transportation Research Part E: Logistics and Transportation Review, Elsevier, vol. 179(C).
    14. Andreas Bärmann & Alexander Martin & Oskar Schneider, 2021. "Efficient Formulations and Decomposition Approaches for Power Peak Reduction in Railway Traffic via Timetabling," Transportation Science, INFORMS, vol. 55(3), pages 747-767, May.
    15. Zhang, Chi & Zeng, Guohong & Wu, Jian & Wei, Shaoyuan & Zhang, Weige & Sun, Bingxiang, 2023. "Integrated optimization of driving strategy and energy management for hybrid diesel multiple units," Energy, Elsevier, vol. 281(C).
    16. Agata Pomykala & Adam Szelag, 2022. "Reduction of Power Consumption and CO 2 Emissions as a Result of Putting into Service High-Speed Trains: Polish Case," Energies, MDPI, vol. 15(12), pages 1-24, June.
    17. Xavier Boulet & Mahdi Zargayouna & Gérard Scemama & Fabien Leurent, 2021. "A Middleware-Based Approach for Multi-Scale Mobility Simulation," Future Internet, MDPI, vol. 13(2), pages 1-21, January.
    18. Blanco, Víctor & Conde, Eduardo & Hinojosa, Yolanda & Puerto, Justo, 2020. "An optimization model for line planning and timetabling in automated urban metro subway networks. A case study," Omega, Elsevier, vol. 92(C).
    19. Yin, Jiateng & Yang, Lixing & Tang, Tao & Gao, Ziyou & Ran, Bin, 2017. "Dynamic passenger demand oriented metro train scheduling with energy-efficiency and waiting time minimization: Mixed-integer linear programming approaches," Transportation Research Part B: Methodological, Elsevier, vol. 97(C), pages 182-213.
    20. Wang, Xuekai & Tang, Tao & Su, Shuai & Yin, Jiateng & Gao, Ziyou & Lv, Nan, 2021. "An integrated energy-efficient train operation approach based on the space-time-speed network methodology," Transportation Research Part E: Logistics and Transportation Review, Elsevier, vol. 150(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jeners:v:12:y:2019:i:18:p:3461-:d:265235. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager (email available below). General contact details of provider: https://www.mdpi.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.