IDEAS home Printed from https://ideas.repec.org/a/eee/transb/v170y2023icp244-278.html
   My bibliography  Save this article

Cooperative train control during the power supply shortage in metro system: A multi-agent reinforcement learning approach

Author

Listed:
  • Wang, Xuekai
  • D’Ariano, Andrea
  • Su, Shuai
  • Tang, Tao

Abstract

In metro system, the fault of traction power supply system may cause the power supply shortage around the failure substation. In this case, the dispatching measure should be immediately taken to reduce the impacts of disruption on the train operation. To deal with this real-time traffic management problem, a cooperative control approach is proposed in this paper. In this approach, the time to apply tractive force and the level of force are simultaneously adjusted for all the operated trains, to maximize the maintained line capacity when considering the power supply capacity. Compared with the existing train timetable rescheduling approach, cooperative control is more flexible to get a better train regulation solution. To solve the challenges for developing the cooperative control model (i.e., undetermined number and dynamically changing of controlled objects), an imaginary section method is newly developed to transform the original problem into an equivalent cooperative control problem with fixed controlled objects. Then, the mathematical models for the transformed problem are constructed by using the space–time–speed network methodology. According to the formulated model, a Decentralized-Markov Decision Process (Dec-MDP) framework is designed as the basis of the applied algorithm. Next, a Collaboration Mechanism Based-Independent Deep Q-Network (CMB-IDQN) algorithm is proposed to solve the cooperative control problem. Compared with classical IDQN algorithm, a credit assignment method based on the collaboration mechanism among trains is novelly considered in the designed multi-agent reinforcement learning algorithm. Finally, the effectiveness of the proposed cooperative control approach is verified by two case studies. When solving the cooperative control problem, the performance by using CMB-IDQN algorithm can be increased by up to 13.0% and 16.8% compared with other two classical reinforcement learning algorithms (i.e., DQN and IDQN), respectively. Compared with two train timetable rescheduling measures during the power supply shortage, the cooperative control approach can improve the solution quality by more than 180.4% and 17.4%, respectively.

Suggested Citation

  • Wang, Xuekai & D’Ariano, Andrea & Su, Shuai & Tang, Tao, 2023. "Cooperative train control during the power supply shortage in metro system: A multi-agent reinforcement learning approach," Transportation Research Part B: Methodological, Elsevier, vol. 170(C), pages 244-278.
    • Handle: RePEc:eee:transb:v:170:y:2023:i:c:p:244-278
    DOI: 10.1016/j.trb.2023.02.015
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0191261523000346
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.trb.2023.02.015?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Ardi Tampuu & Tambet Matiisen & Dorian Kodelja & Ilya Kuzovkin & Kristjan Korjus & Juhan Aru & Jaan Aru & Raul Vicente, 2017. "Multiagent cooperation and competition with deep reinforcement learning," PLOS ONE, Public Library of Science, vol. 12(4), pages 1-15, April.
    2. Huang, Yeran & Mannino, Carlo & Yang, Lixing & Tang, Tao, 2020. "Coupling time-indexed and big-M formulations for real-time train scheduling during metro service disruptions," Transportation Research Part B: Methodological, Elsevier, vol. 133(C), pages 38-61.
    3. Mo, Pengli & D’Ariano, Andrea & Yang, Lixing & Veelenturf, Lucas P. & Gao, Ziyou, 2021. "An exact method for the integrated optimization of subway lines operation strategies with asymmetric passenger demand and operating costs," Transportation Research Part B: Methodological, Elsevier, vol. 149(C), pages 283-321.
    4. Andreas Bärmann & Alexander Martin & Oskar Schneider, 2021. "Efficient Formulations and Decomposition Approaches for Power Peak Reduction in Railway Traffic via Timetabling," Transportation Science, INFORMS, vol. 55(3), pages 747-767, May.
    5. Wang, Xuekai & Tang, Tao & Su, Shuai & Yin, Jiateng & Gao, Ziyou & Lv, Nan, 2021. "An integrated energy-efficient train operation approach based on the space-time-speed network methodology," Transportation Research Part E: Logistics and Transportation Review, Elsevier, vol. 150(C).
    6. Nielsen, L.K. & Maróti, G., 2009. "Disruption Management of Rolling Stock in Passenger Railway Transportation," ERIM Report Series Research in Management ERS-2009-046-LIS, Erasmus Research Institute of Management (ERIM), ERIM is the joint research institute of the Rotterdam School of Management, Erasmus University and the Erasmus School of Economics (ESE) at Erasmus University Rotterdam.
    7. Yin, Jiateng & Tang, Tao & Yang, Lixing & Gao, Ziyou & Ran, Bin, 2016. "Energy-efficient metro train rescheduling with uncertain time-variant passenger demands: An approximate dynamic programming approach," Transportation Research Part B: Methodological, Elsevier, vol. 91(C), pages 178-210.
    8. Wang, Yihui & Zhao, Kangqi & D’Ariano, Andrea & Niu, Ru & Li, Shukai & Luan, Xiaojie, 2021. "Real-time integrated train rescheduling and rolling stock circulation planning for a metro line under disruptions," Transportation Research Part B: Methodological, Elsevier, vol. 152(C), pages 87-117.
    9. Sihui Long & Lingyun Meng & Yihui Wang & Jianrui Miao & Xuan Li, 2020. "A Discrete-Space Train Movement Model for a High-Speed Train under Temporary Speed Restriction," Mathematical Problems in Engineering, Hindawi, vol. 2020, pages 1-11, May.
    10. Zhu, Yongqiu & Goverde, Rob M.P., 2019. "Railway timetable rescheduling with flexible stopping and flexible short-turning during disruptions," Transportation Research Part B: Methodological, Elsevier, vol. 123(C), pages 149-181.
    11. Li, Wenqing & Ni, Shaoquan, 2022. "Train timetabling with the general learning environment and multi-agent deep reinforcement learning," Transportation Research Part B: Methodological, Elsevier, vol. 157(C), pages 230-251.
    12. Ying, Cheng-shuo & Chow, Andy H.F. & Chin, Kwai-Sang, 2020. "An actor-critic deep reinforcement learning approach for metro train scheduling with rolling stock circulation under stochastic demand," Transportation Research Part B: Methodological, Elsevier, vol. 140(C), pages 210-235.
    13. Wang, Dian & D’Ariano, Andrea & Zhao, Jun & Zhong, Qingwei & Peng, Qiyuan, 2022. "Integrated rolling stock deadhead routing and timetabling in urban rail transit lines," European Journal of Operational Research, Elsevier, vol. 298(2), pages 526-559.
    14. Li Wang & Wenting Mo & Yong Qin & Fei Dou & Limin Jia, 2014. "Optimization Based High-Speed Railway Train Rescheduling with Speed Restriction," Discrete Dynamics in Nature and Society, Hindawi, vol. 2014, pages 1-14, January.
    15. Šemrov, D. & Marsetič, R. & Žura, M. & Todorovski, L. & Srdic, A., 2016. "Reinforcement learning approach for train rescheduling on a single-track railway," Transportation Research Part B: Methodological, Elsevier, vol. 86(C), pages 250-267.
    16. Yin, Jiateng & Yang, Lixing & Tang, Tao & Gao, Ziyou & Ran, Bin, 2017. "Dynamic passenger demand oriented metro train scheduling with energy-efficiency and waiting time minimization: Mixed-integer linear programming approaches," Transportation Research Part B: Methodological, Elsevier, vol. 97(C), pages 182-213.
    17. Volodymyr Mnih & Koray Kavukcuoglu & David Silver & Andrei A. Rusu & Joel Veness & Marc G. Bellemare & Alex Graves & Martin Riedmiller & Andreas K. Fidjeland & Georg Ostrovski & Stig Petersen & Charle, 2015. "Human-level control through deep reinforcement learning," Nature, Nature, vol. 518(7540), pages 529-533, February.
    18. Chen, Zebin & Li, Shukai & D’Ariano, Andrea & Yang, Lixing, 2022. "Real-time optimization for train regulation and stop-skipping adjustment strategy of urban rail transit lines," Omega, Elsevier, vol. 110(C).
    19. Meng, Lingyun & Zhou, Xuesong, 2011. "Robust single-track train dispatching model under a dynamic and stochastic environment: A scenario-based rolling horizon solution approach," Transportation Research Part B: Methodological, Elsevier, vol. 45(7), pages 1080-1102, August.
    20. Xu, Peijuan & Corman, Francesco & Peng, Qiyuan & Luan, Xiaojie, 2017. "A train rescheduling model integrating speed management during disruptions of high-speed traffic under a quasi-moving block system," Transportation Research Part B: Methodological, Elsevier, vol. 104(C), pages 638-666.
    21. Yin, Jiateng & Ren, Xianliang & Liu, Ronghui & Tang, Tao & Su, Shuai, 2022. "Quantitative analysis for resilience-based urban rail systems: A hybrid knowledge-based and data-driven approach," Reliability Engineering and System Safety, Elsevier, vol. 219(C).
    22. Zhan, Shuguang & Kroon, Leo G. & Veelenturf, Lucas P. & Wagenaar, Joris C., 2015. "Real-time high-speed train rescheduling in case of a complete blockage," Transportation Research Part B: Methodological, Elsevier, vol. 78(C), pages 182-201.
    23. Scheepmaker, Gerben M. & Goverde, Rob M.P. & Kroon, Leo G., 2017. "Review of energy-efficient train control and timetabling," European Journal of Operational Research, Elsevier, vol. 257(2), pages 355-376.
    24. Zhou, Leishan & Tong, Lu (Carol) & Chen, Junhua & Tang, Jinjin & Zhou, Xuesong, 2017. "Joint optimization of high-speed train timetables and speed profiles: A unified modeling approach using space-time-speed grid networks," Transportation Research Part B: Methodological, Elsevier, vol. 97(C), pages 157-181.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Wang, Yihui & Zhao, Kangqi & D’Ariano, Andrea & Niu, Ru & Li, Shukai & Luan, Xiaojie, 2021. "Real-time integrated train rescheduling and rolling stock circulation planning for a metro line under disruptions," Transportation Research Part B: Methodological, Elsevier, vol. 152(C), pages 87-117.
    2. Lian, Deheng & Mo, Pengli & D’Ariano, Andrea & Gao, Ziyou & Yang, Lixing, 2024. "Energy-saving time allocation strategy with uncertain dwell times in urban rail transit: Two-stage stochastic model and nested dynamic programming framework," European Journal of Operational Research, Elsevier, vol. 317(1), pages 219-242.
    3. Zhan, Shuguang & Wong, S.C. & Shang, Pan & Peng, Qiyuan & Xie, Jiemin & Lo, S.M., 2021. "Integrated railway timetable rescheduling and dynamic passenger routing during a complete blockage," Transportation Research Part B: Methodological, Elsevier, vol. 143(C), pages 86-123.
    4. Wang, Entai & Yang, Lixing & Yin, Jiateng & Zhang, Jinlei & Gao, Ziyou, 2024. "Passenger-oriented rolling stock scheduling in the metro system with multiple depots: Network flow based approaches," Transportation Research Part B: Methodological, Elsevier, vol. 180(C).
    5. Yang, Lin & Gao, Yuan & D’Ariano, Andrea & Xu, Suxiu, 2024. "Integrated optimization of train timetable and train unit circulation for a Y-type urban rail transit system with flexible train composition mode," Omega, Elsevier, vol. 122(C).
    6. Yuan, Yin & Li, Shukai & Yang, Lixing & Gao, Ziyou, 2022. "Real-time optimization of train regulation and passenger flow control for urban rail transit network under frequent disturbances," Transportation Research Part E: Logistics and Transportation Review, Elsevier, vol. 168(C).
    7. Mo, Pengli & D’Ariano, Andrea & Yang, Lixing & Veelenturf, Lucas P. & Gao, Ziyou, 2021. "An exact method for the integrated optimization of subway lines operation strategies with asymmetric passenger demand and operating costs," Transportation Research Part B: Methodological, Elsevier, vol. 149(C), pages 283-321.
    8. Chai, Simin & Yin, Jiateng & D’Ariano, Andrea & Liu, Ronghui & Yang, Lixing & Tang, Tao, 2024. "A branch-and-cut algorithm for scheduling train platoons in urban rail networks," Transportation Research Part B: Methodological, Elsevier, vol. 181(C).
    9. Bešinović, Nikola & Ferrari Nassar, Raphael & Szymula, Christopher, 2022. "Resilience assessment of railway networks: Combining infrastructure restoration and transport management," Reliability Engineering and System Safety, Elsevier, vol. 224(C).
    10. Zhan, Shuguang & Xie, Jiemin & Wong, S.C. & Zhu, Yongqiu & Corman, Francesco, 2024. "Handling uncertainty in train timetable rescheduling: A review of the literature and future research directions," Transportation Research Part E: Logistics and Transportation Review, Elsevier, vol. 183(C).
    11. Chen, Zebin & Li, Shukai & D’Ariano, Andrea & Yang, Lixing, 2022. "Real-time optimization for train regulation and stop-skipping adjustment strategy of urban rail transit lines," Omega, Elsevier, vol. 110(C).
    12. Zhang, Yongxiang & D'Ariano, Andrea & He, Bisheng & Peng, Qiyuan, 2019. "Microscopic optimization model and algorithm for integrating train timetabling and track maintenance task scheduling," Transportation Research Part B: Methodological, Elsevier, vol. 127(C), pages 237-278.
    13. Li, Wenqing & Ni, Shaoquan, 2022. "Train timetabling with the general learning environment and multi-agent deep reinforcement learning," Transportation Research Part B: Methodological, Elsevier, vol. 157(C), pages 230-251.
    14. Zhang, Chuntian & Gao, Yuan & Cacchiani, Valentina & Yang, Lixing & Gao, Ziyou, 2023. "Train rescheduling for large-scale disruptions in a large-scale railway network," Transportation Research Part B: Methodological, Elsevier, vol. 174(C).
    15. Yuan, Jiawei & Gao, Yuan & Li, Shukai & Liu, Pei & Yang, Lixing, 2022. "Integrated optimization of train timetable, rolling stock assignment and short-turning strategy for a metro line," European Journal of Operational Research, Elsevier, vol. 301(3), pages 855-874.
    16. Wang, Pengling & Goverde, Rob M.P., 2017. "Multi-train trajectory optimization for energy efficiency and delay recovery on single-track railway lines," Transportation Research Part B: Methodological, Elsevier, vol. 105(C), pages 340-361.
    17. Pan, Hanchuan & Yang, Lixing & Liang, Zhe, 2023. "Demand-oriented integration optimization of train timetabling and rolling stock circulation planning with flexible train compositions: A column-generation-based approach," European Journal of Operational Research, Elsevier, vol. 305(1), pages 184-206.
    18. Wenxing Wu & Jing Xun & Jiateng Yin & Shibo He & Haifeng Song & Zicong Zhao & Shicong Hao, 2023. "An Integrated Method for Reducing Arrival Interval by Optimizing Train Operation and Route Setting," Mathematics, MDPI, vol. 11(20), pages 1-20, October.
    19. Canca, David & Zarzo, Alejandro, 2017. "Design of energy-Efficient timetables in two-way railway rapid transit lines," Transportation Research Part B: Methodological, Elsevier, vol. 102(C), pages 142-161.
    20. Zhan, Shuguang & Wang, Pengling & Wong, S.C. & Lo, S.M., 2022. "Energy-efficient high-speed train rescheduling during a major disruption," Transportation Research Part E: Logistics and Transportation Review, Elsevier, vol. 157(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:transb:v:170:y:2023:i:c:p:244-278. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/wps/find/journaldescription.cws_home/548/description#description .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.