IDEAS home Printed from https://ideas.repec.org/a/eee/transb/v157y2022icp230-251.html
   My bibliography  Save this article

Train timetabling with the general learning environment and multi-agent deep reinforcement learning

Author

Listed:
  • Li, Wenqing
  • Ni, Shaoquan

Abstract

This paper proposes a multi-agent deep reinforcement learning approach for the train timetabling problem of different railway systems. A general train timetabling learning environment is constructed to model the problem as a Markov decision process, in which the objectives and complex constraints of the problem can be distributed naturally and elegantly. Through subtle changes, the environment can be flexibly switched between the widely used double-track railway system and the more complex single-track railway system. To address the curse of dimensionality, a multi-agent actor–critic algorithm framework is proposed to decompose the large-size combinatorial decision space into multiple independent ones, which are parameterized by deep neural networks. The proposed approach was tested using a real-world instance and several test instances. Experimental results show that cooperative policies of the single-track train timetabling problem can be obtained by the proposed method within a reasonable computing time that outperforms several prevailing methods in terms of the optimality of solutions, and the proposed method can be easily generalized to the double-track train timetabling problem by changing the environment slightly.

Suggested Citation

  • Li, Wenqing & Ni, Shaoquan, 2022. "Train timetabling with the general learning environment and multi-agent deep reinforcement learning," Transportation Research Part B: Methodological, Elsevier, vol. 157(C), pages 230-251.
  • Handle: RePEc:eee:transb:v:157:y:2022:i:c:p:230-251
    DOI: 10.1016/j.trb.2022.02.006
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S019126152200025X
    Download Restriction: Full text for ScienceDirect subscribers only

    File URL: https://libkey.io/10.1016/j.trb.2022.02.006?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Oriol Vinyals & Igor Babuschkin & Wojciech M. Czarnecki & Michaël Mathieu & Andrew Dudzik & Junyoung Chung & David H. Choi & Richard Powell & Timo Ewalds & Petko Georgiev & Junhyuk Oh & Dan Horgan & M, 2019. "Grandmaster level in StarCraft II using multi-agent reinforcement learning," Nature, Nature, vol. 575(7782), pages 350-354, November.
    2. Zhou, Xuesong & Zhong, Ming, 2007. "Single-track train timetabling with guaranteed optimality: Branch-and-bound algorithms with enhanced lower bounds," Transportation Research Part B: Methodological, Elsevier, vol. 41(3), pages 320-341, March.
    3. Carey, Malachy & Crawford, Ivan, 2007. "Scheduling trains on a network of busy complex stations," Transportation Research Part B: Methodological, Elsevier, vol. 41(2), pages 159-178, February.
    4. Guo, Xin & Sun, Huijun & Wu, Jianjun & Jin, Jiangang & Zhou, Jin & Gao, Ziyou, 2017. "Multiperiod-based timetable optimization for metro transit networks," Transportation Research Part B: Methodological, Elsevier, vol. 96(C), pages 46-67.
    5. Chow, Andy H.F. & Pavlides, Aris, 2018. "Cost functions and multi-objective timetabling of mixed train services," Transportation Research Part A: Policy and Practice, Elsevier, vol. 113(C), pages 335-356.
    6. U. Brännlund & P. O. Lindberg & A. Nõu & J.-E. Nilsson, 1998. "Railway Timetabling Using Lagrangian Relaxation," Transportation Science, INFORMS, vol. 32(4), pages 358-369, November.
    7. Yin, Jiateng & Tang, Tao & Yang, Lixing & Gao, Ziyou & Ran, Bin, 2016. "Energy-efficient metro train rescheduling with uncertain time-variant passenger demands: An approximate dynamic programming approach," Transportation Research Part B: Methodological, Elsevier, vol. 91(C), pages 178-210.
    8. Cacchiani, Valentina & Caprara, Alberto & Toth, Paolo, 2010. "Scheduling extra freight trains on railway networks," Transportation Research Part B: Methodological, Elsevier, vol. 44(2), pages 215-231, February.
    9. Dorfman, M. J. & Medanic, J., 2004. "Scheduling trains on a railway network using a discrete event model of railway traffic," Transportation Research Part B: Methodological, Elsevier, vol. 38(1), pages 81-98, January.
    10. Julian Schrittwieser & Ioannis Antonoglou & Thomas Hubert & Karen Simonyan & Laurent Sifre & Simon Schmitt & Arthur Guez & Edward Lockhart & Demis Hassabis & Thore Graepel & Timothy Lillicrap & David , 2020. "Mastering Atari, Go, chess and shogi by planning with a learned model," Nature, Nature, vol. 588(7839), pages 604-609, December.
    11. Ying, Cheng-shuo & Chow, Andy H.F. & Chin, Kwai-Sang, 2020. "An actor-critic deep reinforcement learning approach for metro train scheduling with rolling stock circulation under stochastic demand," Transportation Research Part B: Methodological, Elsevier, vol. 140(C), pages 210-235.
    12. Alberto Caprara & Matteo Fischetti & Paolo Toth, 2002. "Modeling and Solving the Train Timetabling Problem," Operations Research, INFORMS, vol. 50(5), pages 851-861, October.
    13. Cacchiani, Valentina & Toth, Paolo, 2012. "Nominal and robust train timetabling problems," European Journal of Operational Research, Elsevier, vol. 219(3), pages 727-737.
    14. Mor Kaspi & Tal Raviv, 2013. "Service-Oriented Line Planning and Timetabling for Passenger Trains," Transportation Science, INFORMS, vol. 47(3), pages 295-311, August.
    15. David Silver & Aja Huang & Chris J. Maddison & Arthur Guez & Laurent Sifre & George van den Driessche & Julian Schrittwieser & Ioannis Antonoglou & Veda Panneershelvam & Marc Lanctot & Sander Dieleman, 2016. "Mastering the game of Go with deep neural networks and tree search," Nature, Nature, vol. 529(7587), pages 484-489, January.
    16. Šemrov, D. & Marsetič, R. & Žura, M. & Todorovski, L. & Srdic, A., 2016. "Reinforcement learning approach for train rescheduling on a single-track railway," Transportation Research Part B: Methodological, Elsevier, vol. 86(C), pages 250-267.
    17. David Silver & Julian Schrittwieser & Karen Simonyan & Ioannis Antonoglou & Aja Huang & Arthur Guez & Thomas Hubert & Lucas Baker & Matthew Lai & Adrian Bolton & Yutian Chen & Timothy Lillicrap & Fan , 2017. "Mastering the game of Go without human knowledge," Nature, Nature, vol. 550(7676), pages 354-359, October.
    18. Jiang, Feng & Cacchiani, Valentina & Toth, Paolo, 2017. "Train timetabling by skip-stop planning in highly congested lines," Transportation Research Part B: Methodological, Elsevier, vol. 104(C), pages 149-174.
    19. Volodymyr Mnih & Koray Kavukcuoglu & David Silver & Andrei A. Rusu & Joel Veness & Marc G. Bellemare & Alex Graves & Martin Riedmiller & Andreas K. Fidjeland & Georg Ostrovski & Stig Petersen & Charle, 2015. "Human-level control through deep reinforcement learning," Nature, Nature, vol. 518(7540), pages 529-533, February.
    20. Yang, Xin & Chen, Anthony & Ning, Bin & Tang, Tao, 2017. "Bi-objective programming approach for solving the metro timetable optimization problem with dwell time uncertainty," Transportation Research Part E: Logistics and Transportation Review, Elsevier, vol. 97(C), pages 22-37.
    21. Cacchiani, Valentina & Furini, Fabio & Kidd, Martin Philip, 2016. "Approaches to a real-world Train Timetabling Problem in a railway node," Omega, Elsevier, vol. 58(C), pages 97-110.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Wang, Xuekai & D’Ariano, Andrea & Su, Shuai & Tang, Tao, 2023. "Cooperative train control during the power supply shortage in metro system: A multi-agent reinforcement learning approach," Transportation Research Part B: Methodological, Elsevier, vol. 170(C), pages 244-278.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Zhang, Chuntian & Gao, Yuan & Yang, Lixing & Gao, Ziyou & Qi, Jianguo, 2020. "Joint optimization of train scheduling and maintenance planning in a railway network: A heuristic algorithm using Lagrangian relaxation," Transportation Research Part B: Methodological, Elsevier, vol. 134(C), pages 64-92.
    2. Zhang, Yongxiang & Peng, Qiyuan & Yao, Yu & Zhang, Xin & Zhou, Xuesong, 2019. "Solving cyclic train timetabling problem through model reformulation: Extended time-space network construct and Alternating Direction Method of Multipliers methods," Transportation Research Part B: Methodological, Elsevier, vol. 128(C), pages 344-379.
    3. Yin, Jiateng & Yang, Lixing & Tang, Tao & Gao, Ziyou & Ran, Bin, 2017. "Dynamic passenger demand oriented metro train scheduling with energy-efficiency and waiting time minimization: Mixed-integer linear programming approaches," Transportation Research Part B: Methodological, Elsevier, vol. 97(C), pages 182-213.
    4. Zhou, Wenliang & Teng, Hualiang, 2016. "Simultaneous passenger train routing and timetabling using an efficient train-based Lagrangian relaxation decomposition," Transportation Research Part B: Methodological, Elsevier, vol. 94(C), pages 409-439.
    5. Meng, Lingyun & Zhou, Xuesong, 2014. "Simultaneous train rerouting and rescheduling on an N-track network: A model reformulation with network-based cumulative flow variables," Transportation Research Part B: Methodological, Elsevier, vol. 67(C), pages 208-234.
    6. Tian, Xiaopeng & Niu, Huimin, 2020. "Optimization of demand-oriented train timetables under overtaking operations: A surrogate-dual-variable column generation for eliminating indivisibility," Transportation Research Part B: Methodological, Elsevier, vol. 142(C), pages 143-173.
    7. Shi, Jungang & Yang, Lixing & Yang, Jing & Gao, Ziyou, 2018. "Service-oriented train timetabling with collaborative passenger flow control on an oversaturated metro line: An integer linear optimization approach," Transportation Research Part B: Methodological, Elsevier, vol. 110(C), pages 26-59.
    8. Xiaoming Xu & Keping Li & Lixing Yang & Ziyou Gao, 2019. "An efficient train scheduling algorithm on a single-track railway system," Journal of Scheduling, Springer, vol. 22(1), pages 85-105, February.
    9. Gao, Yuan & Kroon, Leo & Yang, Lixing & Gao, Ziyou, 2018. "Three-stage optimization method for the problem of scheduling additional trains on a high-speed rail corridor," Omega, Elsevier, vol. 80(C), pages 175-191.
    10. Liang, Jinpeng & Zang, Guangzhi & Liu, Haitao & Zheng, Jianfeng & Gao, Ziyou, 2023. "Reducing passenger waiting time in oversaturated metro lines with passenger flow control policy," Omega, Elsevier, vol. 117(C).
    11. Sairong Peng & Xin Yang & Hongwei Wang & Hairong Dong & Bin Ning & Haichuan Tang & Zhipeng Ying & Ruijun Tang, 2019. "Dispatching High-Speed Rail Trains via Utilizing the Reverse Direction Track: Adaptive Rescheduling Strategies and Application," Sustainability, MDPI, vol. 11(8), pages 1-20, April.
    12. Yu-Jun Zheng, 2018. "Emergency Train Scheduling on Chinese High-Speed Railways," Transportation Science, INFORMS, vol. 52(5), pages 1077-1091, October.
    13. E. Ursavas & Stuart X. Zhu, 2018. "Integrated Passenger and Freight Train Planning on Shared-Use Corridors," Service Science, INFORMS, vol. 52(6), pages 1376-1390, December.
    14. Jiang, Feng & Cacchiani, Valentina & Toth, Paolo, 2017. "Train timetabling by skip-stop planning in highly congested lines," Transportation Research Part B: Methodological, Elsevier, vol. 104(C), pages 149-174.
    15. Xu, Xiaoming & Li, Keping & Yang, Lixing, 2015. "Scheduling heterogeneous train traffic on double tracks with efficient dispatching rules," Transportation Research Part B: Methodological, Elsevier, vol. 78(C), pages 364-384.
    16. Kang, Liujiang & Meng, Qiang, 2017. "Two-phase decomposition method for the last train departure time choice in subway networks," Transportation Research Part B: Methodological, Elsevier, vol. 104(C), pages 568-582.
    17. Zhang, Di & Gao, Yuan & Yang, Lixing & Cui, Lixin, 2024. "Timetable synchronization of the last several trains at night in an urban rail transit network," European Journal of Operational Research, Elsevier, vol. 313(2), pages 494-512.
    18. Ying, Cheng-shuo & Chow, Andy H.F. & Nguyen, Hoa T.M. & Chin, Kwai-Sang, 2022. "Multi-agent deep reinforcement learning for adaptive coordinated metro service operations with flexible train composition," Transportation Research Part B: Methodological, Elsevier, vol. 161(C), pages 36-59.
    19. Wenliang Zhou & Junli Tian & Jin Qin & Lianbo Deng & TangJian Wei, 2015. "Optimization of Multiperiod Mixed Train Schedule on High-Speed Railway," Discrete Dynamics in Nature and Society, Hindawi, vol. 2015, pages 1-14, April.
    20. Yin, Jiateng & Tang, Tao & Yang, Lixing & Gao, Ziyou & Ran, Bin, 2016. "Energy-efficient metro train rescheduling with uncertain time-variant passenger demands: An approximate dynamic programming approach," Transportation Research Part B: Methodological, Elsevier, vol. 91(C), pages 178-210.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:transb:v:157:y:2022:i:c:p:230-251. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/wps/find/journaldescription.cws_home/548/description#description .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.