A Dynamic Holding Approach to Stabilizing a Bus Line Based on the Q-Learning Algorithm with Multistage Look-Ahead

The unreliable service and the unstable operation of a high-frequency bus line are shown as bus bunching and the uneven distribution of headways along the bus line. Although many control strategies, such as the static and dynamic holding strategies, have been proposed to solve the above problems, many of them take on some oversimplified assumptions about the real bus line operation. So it is hard for them to continuously adapt to the evolving complex system. In view of this dynamic setting, we present an adaptive holding method that combines the classic approximate dynamic programming (ADP) with the multistage look-ahead mechanism. The holding time, the only control means used in this study, will be determined by estimating its impact on the operation stability of the bus line system in the remaining observation period. The multistage look-ahead mechanism introduced into the classic Q-learning algorithm of the ADP model makes it easy that the algorithm gets through its earlier unstable phase more quickly and easily. During the implementation of the new holding approach, the past experiences of holding operations can be cumulated effectively into an artificial neural network used to approximate the unavailable Q-factor. The use of a detailed simulation system in the new approach makes it possible to take into account most of the possible causes of instability. The numerical experiments show that the new holding approach can stabilize the system by producing evenly distributed headway and removing bus bunching thoroughly. Compared with the terminal station holding strategies, the new method brings a more reliable bus line with shorter waiting times for passengers.