IDEAS home Printed from https://ideas.repec.org/a/gam/jmathe/v11y2023i6p1556-d1104571.html
   My bibliography  Save this article

Reinforcement Learning-Based Lane Change Decision for CAVs in Mixed Traffic Flow under Low Visibility Conditions

Author

Listed:
  • Bowen Gong

    (Department of Traffic Information and Control Engineering, Jilin University, Changchun 130022, China)

  • Zhipeng Xu

    (Department of Traffic Information and Control Engineering, Jilin University, Changchun 130022, China)

  • Ruixin Wei

    (Department of Traffic Information and Control Engineering, Jilin University, Changchun 130022, China)

  • Tao Wang

    (China Academy of Transportation Sciences, Beijing 100029, China)

  • Ciyun Lin

    (Department of Traffic Information and Control Engineering, Jilin University, Changchun 130022, China
    Jilin Engineering Research Center for Intelligent Transportation System, Changchun 130022, China)

  • Peng Gao

    (Qingdao Transportation Public Service Center, Qingdao Municipal Transport Bureau, Qingdao 266061, China)

Abstract

As an important stage in the development of autonomous driving, mixed traffic conditions, consisting of connected autonomous vehicles (CAVs) and human-driven vehicles (HDVs), have attracted more and more attention. In fact, the randomness of human-driven vehicles (HDV) is the largest challenge for connected autonomous vehicles (CAV) to make reasonable decisions, especially in lane change scenarios. In this paper, we propose the problem of lane change decisions for CAV in low visibility and mixed traffic conditions for the first time. First, we consider the randomness of HDV in this environment and construct a finite state machine (FSM) model. Then, this study develops a partially observed Markov decision process (POMDP) for describing the problem of lane change. In addition, we use the modified deep deterministic policy gradient (DDPG) to solve the problem and get the optimal lane change decision in this environment. The reward designing takes the comfort, safety and efficiency of the vehicle into account, and the introduction of transfer learning accelerates the adaptation of CAV to the randomness of HDV. Finally, numerical experiments are conducted. The results show that, compared with the original DDPG, the modified DDPG has a faster convergence velocity. The strategy learned by the modified DDPG can complete the lane change in most of the scenarios. The comparison between the modified DDPG and the rule-based decisions indicates that the modified DDPG has a stronger adaptability to this special environment and can grasp more lane change opportunities.

Suggested Citation

  • Bowen Gong & Zhipeng Xu & Ruixin Wei & Tao Wang & Ciyun Lin & Peng Gao, 2023. "Reinforcement Learning-Based Lane Change Decision for CAVs in Mixed Traffic Flow under Low Visibility Conditions," Mathematics, MDPI, vol. 11(6), pages 1-24, March.
    • Handle: RePEc:gam:jmathe:v:11:y:2023:i:6:p:1556-:d:1104571
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/2227-7390/11/6/1556/pdf
    Download Restriction: no
    File URL: https://www.mdpi.com/2227-7390/11/6/1556/
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Sheikh, Muhammad Sameer & Wang, Ji & Regan, Amelia, 2021. "A game theory-based controller approach for identifying incidents caused by aberrant lane changing behavior," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 580(C).
    2. Gipps, P. G., 1986. "A model for the structure of lane-changing decisions," Transportation Research Part B: Methodological, Elsevier, vol. 20(5), pages 403-414, October.
    3. Kexuan Lv & Xiaofei Pei & Ci Chen & Jie Xu, 2022. "A Safe and Efficient Lane Change Decision-Making Strategy of Autonomous Driving Based on Deep Reinforcement Learning," Mathematics, MDPI, vol. 10(9), pages 1-24, May.
    4. Zheng, Zuduo, 2014. "Recent developments and research needs in modeling lane changing," Transportation Research Part B: Methodological, Elsevier, vol. 60(C), pages 16-32.
    5. Deng, Jian-Hua & Feng, Huan-Huan, 2019. "A multilane cellular automaton multi-attribute lane-changing decision model," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 529(C).
    6. Rickert, M. & Nagel, K. & Schreckenberg, M. & Latour, A., 1996. "Two lane traffic simulations using cellular automata," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 231(4), pages 534-550.
    7. Xu, Ting & Zhang, Zhishun & Wu, Xingqi & Qi, Long & Han, Yi, 2021. "Recognition of lane-changing behaviour with machine learning methods at freeway off-ramps," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 567(C).
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Yanzhan Chen & Fan Yu, 2023. "A Novel Simulation-Based Optimization Method for Autonomous Vehicle Path Tracking with Urban Driving Application," Mathematics, MDPI, vol. 11(23), pages 1-30, November.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Li, Gen & Zhao, Le & Tang, Wenyun & Wu, Lan & Ren, Jiaolong, 2023. "Modeling and analysis of mandatory lane-changing behavior considering heterogeneity in means and variances," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 622(C).
    2. Khelfa, Basma & Ba, Ibrahima & Tordeux, Antoine, 2023. "Predicting highway lane-changing maneuvers: A benchmark analysis of machine and ensemble learning algorithms," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 612(C).
    3. Ma, Changxi & Li, Dong, 2023. "A review of vehicle lane change research," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 626(C).
    4. Wang, Jinghui & Lv, Wei & Jiang, Yajuan & Qin, Shuangshuang & Li, Jiawei, 2021. "A multi-agent based cellular automata model for intersection traffic control simulation," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 584(C).
    5. Kai Nagel & Peter Wagner & Richard Woesler, 2003. "Still Flowing: Approaches to Traffic Flow and Traffic Jam Modeling," Operations Research, INFORMS, vol. 51(5), pages 681-710, October.
    6. Espadaler-Clapés, Jasso & Barmpounakis, Emmanouil & Geroliminis, Nikolas, 2023. "Empirical investigation of lane usage, lane changing and lane choice phenomena in a multimodal urban arterial," Transportation Research Part A: Policy and Practice, Elsevier, vol. 172(C).
    7. Ma, Yanli & Lv, Zhiliang & Zhang, Peng & Chan, Ching-Yao, 2021. "Impact of lane changing on adjacent vehicles considering multi-vehicle interaction in mixed traffic flow: A velocity estimating model," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 566(C).
    8. Zheng, Zuduo, 2014. "Recent developments and research needs in modeling lane changing," Transportation Research Part B: Methodological, Elsevier, vol. 60(C), pages 16-32.
    9. Li, Linheng & Gan, Jing & Zhou, Kun & Qu, Xu & Ran, Bin, 2020. "A novel lane-changing model of connected and automated vehicles: Using the safety potential field theory," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 559(C).
    10. Sheikh, Muhammad Sameer & Wang, Ji & Regan, Amelia, 2021. "A game theory-based controller approach for identifying incidents caused by aberrant lane changing behavior," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 580(C).
    11. Kuang, Xianyan & Chen, Ziru, 2022. "Trajectory research of Cellular Automaton Model based on real driving behaviour," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 602(C).
    12. Jiang, Yangsheng & Tan, Li & Xiao, Guosheng & Wu, Yunxia & Yao, Zhihong, 2024. "Platoon-aware cooperative lane-changing strategy for connected automated vehicles in mixed traffic flow," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 640(C).
    13. Shanchuan Yu & Yu Chen & Lang Song & Zhaoze Xuan & Yi Li, 2023. "Modelling and Mitigating Secondary Crash Risk for Serial Tunnels on Freeway via Lighting-Related Microscopic Traffic Model with Inter-Lane Dependency," IJERPH, MDPI, vol. 20(4), pages 1-29, February.
    14. Ji Ang & David Levinson, 2020. "A Review of Game Theory Models of Lane Changing," Working Papers 2022-01, University of Minnesota: Nexus Research Group.
    15. Lv, Wei & Song, Wei-guo & Fang, Zhi-ming, 2011. "Three-lane changing behaviour simulation using a modified optimal velocity model," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 390(12), pages 2303-2314.
    16. Shang, Xue-Cheng & Li, Xin-Gang & Xie, Dong-Fan & Jia, Bin & Jiang, Rui, 2020. "Two-lane traffic flow model based on regular hexagonal cells with realistic lane changing behavior," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 560(C).
    17. Mehr, Negar & Li, Ruolin & Horowitz, Roberto, 2021. "A game theoretic macroscopic model of lane choices at traffic diverges with applications to mixed–autonomy networks," Transportation Research Part B: Methodological, Elsevier, vol. 144(C), pages 45-59.
    18. Minh Sang Pham Do & Ketoma Vix Kemanji & Man Dinh Vinh Nguyen & Tuan Anh Vu & Gerrit Meixner, 2023. "The Action Point Angle of Sight: A Traffic Generation Method for Driving Simulation, as a Small Step to Safe, Sustainable and Smart Cities," Sustainability, MDPI, vol. 15(12), pages 1-27, June.
    19. Zhao, Jing & Knoop, Victor L. & Wang, Meng, 2020. "Two-dimensional vehicular movement modelling at intersections based on optimal control," Transportation Research Part B: Methodological, Elsevier, vol. 138(C), pages 1-22.
    20. Xue Wang & Yu Xue & Suwei Feng, 2023. "Traffic fuel consumption evaluation of the on-ramp with acceleration lane based on cellular automata," The European Physical Journal B: Condensed Matter and Complex Systems, Springer;EDP Sciences, vol. 96(6), pages 1-11, June.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jmathe:v:11:y:2023:i:6:p:1556-:d:1104571. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager (email available below). General contact details of provider: https://www.mdpi.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.