IDEAS home Printed from https://ideas.repec.org/a/eee/chsofr/v178y2024ics0960077923012602.html
   My bibliography  Save this article

Exploring cooperative evolution with tunable payoff’s loners using reinforcement learning

Author

Listed:
  • Zhang, Huizhen
  • An, Tianbo
  • Yan, Pingping
  • Hu, Kaipeng
  • An, Jinjin
  • Shi, Lijuan
  • Zhao, Jian
  • Wang, Jingrui

Abstract

Imitation and replication have emerged as a paradigm in numerous studies that explore the evolution of cooperative behavior. Since they embrace the essence of natural selection, it is widely recognized in exploring the evolution of biological behaviors. However, it is not easy to express the way individuals select and optimize in these simple and elegant ways in the complex and variable interactive environments. Currently, reinforcement learning is widely used in the study of strategy updating dynamics and agent learning processes in game theory. Therefore, we introduce the Q-learning algorithms into the voluntary public goods game to explore the impact of cooperative evolution. Simulation results demonstrate that when the synergy factor is large and the adjust loner payoff’s multiply factor is smaller, the number of cooperators becomes gradually consistent. As the synergy factor increases, the evolution of the proportion of defectors become nonlinear. Furthermore, we further explore the Q-table and strategy updating processes of agents in the steady state under a smaller multiply factor that adjusts the loner’s payoff. The results find inconsistency between the average Q-values and the steady state population strategy distribution. Subsequently, we explain the reason for the inconsistency by analyzing strategy sequence, namely that there are a number of agents who constantly change strategies in the population, and the Q-values of these agents have an impact on the overall Q-values. In addition, evolutionary snapshots of agent strategy sequences are observed. The results find that the agent’s strategic selection shows greater instability when the proportions of cooperators, defectors, and loners in the population are relatively balanced. Finally, the effect of parameters in the Q-learning algorithm on cooperative behavior is analyzed. This study hopes to provide valuable insights into understanding the dynamics of cooperation in complex social interactions.

Suggested Citation

  • Zhang, Huizhen & An, Tianbo & Yan, Pingping & Hu, Kaipeng & An, Jinjin & Shi, Lijuan & Zhao, Jian & Wang, Jingrui, 2024. "Exploring cooperative evolution with tunable payoff’s loners using reinforcement learning," Chaos, Solitons & Fractals, Elsevier, vol. 178(C).
    • Handle: RePEc:eee:chsofr:v:178:y:2024:i:c:s0960077923012602
    DOI: 10.1016/j.chaos.2023.114358
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0960077923012602
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.chaos.2023.114358?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Cui, Guang-Hai & Wang, Zhen & Li, Jun-Li & Jin, Xing & Zhang, Zhi-Wang, 2021. "Influence of precaution and dynamic post-indemnity based insurance policy on controlling the propagation of epidemic security risks in networks," Applied Mathematics and Computation, Elsevier, vol. 392(C).
    2. Gabriela Koľveková & Manuela Raisová & Martin Zoričak & Vladimír Gazda, 2021. "Endogenous Shared Punishment Model in Threshold Public Goods Games," Computational Economics, Springer;Society for Computational Economics, vol. 58(1), pages 57-81, June.
    3. Pan, Jianchen & Zhang, Lan & Han, Wenchen & Huang, Changwei, 2023. "Heterogeneous investment promotes cooperation in spatial public goods game on hypergraphs," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 609(C).
    4. Jingrui Wang & Xing Jin & Yixuan Yang & Qingfang Chen & Zhen Wang & Hong Ding, 2021. "The spread of epidemic under voluntary vaccination with heterogeneous infection rates," International Journal of Modern Physics C (IJMPC), World Scientific Publishing Co. Pte. Ltd., vol. 32(03), pages 1-14, March.
    5. Wang, Jingrui & Zhang, Huizhen & Jin, Xing & Ma, Leyu & Chen, Yueren & Wang, Chao & Zhao, Jian & An, Tianbo, 2023. "Subsidy policy with punishment mechanism can promote voluntary vaccination behaviors in structured populations," Chaos, Solitons & Fractals, Elsevier, vol. 174(C).
    6. Yang, Zhengzhi & Zheng, Lei & Perc, Matjaž & Li, Yumeng, 2024. "Interaction state Q-learning promotes cooperation in the spatial prisoner's dilemma game," Applied Mathematics and Computation, Elsevier, vol. 463(C).
    7. Huang, Changwei & Hou, Yongzhao & Han, Wenchen, 2023. "Coevolution of consensus and cooperation in evolutionary Hegselmann–Krause dilemma with the cooperation cost," Chaos, Solitons & Fractals, Elsevier, vol. 168(C).
    8. Jia, Danyang & Li, Tong & Zhao, Yang & Zhang, Xiaoqin & Wang, Zhen, 2022. "Empty nodes affect conditional cooperation under reinforcement learning," Applied Mathematics and Computation, Elsevier, vol. 413(C).
    9. Jin, Xing & Tao, Yuchen & Wang, Jingrui & Wang, Chao & Wang, Yongheng & Zhang, Zhouyang & Wang, Zhen, 2023. "Strategic use of payoff information in k-hop evolutionary Best-shot networked public goods game," Applied Mathematics and Computation, Elsevier, vol. 459(C).
    10. Gao, Bo & Liu, Xuan & Lan, Zhong-Zhou & Hong, Jie & Zhang, Wenguang, 2021. "The evolution of cooperation with preferential selection in voluntary public goods game," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 584(C).
    11. Quan, Ji & Yu, Junyu & Li, Xia & Wang, Xianjia, 2023. "Conditional switching between social excluders and loners promotes cooperation in spatial public goods game," Chaos, Solitons & Fractals, Elsevier, vol. 169(C).
    12. Haozheng Xu & Yiwen Zhang & Xing Jin & Jingrui Wang & Zhen Wang, 2023. "The Evolution of Cooperation in Multigames with Uniform Random Hypergraphs," Mathematics, MDPI, vol. 11(11), pages 1-11, May.
    13. Lee, Hsuan-Wei & Cleveland, Colin & Szolnoki, Attila, 2022. "Mercenary punishment in structured populations," Applied Mathematics and Computation, Elsevier, vol. 417(C).
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Wang, Jingrui & Zhang, Huizhen & Jin, Xing & Ma, Leyu & Chen, Yueren & Wang, Chao & Zhao, Jian & An, Tianbo, 2023. "Subsidy policy with punishment mechanism can promote voluntary vaccination behaviors in structured populations," Chaos, Solitons & Fractals, Elsevier, vol. 174(C).
    2. Wang, Chengjie & Deng, Juan & Zhao, Hui & Li, Li, 2024. "Effect of Q-learning on the evolution of cooperation behavior in collective motion: An improved Vicsek model," Applied Mathematics and Computation, Elsevier, vol. 482(C).
    3. Wang, Xianjia & Yang, Zhipeng & Liu, Yanli & Chen, Guici, 2023. "A reinforcement learning-based strategy updating model for the cooperative evolution," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 618(C).
    4. Cui, Guang-Hai & Li, Jun-Li & Dong, Kun-Xiang & Jin, Xing & Yang, Hong-Yong & Wang, Zhen, 2024. "Influence of subsidy policies against insurances on controlling the propagation of epidemic security risks in networks," Applied Mathematics and Computation, Elsevier, vol. 476(C).
    5. Bai, Pengzhou & Qiang, Bingzhuang & Zou, Kuan & Huang, Changwei, 2024. "Preferential selection based on adaptive attractiveness induce by reinforcement learning promotes cooperation," Chaos, Solitons & Fractals, Elsevier, vol. 180(C).
    6. Zhang, Yao & Hao, Qing-Yi & Qian, Jia-Li & Wu, Chao-Yun & Bi, Yan, 2024. "The influence of the heterogeneities of social institutions and individuals’ tendency to establish social institutions on cooperation," Chaos, Solitons & Fractals, Elsevier, vol. 186(C).
    7. Liu, Zhuo & Wang, Juan & Li, Xiaopeng, 2024. "Evolutionary dynamics of networked N-player trust games with exclusion strategy," Chaos, Solitons & Fractals, Elsevier, vol. 186(C).
    8. Lee, Hsuan-Wei & Cleveland, Colin & Szolnoki, Attila, 2024. "Supporting punishment via taxation in a structured population," Chaos, Solitons & Fractals, Elsevier, vol. 178(C).
    9. Yang, Zhengzhi & Zheng, Lei & Perc, Matjaž & Li, Yumeng, 2024. "Interaction state Q-learning promotes cooperation in the spatial prisoner's dilemma game," Applied Mathematics and Computation, Elsevier, vol. 463(C).
    10. Lee, Hsuan-Wei & Cleveland, Colin & Szolnoki, Attila, 2024. "Suppressing defection by increasing temptation: The impact of smart cooperators on a social dilemma situation," Applied Mathematics and Computation, Elsevier, vol. 479(C).
    11. Dai, Hui & Wang, Xiaoyue & Lu, Yikang & Hou, Yunxiang & Shi, Lei, 2024. "The effect of intraspecific cooperation in a three-species cyclic predator-prey model," Applied Mathematics and Computation, Elsevier, vol. 470(C).
    12. Yang, Guoli & Wu, Yu'e & Cavaliere, Matteo, 2024. "Information-driven cooperation on adaptive cyber-physical systems," Applied Mathematics and Computation, Elsevier, vol. 466(C).
    13. Wang, Jianwei & Xu, Wenshu & Yu, Fengyuan & He, Jialu & Chen, Wei & Dai, Wenhui, 2024. "Evolution of cooperation under corrupt institutions," Chaos, Solitons & Fractals, Elsevier, vol. 184(C).
    14. Yong Shen & Jin Guo & Hongwei Kang, 2024. "The Influence of Fine Distribution and Compensation on Cooperation in Public Goods Game," Mathematics, MDPI, vol. 12(24), pages 1-21, December.
    15. Du, Chunpeng & Guo, Keyu & Lu, Yikang & Jin, Haoyu & Shi, Lei, 2023. "Aspiration driven exit-option resolves social dilemmas in the network," Applied Mathematics and Computation, Elsevier, vol. 438(C).
    16. Yan, Zeyuan & Zhao, Hui & Liang, Shu & Li, Li & Song, Yanjie, 2024. "Inter-layer feedback mechanism with reinforcement learning boosts the evolution of cooperation in multilayer network," Chaos, Solitons & Fractals, Elsevier, vol. 185(C).
    17. Zhu, Wenqiang & Pan, Qiuhui & Song, Sha & He, Mingfeng, 2023. "Effects of exposure-based reward and punishment on the evolution of cooperation in prisoner’s dilemma game," Chaos, Solitons & Fractals, Elsevier, vol. 172(C).
    18. Ding, Zhen-Wei & Zheng, Guo-Zhong & Cai, Chao-Ran & Cai, Wei-Ran & Chen, Li & Zhang, Ji-Qiang & Wang, Xu-Ming, 2023. "Emergence of cooperation in two-agent repeated games with reinforcement learning," Chaos, Solitons & Fractals, Elsevier, vol. 175(P1).
    19. Lee, Hsuan-Wei & Cleveland, Colin & Szolnoki, Attila, 2023. "Restoring spatial cooperation with myopic agents in a three-strategy social dilemma," Applied Mathematics and Computation, Elsevier, vol. 458(C).
    20. Xu, Wei & Li, Dandan & Han, Dun & Sun, Mei, 2024. "The impact of relationship stickiness and memory on the evolution of individual behavior," Chaos, Solitons & Fractals, Elsevier, vol. 183(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:chsofr:v:178:y:2024:i:c:s0960077923012602. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Thayer, Thomas R. (email available below). General contact details of provider: https://www.journals.elsevier.com/chaos-solitons-and-fractals .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.