Spatial structure might impede cooperation in evolutionary games with reinforcement learning

Cooperation has attracted considerable attention in recent years. In order to explain altruistic cooperation behaviors that emerged in social dilemmas, a large number of mechanisms have been proposed under the framework of traditional evolutionary game (EG) theory, especially network reciprocity, which has achieved great success. On the other hand, the design of AI algorithm provides a new idea for agents’ decision-making behaviors. The influence of multi-agent reinforcement learning (RL) on cooperation has also received a lot of attention. However, the study of EG with AI algorithm in a population located on spatial structures has not been thoroughly considered. In this paper, we incorporate RL into EGs conducted on spatial structures. By numerical simulations, we find that, in comparison to the well-mixed case, cooperation might be impeded in the population located on spatial structures when the agents update their strategies by adopting Q-learning algorithm instead of pairwise imitation that widely used in traditional EGs. We further reveal the mechanism for the evolution of cooperation in the EGs with Q-learning algorithm, by investigating the distributions of the Q-tables held by agents in the population. Our findings may help understand the different outcomes on spatial structures in EGs with Q-learning algorithm when compared with that in traditional EGs.