Real-time dispatch of an integrated energy system based on multi-stage reinforcement learning with an improved action-choosing strategy

The uncertainty of renewable energy has brought challenges to the real-time dispatch of integrated energy systems (IES). Nowadays, reinforcement learning (RL) is widely used in IES real-time dispatch to deal with the uncertainty of renewable energy. However, traditional RL algorithms often face the problem of dimensionality when there are numerous controllable units in the system, which will increase operating costs and training time significantly. Based on the above issues, we developed a novel real-time dispatch method for IES with RL model training in stages based on the dueling double deep quality network (D3QN). Dispatches of different controllable units in IES are decomposed into a multi-stage training process according to the degree of thermoelectric coupling and the complexity of equipment operation. This makes the action space of each training stage independent, alleviating the problem of extra-large action space in the traditional RL method. In addition, an improved action-choosing strategy is proposed to enhance local optimization in the process of algorithm training by introducing “offset” according to probability in the training progress. The simulations were carried out on four different types of days in an IES. The results show that the proposed method can effectively reduce operating costs and accelerate convergence.