Multi-agent deep reinforcement learning-based truck-drone collaborative routing with dynamic emergency response

In emergency disaster response, the dynamic nature and uncertainty of resource transportation pose significant challenges for vehicle routing planning. We address a truck-drone collaborative routing problem in humanitarian logistics, where a set of truck-drone tandems collaboratively deliver relief resources from a distribution center to a set of affected areas which is dynamically updated as disaster changes. In the truck-drone collaborative mode, as each truck performs the delivery services and serves as a mobile depot for the drone associated with it, the drone launches from its associated truck at a node, delivers relief resources to one affected area, and returns to rendezvous with the truck at the node or another node along the truck route. We cast the problem as a Markov game model with an event-driven method, which can effectively capture the dynamic changes in the states and node information of trucks and drones during relief resources delivery. To solve the model, we develop a multi-agent deep reinforcement learning algorithm, which combines prioritized experience replay and invalid action masking to improve the sample efficiency and reduce the decision space. We conduct extensive numerical studies to validate the effectiveness of the proposed method by comparing it with existing solution methods and two well-known heuristic rules, and discuss the impacts of some model parameters on the solution performance. We also assess the advantages of the truck-drone collaborative mode over the truck/helicopter-only mode through a case study of the 2008 Wenchuan earthquake.