One versus all: identifiability with a multi-hazard and multiclass building damage imagery dataset and a deep learning neural network

This paper analyzed the quality of the xBD image-training dataset for identifying building damage across a variety of natural hazards using deep learning convolutional neural networks. Specifically, we evaluated the pros and cons of combining training datasets across multiple natural hazards and provided recommendations on using the provided training dataset to optimize classification accuracy for building damage detection. The xBD dataset was rebalanced, using random over-sampling and under-sampling methods. Random over-sampling randomly duplicates the minority class, while random under-sampling randomly cuts-off the majority class. With the balanced dataset, we used the xBD baseline architecture as a starting point in the classification and find that it overfit to the no damage class; therefore, we improved the base classification algorithm by modifying the top layers of ResNet50. We found that not all classes (destroyed, major damage, minor damage, and no damage) were uniformly identifiable across natural hazards; therefore, we retrained the weights from ImageNet, adding five new convolution, batch normalization, and max pooling layers on top of ResNet50. One dropout layer, with a rate of 0.5 was also added in-between the fully connected layers to reduce overfitting and improve performance. We also evaluate the identifiability of the four damage classes in the xbd dataset. Because classification performance was significantly higher for the “no damage” class as compared to “minor”, “major”, and “destroyed” classes, we evaluated merging classes. We kept the “no damage” class and created a second merged class (“damaged”) representing “minor damage,” “major damage,” and “destroyed.” We used the same architecture for the multiclass classification and the binary classification but without the ImageNet weights. Based on this work, we recommend that users be aware of performance differences across natural hazards and across damage classes. Earthquake building damage is extremely limited in the training data and, as a result, application of the trained algorithm on earthquake data cannot be evaluated given the xBD dataset. Building damage due to volcano and tsunami are also poorly represented in the training data, and do not have sufficient data for model validation (especially within all damage classes). Wind hazards are well-represented and therefore application of the algorithm trained using either the wind-only data or the multi-hazard dataset is reliable. The multi-class algorithm trained with wind hazard specific data slightly outperforms a multihazard trained multiclass model (F1 score 0.70 vs. 0.67). Both models have similar performance across all four classes (F1 > 0.5). For flood, fire, and tsunami hazards, we recommend using the binary damage classes as identifiability is low for at least two of the classes in each hazard. For flood building damage, binary classification performance resulted in a significantly higher F1 score when trained with the flood specific dataset versus the multihazard data (0.72 vs. 0.54). On the other hand, for fire building damage, classification performance is slightly higher when the model is trained on multi-hazard data, rather than trained using a fire specific dataset (F1 score 0.46 vs. 0.42).