A Hierarchical Static-Dynamic Encoder-Decoder Structure for 3D Human Motion Prediction with Residual CNNs

Human motion prediction aims at predicting the future poses according to the motion dynamics given by the sequence of history poses. We present a new hierarchical static-dynamic encoder-decoder structure to predict the human motion with residual CNNs. Specifically, to better mine the law of the motion, a new residual CNN-based structure, v-CMU, is proposed to encode not only the static information but also the dynamic information. Based on v-CMU, a hierarchical structure is proposed to model different correlations between the different given poses and the predicted pose. Moreover, a new loss function combining the static and dynamic information is introduced in the decoder to guide the prediction of the future poses. Our framework features two-folds: (1) more effective dynamics mined due to the fusion of information of the poses and the dynamic information between poses and the hierarchical structure; (2) better decoding or prediction performance, thanks to the mid-level supervision introduced by the new loss function considering both the static and dynamic losses. Extensive experiments show that our algorithm can achieve state-of-the-art performance on the challenging G3D and FNTU datasets. The code is available at https://github.com/liujin0/SDnet .