An unsupervised embedding method based on streaming videos for process monitoring in repetitive production systems

Advanced production systems, such as multi-step assembly processes, predominantly comprise of repetitive operations. The repetitive manual or human–robot integrated production operations call for new real-time process management technologies such as the increasing use of sensors and the development of system intelligence. Conventional process monitoring and management methods, which are often labor-intensive, fall short in providing immediate and actionable insights. To tackle this limitation, we develop an unsupervised embedding method to automatically delineate the process into different stages and predict real-time progress information. We propose a Contrastive Variational Autoencoder as a feature extractor to adeptly embed repetitive processes into a Gaussian Mixture Model. Based on the extracted features, we propose an adaptive change-point detection and an Iterative Dynamic Time Wrapping algorithm to identify and segment multiple standardized process stages automatically. Theoretically, we establish the asymptotic optimality of the detected change-points associated with the given precision of image and feature extractors, ensuring the high-quality process stage separation and labeling. The proposed method autonomously extracts essential features encapsulating progress information from a limited set of unlabelled process videos. Through four diverse case studies including production of an actual aircraft spoiler, our method exhibits very promising performance. Specifically, it achieves an average of 98.14% accuracy in predicting production progress and 0.9202 area under the curve in predicting progress deviation across three distinct production environments. The proposed process monitoring method in repetitive production systems has the potential to significantly improve productivity, promote standardization of repetitive operations, and predict production deviations.