DFNet: Decoupled Fusion Network for Dialectal Speech Recognition

Deep learning is often inadequate for achieving effective dialect recognition in situations where data are limited and model training is complex. Differences between Mandarin and dialects, such as the varied pronunciation variants and distinct linguistic features of dialects, often result in a significant decline in recognition performance. In addition, existing work often overlooks the similarities between Mandarin and its dialects and fails to leverage these connections to enhance recognition accuracy. To address these challenges, we propose the Decoupled Fusion Network (DFNet). This network extracts acoustic private and shared features of different languages through feature decoupling, which enhances adaptation to the uniqueness and similarity of these two speech patterns. In addition, we designed a heterogeneous information-weighted fusion module to effectively combine the decoupled Mandarin and dialect features. This strategy leverages the similarity between Mandarin and its dialects, enabling the sharing of multilingual information, and notably enhance the model’s recognition capabilities on low-resource dialect data. An evaluation of our method on the Henan and Guangdong datasets shows that the DFNet performance has improved by 2.64% and 2.68%, respectively. Additionally, a significant number of ablation comparison experiments demonstrate the effectiveness of the method.