Audio-Visual Speech Enhancement with Pace-Synchronized Fusion and Dynamic Modality Balancing

Abstract

Speech enhancement in noisy environments remains a formidable challenge in computational audition, particularly when relying solely on acoustic signals. While the integration of visual modalities, such as lip movements, has demonstrated substantial potential in mitigating background noise, existing audio-visual speech enhancement systems frequently struggle with asynchronous temporal dynamics and suboptimal modality weighting. This paper introduces a comprehensive framework termed Audio-Visual Speech Enhancement with Pace-Synchronized Fusion and Dynamic Modality Balancing. The proposed system addresses the fundamental temporal mismatch between audio and visual streams caused by varying frame rates and natural human articulation inconsistencies. By employing a pace-synchronized fusion mechanism, the architecture temporally aligns cross-modal feature representations before integration. Furthermore, a dynamic modality balancing module is introduced to continuously evaluate the reliability of each modality in real-time, assigning adaptive fusion weights based on the instantaneous signal-to-noise ratio and visual occlusion metrics. Extensive evaluations across multiple standardized datasets demonstrate that this approach significantly outperforms conventional audio-only and static audio-visual models, particularly in highly dynamic, non-stationary noise environments. The findings indicate a paradigm shift in how multimodal features should be aligned and weighted, paving the way for more robust and perceptually accurate speech processing systems.