project
CoGenAV - A multimodal speech representation model jointly developed by Tongyi and Shenzhen Technology University
CoGenAV - A multimodal speech representation model jointly developed by Tongyi and Shenzhen Technology University
What is CoGenAV?
CoGenAV (Contrastive-Generative Audio-Visual Representation Learning) is an advanced multimodal learning model focused on the alignment and fusion of audio and visual signals. It is trained with the dual objectives of contrastive feature alignment and text prediction generation, utilizing synchronized audio, video, and text data to learn and capture temporal correspondences and semantic information. CoGenAV demonstrates extremely high data efficiency, requiring only 223 hours of labeled data for training.
CoGenAV's main functions
- Audio Visual Speech Recognition (AVSR)CoGenAV can combine audio and visual information (such as the speaker's mouth movements) to improve the accuracy of speech recognition.
- Visual Speech Recognition (VSR)Speech recognition uses only visual information (such as the speaker's mouth movements) and does not rely on audio signals.
- Speech processing in noisy environmentsIn high-noise environments, CoGenAV enhances the robustness of speech processing by using visual information to assist audio signals.
- Speech reconstruction and augmentationCoGenAV can be used for speech reconstruction and enhancement tasks, improving speech quality through multimodal information fusion.
- Active Speaker Detection (ASD)It detects the person currently speaking by analyzing audio and visual signals.
CoGenAV's technical principles
- Feature extractionIn the feature extraction stage, CoGenAV uses ResNet3D CNN to analyze the speaker's lip movements in the video, capturing the dynamic relationship between sound and lip shape. A Transformer encoder is used to extract speech information from the audio, precisely aligning audio and video features.
- Comparison and synchronizationThe Seq2Seq Contrastive Learning method is adopted to enhance the correspondence between audio and video features, and the ReLU activation function is introduced to filter out interference frames, thereby improving the stability of the model in complex environments.
- Generate synchronizationBy leveraging pre-trained ASR models (such as Whisper) to align audio and video features with their acoustic-text representations, a lightweight adaptation module (Delta Upsampler + GatedFFN MHA) is designed to effectively improve cross-modal fusion efficiency.
CoGenAV's project address
- Github repository:https://github.com/HumanMLLM/CoGenAV
- HuggingFace model library:https://huggingface.co/detao/CoGenAV
- arXiv technical paper:https://arxiv.org/pdf/2505.03186
Application scenarios of CoGenAV
- Smart assistants and robotsCoGenAV's multimodal representations can be integrated into intelligent assistants and robots to better understand and respond to voice commands in complex environments.
- Video content analysisCoGenAV can be used for the analysis and understanding of video content. By analyzing the audio and visual information in the video, it can provide more accurate subtitle generation, content recommendation and other functions.
- Industrial applicationsIn industrial environments, CoGenAV can be used in scenarios such as voice control equipment and voice monitoring, improving the robustness and reliability of the system through multimodal information fusion.
- HealthcareCoGenAV can be used for voice interaction in medical devices, such as intelligent medical assistants and voice-controlled medical devices, improving the ease of use and interactivity of medical devices.