Audio-Visual Flamingo - An open-source audio-visual language model from NVIDIA and other companies.

What is Audio-Visual Flamingo?

Audio-Visual Flamingo (AV-Flamingo for short) is an open-source audio-visual large language model jointly developed by NVIDIA and the University of Maryland. It is designed for the joint understanding and reasoning of long videos and complex real-world audio and video content. The model can process visual and auditory inputs simultaneously and also supports voice interaction output. In more than 15 audio-visual, multimodal, and visual benchmark tests, it outperforms many larger open-source and closed-source models with a parameter scale of 7B.

Main functions of Audio-Visual Flamingo

• Long-form video audio-visual joint understandingSupports video input up to 15 minutes, simultaneously analyzing video, audio, ambient sound, and music.
• Cross-modal reasoningAnswer questions that require both visual and auditory information (such as "What image corresponds to a certain sound in the video").
• Time-based reasoningBy explicitly anchoring inference steps to video timestamps using the TAVIT framework, time alignment and interpretability are improved.
• Voice dialogue outputIntegrated streaming TTS, supporting real-time voice responses based on audio and video content.
• Multi-skill coverageIt covers 13 types of abilities, including relational reasoning, emotion change tracking, causal reasoning, spatiotemporal perception, hallucination detection, and event counting.

The technical principles of Audio-Visual Flamingo

• ArchitectureThe SigLip visual encoder and AF-Whisper audio encoder are used to extract image/video frame features and audio features respectively; the two are mapped to the LLM embedding space through a 2-layer MLP projector.
• Cross-modal timing alignmentVisual and audio tokens are arranged in an alternating manner according to time windows, and constrained rotational temporal embedding (CRTE) is introduced to enable the model to perceive the absolute temporal sequence and relative order of multimodal events.
• Three-stage training course(1) Pre-training: using AV-Skills short context data; (2) Intermediate training: using AV-Skills long context data; (3) Post-training: using chained thinking data to gradually transition from short-range perception to long-range multi-event reasoning.
• DatasetAudio-Visual-Skills contains approximately 7 million training instances, covering 240,000 hours of video, with a particular emphasis on temporal, compositional, and cross-modal reasoning.
• Base modelBased on Qwen2.5-7B, a hybrid sequence parallelism and full-shard data parallelism strategy is adopted to process long contexts.

Follow us on WeChat and reply with "open source",join inAI open source project discussion group

How to use Audio-Visual Flamingo

• Get the modelVisit AV-Flamingo's GitHub repository or Hugging Face page to download the open-source model weights, inference code, and configuration files.
• Environment configurationConfigure the Python environment and install project dependencies, ensuring that the GPU has sufficient memory to load the 7B parameter model and its multimodal encoder.
• Ready to inputTake the image, video, or audio file to be analyzed as input, and write specific text questions or instructions.
• Visual encodingThe visual input is extracted with multi-scale spatial features by the SigLip encoder and compressed into a compact visual token sequence by the Dynamic S² module.
• Audio encodingThe audio input is resampled and features are extracted by the AF-Whisper encoder using a 30-second sliding window, supporting audio streams of any length.
• Cross-modal alignmentThe model automatically maps audio and video features to the LLM embedding space through a projection layer and arranges them in an alternating manner according to time windows, with the addition of rotational temporal embedding to preserve absolute temporal order.
• Inference generationThe processed multimodal token is concatenated with the text prompt, and input into Qwen2.5-7B for autoregressive inference to generate a text response.

The core advantages of Audio-Visual Flamingo

• Long video specialization: Optimize for long videos of 1 minute or more, even up to 15 minutes, and solve the problem of the performance drop of existing models on long videos.
• True joint reasoningBoth the dataset and the training objective are designed for cross-modal use, rather than simply splicing together single-modal capabilities.
• Temporal interpretabilityTAVIT allows models to explicitly cite specific time intervals, such as "seconds 22 to 45," as the basis for inference when answering questions about long videos.
• Fully open sourceCompared to closed-source solutions such as GPT-4o and Gemini, AV-Flamingo is fully open in terms of model, data, and code.

The project address for Audio-Visual Flamingo

• Project official website：https://avflamingo.pages.dev/
• GitHub repository：https://github.com/NVIDIA/audio-flamingo
• HuggingFace model library：https://huggingface.co/nvidia/nemotron-labs-audio-visual-flamingo-hf
• arXiv technical paper：https://arxiv.org/pdf/2607.16107

Audio-Visual Flamingo's Competitive Product Comparison

Application scenarios of Audio-Visual Flamingo

• Film and television content analysisAutomatically generates detailed commentary with timestamps for long movies/TV series, pinpointing key plot points and sound effects.
• Education and Online CoursesFor long lecture videos, cross-modal question and answer can be conducted, allowing students to ask, "At what minute did the teacher mention a certain formula?"
• Sports and Event ReviewCombining video footage with on-site commentary, this analysis examines tactical changes, goal moments, and commentator opinions.
• Podcasts and Interview ProcessingGenerate structured summaries for pure audio or audio-visual podcasts and support precise timestamp-based retrieval.
• Security and surveillance reviewIn long-term surveillance video, abnormal events can be located by combining sound (such as broken glass) and video.