Audio Flamingo Next - An open-source audio language model from NVIDIA and others

What is Audio Flamingo Next?

Audio Flamingo Next is a next-generation audio language model (LALM) jointly open-sourced by NVIDIA and the University of Maryland. As the latest version in the Audio Flamingo series, it supports complex audio inputs up to 30 minutes long, covering a unified understanding of speech, ambient sound, and music. The model uses Temporal Audio Chain-of-Thought technology for time-anchored inference, has over 1 million hours of training data, and outperforms similarly sized open-source models in more than 20 audio understanding benchmarks, making it a strong competitor to commercial closed-source models.

Main features of Audio Flamingo Next

• Long-duration audio understandingIt supports input of complex audio segments up to 30 minutes long and can handle mixed audio scenarios containing speech, music and ambient sound, achieving unified understanding ranging from seconds to hours.
• Time-anchored reasoningEmploying Temporal Audio Chain-of-Thought technology, the reasoning process is explicitly anchored to audio timestamps, giving the model fine-grained time awareness and enabling it to accurately locate and aggregate scattered evidence in long audio clips.
• Multimodal audio processingA single model architecture covers three major tasks simultaneously: speech recognition, music analysis, and environmental sound understanding, eliminating the need to switch between dedicated models for tasks such as speech ASR, music classification, and soundscape recognition.
• Multi-speaker trackingIt supports speech recognition and turn detection in multi-speaker scenarios, distinguishes the identities of different speakers and tracks the turn of dialogue, and is suitable for multi-speaker scenarios such as meeting recording and podcast analysis.
• Fine-grained information retrievalIt possesses the ability to search long audio files like "finding a needle in a haystack," accurately locating specific keywords, events, or speech content within tens of minutes of audio, and answering detailed questions about specific time points.
• Mission-specific variantIt provides three dedicated model versions: Instruct, Think, and Captioner, which are optimized for general question answering, complex reasoning tasks, and detailed audio description tasks, respectively, and support flexible scenario adaptation.

How to use Audio Flamingo Next

• Model AcquisitionDownload the open-source weights through the Hugging Face platform (huggingface.co/nvidia). There are three variants: Instruct, Think, and Captioner. You can choose the appropriate version according to your task requirements.
• Rapid deploymentUse the provided Colab notebook or Grado application template to launch the cloud inference environment with one click, supporting a quick experience for those with zero coding experience.
• run locallyClone the GitHub repository and install dependencies, load the model weights, and then call it through the Python interface. It supports command-line interaction and API service deployment.
• Variant Selection GuideChoose the Instruct variant for general audio question and answer, the Think variant for complex tasks requiring time-consuming reasoning, and the Captioner variant for generating detailed audio descriptions.
• Input formatSupports standard audio file formats (WAV, MP3, etc.), with a recommended sampling rate of 16kHz and a maximum single file length of 30 minutes. Longer audio files can be supported through chunked processing.
• Hardware RequirementsLocal deployment requires GPU support (an environment with sufficient video memory is recommended), while the cloud-based Colab version offers a free T4 GPU experience option.

The technical principles of Audio Flamingo Next

• Time-anchored inference mechanismThe Temporal Audio Chain-of-Thought paradigm is introduced, which explicitly anchors intermediate reasoning steps to audio timestamps, solving the problem of time-dispersed evidence aggregation in long audio files. Time-aware location encoding is achieved by replacing the standard RoPE with RoTE (Rotary Time Embeddings).
• Four-stage course learningThe system employs a progressive training strategy: pre-training (aligning the audio encoder and adapter), mid-training (expanding to 10-30 minute long audio clips), post-training (GRPO reinforcement learning to optimize dialogue safety and instruction compliance), and CoT training (time-anchored mind chain fine-tuning).

Key information and usage requirements for Audio Flamingo Next

• Development TeamNVIDIA and the University of Maryland jointly developed
  Open source situationThe model weights, training code, and dataset are completely open source (research use license).
  Model sizeBuilt on Qwen-2.5-7B, with a total of approximately 7B parameters.
  Hardware RequirementsRequires GPU support and supports 128K tokens in long context.
  Audio supportMaximum 30 minutes, 16kHz sampling rate
  License AgreementResearch use license (non-commercial)

The core advantages of Audio Flamingo Next

• Leading in long audioLongAudioBench score of 73.9 surpasses Gemini 2.5 Pro's 60.4.
• Full Modal UnificationA single model can process speech, music, and ambient sound simultaneously without requiring task-specific switching.
• Temporal interpretabilityThe reasoning process is explicitly linked to timestamps, supporting fine-grained evidence tracing.
• ASR Performance BreakthroughThe LibriSpeech test-clean error rate dropped to 1.54, achieving the best result in the LALM category.

Audio Flamingo Next project address

• Project official websitehttps://afnext-umd-nvidia.github.io/
• GitHub repositoryhttps://github.com/NVIDIA/audio-flamingo
• HuggingFace model libraryhttps://huggingface.co/nvidia/audio-flamingo-next-hf
• arXiv technical paper: https://arxiv.org/pdf/2604.10905

Audio Flamingo Next vs. Competitors

Application Scenarios of Audio Flamingo Next

• Podcasts and Long Audio AnalysisIt supports content summarization and in-depth Q&A for podcast programs and interview recordings up to 30 minutes long, helping users quickly grasp the core topics and key discussion points of long audio programs.
• Meeting minutes and multi-speaker identificationAutomatically transcribes multi-speaker meeting content and tracks speaker turns, generating structured meeting minutes and action item extraction to improve enterprise meeting efficiency and information archiving quality.
• Music Education and AnalysisIn music teaching scenarios, it helps students identify instrument types, analyze musical form and structure, and answer music theory-related questions, thereby assisting them in understanding the constituent elements and performance techniques of complex musical works.
• Film and television post-production assistanceIt generates detailed audio descriptions and metadata tags for video content, supports sound effect retrieval, background music analysis, and content annotation, and accelerates the audio material management process in film and television production.