AB
AB
AiBoss

All Categories

project

FLM-Audio - A full-duplex audio dialogue model open-sourced by the Beijing Academy of Artificial Intelligence.

FLM-Audio is a native full-duplex audio dialogue model jointly released by the Beijing Academy of Artificial Intelligence, Spin Matrix, and Nanyang Technological University in Singapore. It supports both Chinese and English. Employing a native full-duplex architecture, it can...

What is FLM-Audio?

FLM-Audio is a native full-duplex audio dialogue model jointly released by the Beijing Academy of Artificial Intelligence (BAAI), Spin Matrix, and Nanyang Technological University (NTU) in Singapore. It supports both Chinese and English. Employing a native full-duplex architecture, it merges auditory, speaking, and monologue channels at each time step, avoiding the high latency issues of traditional time-division multiplexing schemes. Its unique natural monologue and dual training paradigm make the model more closely resemble natural human communication in dialogues, effectively solving the asynchronous alignment problem. FLM-Audio was trained using only 1 million hours of data, significantly reducing the data volume, yet achieving high-quality responses and agile, natural reactions. It also exhibits strong robustness against noise and user interruptions.

Main functions of FLM-Audio

  • Full-duplex voice interactionIt enables "listening and speaking simultaneously," allowing users to interrupt the model at any time. The model can immediately pause its output, understand the new question, and respond, resulting in a natural and smooth interaction with low latency.
  • Supports multiple languagesThe model supports both Chinese and English, meeting the dialogue needs of users who speak different languages.
  • Natural speech modelingIt adopts a "natural monologue" approach to simulate the rhythm of human speech, and strengthens the alignment of language and acoustic semantics through "dual training" to balance low latency and language modeling performance.
  • Low-data efficient trainingThe model, trained with only about 1 million hours of audio data, has 7 billion parameters and maintains high robustness and naturalness even in noisy environments and frequently interrupted scenarios.
  • Strong robustnessIt is highly robust to noise and user interruptions, can quickly pause the current output, accurately understand new questions and answer them in real time, ensuring the fluency and accuracy of the dialogue.
  • Fully open source supportThe paper, model weights, and code are all publicly available, supporting local deployment and secondary development, which facilitates research and application expansion.

FLM-Audio Technical Principles

  • Native full-duplex architectureThe model design supports simultaneous voice input and output, and can process voice streams in real time, enabling an interactive mode where you can listen and speak simultaneously.
  • Natural Monologue TrainingInstead of word-by-word alignment, it uses "natural monologues" composed of continuous sentences and pauses, which is closer to the way humans actually speak and improves the naturalness of voice interaction.
  • Dual training strategyBy alternating monologues at the beginning and end of the audio recordings during training, the alignment between language and acoustic semantics is strengthened, thereby improving the model's ability to understand and generate speech content.
  • Efficient training with small dataA high-parameter model was trained using a small amount of audio data (approximately 1 million hours), and low latency and high robustness were achieved by optimizing the training methods and architecture.

FLM-Audio project address

  • GitHub repositoryhttps://github.com/cofe-ai/flm-audio
  • HuggingFace model libraryhttps://huggingface.co/CofeAI/FLM-Audio
  • arXiv technical paper: https://arxiv.org/pdf/2509.02521

Application scenarios of FLM-Audio

  • Online EducationAI teaching assistants can answer students' questions in real time, providing a more natural and efficient interactive experience.
  • Gaming and Virtual Reality (VR)NPCs can engage in uninterrupted, interruptible natural voice interactions, enhancing the game's immersion.
  • Intelligent Customer ServiceLow-latency conversations reduce user waiting time, improve customer service efficiency, and enhance user experience.
  • Smart CompanionIt provides users with more realistic voice interaction, enhancing the sense of companionship.
  • voice assistantIt provides a more natural voice interaction experience in smart home, smart office and other scenarios.
  • Meeting SupportIn multi-person meetings, it enables real-time translation, recording, and interaction, improving meeting efficiency.
Back