Fun-Audio-Chat - An open-source end-to-end voice interaction model from Alibaba Tongyi

What is Fun-Audio-Chat?

Fun-Audio-Chat is a new generation end-to-end voice interaction model open-sourced by Alibaba Cloud's Tongyi Bailing team, possessing voice understanding, emotion perception, and task execution capabilities. The model employs a dual-resolution design, working collaboratively at 5Hz and 25Hz frame rates, saving nearly 50% of GPU computing resources compared to similar products. Its core innovation lies in the Core-Cocktail two-stage training strategy, effectively avoiding catastrophic forgetting, while also supporting multilingual voice translation and role-playing functions. In international benchmarks such as OpenAudioBench, the open-source Fun-Audio-Chat-8B surpasses competitors like GLM4-Voice in tasks such as voice dialogue and emotion recognition. It has already been applied to scenarios such as intelligent customer service and emotional companionship, and users can download and use it for free through the ModelScope and HuggingFace platforms.

Main functions of Fun-Audio-Chat

• End-to-end voice interactionIt adopts an end-to-end voice interaction mode, directly generating voice output from voice input, eliminating the need for the multi-module splicing of speech recognition (ASR) + language model (LLM) + text-to-speech (TTS) in the traditional mode, thus achieving higher efficiency and lower latency.
• Emotional perceptionIt can perceive a user's emotional state through details such as semantics, tone, speaking speed, and pauses, even if the user does not directly express emotions.
• Emotional responseWhen users are angry, anxious, or happy, the model provides just the right amount of comfort, companionship, or empathy, offering a conversational experience similar to that with a friend.
• Natural speech command executionThe model supports Speech Function Call, allowing users to issue commands using natural language, and the system can automatically call relevant functions to complete complex tasks.
• Open source and ease of useThe 8B model weights, inference code, and Function Call access examples are all open source, making it convenient for developers to download and use.

The technical principles of Fun-Audio-Chat

• End-to-end S2S architectureFun-Audio-Chat employs a Speech-to-Speech (S2S) end-to-end architecture, directly generating speech output from voice input, eliminating the need for traditional multi-module concatenation of ASR (Audio Recognition), LLM (Large Language Model), and TTS (Text-to-Speech). This significantly improves processing efficiency, reduces system latency, and delivers a smoother voice interaction experience.
• Dual resolution designThe model employs an innovative dual-resolution processing mechanism: the Shared LLM layer performs efficient semantic processing at a 5Hz frame rate, while the SRH (Speech Reconstruction Head) generates high-quality speech at a 25Hz frame rate. This reduces GPU computational overhead by nearly 50% while maintaining speech quality, achieving a balance between performance and efficiency.
• Millions of hours of multi-task data trainingThe model has been trained on millions of hours of multi-task data, covering real-world scenarios such as audio understanding, voice question answering, emotion recognition, and tool invocation. It can understand user intent in a more "down-to-earth" way. On multiple authoritative benchmarks such as OpenAudioBench, MMAU, Speech-ACEBench, and VStyle, it ranks first among models of the same size, and its overall performance surpasses that of competitors such as GLM4-Voice, Kimi-Audio, and Baichuan-Omni.
• Emotional perception abilityFun-Audio-Chat boasts excellent emotion recognition capabilities, sensing emotions from paralinguistic cues such as tone of voice, speech rate, and pauses. Even when users don't directly express emotions, it accurately identifies them and provides appropriate responses, making the conversational experience more natural and human.
• Speech Function Call functionThe model supports function calls via natural voice commands. Users simply issue commands via voice, and the system can automatically call the relevant functions to complete complex tasks. This expands the model's application scenarios, enabling it not only to chat but also to truly "get things done."

Fun-Audio-Chat project address

• Project official websitehttps://funaudiollm.github.io/funaudiochat/
• Github repository: https://github.com/FunAudioLLM/Fun-Audio-Chat
• HuggingFace model library：https://huggingface.co/FunAudioLLM/Fun-Audio-Chat-8B
• Technical Papers: https://github.com/FunAudioLLM/Fun-Audio-Chat/blob/main/Fun-Audio-Chat-Technical-Report.pdf

Application scenarios of Fun-Audio-Chat

• voice chatFun-Audio-Chat allows for natural and fluent voice conversations with users, providing an experience similar to real-person interaction, making it suitable for everyday chatting and social interaction.
• Emotional companionshipThe model can sense a user's emotions and respond accordingly, such as offering comfort, encouragement, or empathy. It is suitable for use when users feel lonely, anxious, or need to talk.
• Intelligent device controlUsers can control smart devices, such as smart home devices and smart wearables, through voice commands for more convenient operation.
• Voice Customer ServiceIn customer service scenarios, Fun-Audio-Chat can understand users' questions and provide accurate answers, improving customer service efficiency and user experience.
• role playUsers can specify models to play specific roles, such as e-sports commentators or virtual assistants, to meet different entertainment or work needs.