Qwen3-Omni - A native end-to-end full-modal large model open sourced by Alibaba Tongyi

What is Qwen3-Omni?

Qwen3-Omni is the industry's first native end-to-end multimodal AI model launched by Alibaba's Tongyi team. It can seamlessly process multimodal data, including text, images, audio, and video. The model achieves state-of-the-art (SOTA) performance in 22 out of 36 audio and video benchmark tests, supports text processing in 119 languages, and possesses global language capabilities. The model boasts a latency as low as 211 milliseconds, is highly controllable, supports custom system prompts, and has powerful built-in tool functionality. The Qwen team has open-sourced multiple versions, including Qwen3-Omni-30B-A3B-Instruct, Qwen3-Omni-30B-A3B-Thinking, and Qwen3-Omni-30B-A3B-Captioner, driving technological development and application innovation. The model is now available on the Qwen Chat model experience platform.

Qwen3-Omni's main functions

• Native full modalityQwen3-Omni is a native full-modal large model, and pre-training full-modal models do not reduce intelligence.
• powerful performanceIt achieved 32 state-of-the-art (SOTA) scores in open-source tests and 22 overall SOTA scores in 36 audio and video benchmark tests, surpassing closed-source strong models such as Gemini-2.5-Pro, Seed-ASR, and GPT-4o-Transcribe. Meanwhile, its image and text performance reached SOTA levels among models of the same size.
• MultilingualIt supports 119 text-based interactive languages, 19 speech-understanding languages, and 10 speech-generated languages.
• Faster responsePure model end-to-end audio dialogue latency as low as 211ms, video dialogue latency as low as 507ms.
• Long audioSupports audio comprehension up to 30 minutes.
• PersonalizationIt supports customization of the system prompt, allowing users to modify reply styles, user personas, and more.
• Tool callSupports function calls, enabling efficient integration with external tools/services.
• Open source general audio captioner: Open source Qwen3-Omni-30B-A3B-Captioner, a low-illusion and highly detailed general audio caption model, filling a gap in the open source community.

Qwen3-Omni's technical principles

• Thinker-Talker Architecture:
  • ThinkerIt is primarily responsible for text generation tasks, processing text input and outputting high-level semantic representations to provide basic information for subsequent speech generation.
  • TalkerIt focuses on the generation of streaming speech tokens, directly using the semantic representation output by Thinker to predict multi-codebook sequences through autoregression, thereby achieving low-latency frame-by-frame streaming speech generation.
  • MTP moduleDuring the decoding process, the MTP module is responsible for outputting the residual codebook of the current frame. The Code2Wav module then synthesizes the codebook into the corresponding audio waveform, achieving efficient streaming audio generation.
• Innovative architecture design:
  • AuT audio encoderTrained on massive amounts of audio data (20 million hours), it possesses powerful general audio representation capabilities, providing a solid foundation for models to process audio tasks.
  • MoE architectureBoth Thinker and Talker adopt the MoE (Mixture of Experts) architecture, which supports high concurrency processing and fast inference, significantly improving the efficiency and performance of the model in multi-task processing.
  • Multicodebook technologyTalker uses a multi-codebook autoregressive scheme, which generates an encoding/decoding frame in each generation step. The MTP module will synchronously output the remaining residual codebook, further optimizing the efficiency and quality of speech generation.
• Full-modal intelligence without degradationDuring the text pre-training phase, the model is trained using a mixture of unimodal and cross-modal data. This mixed training approach ensures that the performance of each modality is comparable to that of pure unimodal training, significantly enhancing the model's cross-modal capabilities and making it more adept at handling multimodal tasks. Qwen3-Omni performs exceptionally well in speech recognition and command following tasks, with performance comparable to top models such as Gemini-2.5-Pro. It accurately understands and executes voice commands, providing users with a smooth voice interaction experience.
• Real-time audio and audio-visual interactionThe entire process (including AuT audio encoding, Thinker text processing, Talker speech generation, and Code2wav audio synthesis) is fully streamed, supporting direct streaming decoding of the first frame token into audio output, ensuring the efficiency and smoothness of real-time audio and audio-visual interaction.

Qwen3-Omni's performance

• In a comprehensive performance evaluationThe Qwen3-Omni performs comparably to other Qwen series single-modal models of similar size on single-modal tasks, and is particularly outstanding on audio tasks, demonstrating a significant advantage.
• In 36 audio and video benchmark testsQwen3-Omni achieved the best performance in the open-source field in 32 tests and reached the industry-leading level (SOTA) in 22 tests, outperforming powerful closed-source models such as Gemini-2.5-Pro, Seed-ASR, and GPT-4o-Transcribe.

Qwen3-Omni's project address

• Project official website: https://qwen.ai/blog?id=65f766fc2dcba7905c1cb69cc4cab90e94126bf4&from=research.latest-advancements-list
• GitHub repository: https://github.com/QwenLM/Qwen3-Omni
• HuggingFace model library: https://huggingface.co/collections/Qwen/qwen3-omni-68d100a86cd0906843ceccbe
• Technical Papers: https://github.com/QwenLM/Qwen3-Omni/blob/main/assets/Qwen3_Omni.pdf

Application scenarios of Qwen3-Omni

• Content creationThe model can generate high-quality text, images, audio, and video content, providing creators with abundant creative materials and improving creative efficiency.
• Intelligent Customer ServiceThe model supports multilingual text and voice interaction, quickly and accurately understands user problems and provides solutions, improving the customer service experience.
• EducationThe model can generate personalized learning materials and interactive content, such as audio explanations and image examples, to meet the learning needs of different students.
• Medical assistanceThe model can process multimodal data such as medical images and voice recordings, assisting doctors in making diagnoses and developing treatment plans.
• Multimedia EntertainmentThe model can create multimedia content such as music and videos, providing users with a personalized entertainment experience.