OmniTalker - A real-time text-driven avatar generation framework launched by Alibaba.

What is OmniTalker?

OmniTalker is a real-time text-driven speaking avatar generation technology released by Alibaba. It can simultaneously process multiple modal inputs, including text, images, audio, and video, generating natural speech responses in a streaming manner. Its core architecture is the Thinker-Talker architecture. Thinker is responsible for processing multimodal input and generating semantic representations and text content, while Talker transforms this information into fluent speech output. OmniTalker employs TMRoPE (Time-Aligned Multimodal Rotation Position Embedding) technology to ensure precise synchronization between video and audio input.

OmniTalker's main functions

• Multimodal input processingIt can perceive multiple modalities such as text, images, audio, and video.
• Streaming text and speech responsesIt generates text and natural speech responses in a streaming manner, and the audio and video encoders use a block-based processing method to decouple the processing of long-sequence multimodal data.
• Precise audio and video synchronizationBy proposing the TMRoPE (Time-aligned Multimodal RoPE) technology, audio and video are organized sequentially in an interleaved manner to achieve precise synchronization of input.
• Real-time interactionIt supports chunked input and instant output, enabling fully real-time interaction.
• The speech generation is natural and fluent.It excels in the naturalness and stability of speech generation, surpassing many existing streaming and non-streaming alternatives.
• performance advantagesIt performs well in multimodal benchmark tests, with audio capabilities superior to the similarly sized Qwen2-Audio and on par with the Qwen2.5-VL-7B.

OmniTalker's technical principles

• Thinker-Talker ArchitectureOmniTalker employs a Thinker-Talker architecture, where Thinker handles multimodal input (including text, images, audio, and video) and generates high-dimensional semantic representations and text content; Talker, based on the semantic representations and text provided by Thinker, generates natural speech responses in a streaming manner. This avoids interference between text generation and speech generation, ensuring consistency and real-time performance of semantic expression.
  • ThinkerBased on the Transformer decoder architecture, it is equipped with audio and image encoders and is responsible for the extraction and understanding of multimodal information.
  • TalkerIt adopts a dual-track autoregressive Transformer decoder structure, directly utilizing Thinker's high-dimensional semantic representation to generate speech tokens, ensuring the naturalness and fluency of speech output.
• TMRoPE (Time-Aligned Multimodal Rotational Position Embedding)To address the time synchronization issue of audio and video input, OmniTalker introduced the TMRoPE technology. By interleaving audio and video frames in chronological order and performing positional encoding, it ensures seamless continuity of information from different modalities along the timeline. This enables the model to more accurately understand and generate audio and video content.
• StreamingOmniTalker supports streaming input and output, enabling real-time processing of multimodal information and rapid response. Its audio and visual encoders employ a chunking approach, breaking down long sequences of data into smaller chunks for processing, reducing latency and improving efficiency.
  • Block prefillingThe audio encoder uses a 2-second block attention mechanism, while the visual encoder uses flash attention to increase the efficiency of the MLP layer.
  • Sliding window DiT modelUsed for streaming generation of MEL spectrograms, further supporting high-quality streaming speech generation.
• End-to-end trainingThe Thinker and Talker modules are jointly trained end-to-end, sharing historical context information. This avoids the accumulation of errors that may occur between modules trained individually, ensuring the overall performance and consistency of the model.
• High-efficiency speech generationOmniTalker's speech generation module employs a highly efficient speech codec (qwen-tts-tokenizer) to stream audio tokens in an autoregressive manner. This reduces data requirements and inference complexity, while improving the naturalness and robustness of the generated speech.

OmniTalker project address

• Project official website:https://humanaigc.github.io/omnitalker/
• arXiv technical paper:https://arxiv.org/pdf/2504.02433v1

OmniTalker Application Scenarios

• Intelligent voice assistantOmniTalker's real-time audio and video interaction capabilities and natural, fluent voice generation make it an ideal intelligent voice assistant. It can process user voice commands and generate real-time voice responses, providing users with a more natural and convenient interactive experience.
• Multimodal content creationIn the field of content creation, OmniTalker can process text, image, and video input simultaneously to generate corresponding text or voice descriptions.
• Education and TrainingOmniTalker can be used in the education and training fields to provide students with a richer and more personalized learning experience by handling multimodal input.
• Intelligent Customer ServiceIn the field of intelligent customer service, OmniTalker can process customers' voice or text questions in real time and generate accurate responses. This can improve customer service efficiency and enhance customer experience.
• Industrial quality inspectionIn the manufacturing sector, OmniTalker can detect defective parts on the production line in real time by simultaneously processing product appearance images and process parameter text.