Wan-Streamer - Alibaba's open-source real-time full-duplex multimodal foundational model

What is Wan-Streamer?

Wan-Streamer is an open-source end-to-end real-time full-duplex multimodal basic model from Alibaba DAMO Academy. It integrates the input and output tokens of text, audio, and video into the same causal sequence through a unified causal Transformer architecture, achieving sub-second real-time bidirectional video interaction. The model response latency is only 200ms, the total end-to-end latency is 550ms, and it stably outputs 25FPS synchronized audio and video.

Main functions of Wan-Streamer

• Real-time audio and video dialogueIt supports two-way video calls between users and AI digital humans, and the AI can output voice and facial expressions simultaneously.
• Full-duplex interactionIt supports real-time interruption by users and proactive questioning by AI, enabling natural and fluent human-computer dialogue.
• Multimodal input understandingIt can simultaneously receive and understand the user's video feed, voice, and text input.
• Streaming fragment generationIt adopts 160ms short-time streaming fragmentation, and generates feedback while receiving information, without waiting for the full frame.
• Long-term consistencyGlobal key-value context caching ensures that the character's appearance and tone remain stable and consistent throughout long conversations.

Wan-Streamer's technical principles

• Unified single Transformer streaming architectureThe system interweaves user visuals, voice, and text input with AI-generated voice, facial expressions, and subtitles into a causal token stream, using text autoregressive prediction and audio/video conditional stream matching for joint generation.
• Fully causal technology stack designThe encoder, decoder, VAE and attention layer all follow causal constraints, and only use historical time information to predict the next unit. The core block causal attention restricts the visibility of future tokens.
• Three-stage training processMulti-task pre-training of mixed text, image, and speech dialogue data; full-duplex fine-tuning of learning listening, interruption, and pause behaviors; lightweighting of the large teacher model using streaming distillation and the adoption of a rolling self-enforcing strategy.
• Thinker-Performer Dual GPU InferenceThinker handles user audio and video encoding and context updates, while Performer performs stream matching, audio and video denoising, and generation. Both share a context and execute in parallel.

How to use Wan-Streamer

Currently, Wan-Streamer has only released a paper and a demo on its official website; the complete code and model weights have not yet been open-sourced, and local deployment is not yet possible.

Wan-Streamer's core advantages

• Ultra-low latencyThe model-side response time is 200ms, and the end-to-end response time is only 550ms, which is far lower than the industry average of more than 1 second.
• End-to-end integrationA single model completes the entire process of perception, understanding, and generation, eliminating the need for multiple modules such as ASR, LLM, TTS, and rendering.
• Full-duplex real-time interactionIt supports listening and responding simultaneously, and can be interrupted midway, with an interaction level as natural as a real person.
• Audio and video synchronizationVoice and facial movements are generated synchronously, eliminating the need for post-alignment and repair, and ensuring zero lip misalignment.
• Long dialogue stabilityThe global key-value context ensures that the character's appearance and tone do not drift over long periods of time.

Wan-Streamer project address

• Project official websitehttps://wan-streamer.com/
• arXiv technical paper: https://arxiv.org/pdf/2606.25041

Comparison of Wan-Streamer with similar competing products

Application scenarios of Wan-Streamer

• Virtual Customer ServiceProvides face-to-face real-time video consultation services in scenarios such as banking and e-commerce.
• Live InteractionAI anchors respond to viewers' comments and voice questions in real time, enhancing the interactive experience in the live streaming room.
• AI companionThe emotional companion digital human supports real-time video chat and provides an immersive companionship experience.
• Game NPCsInteractive game characters engage in real-time video conversations with players, enhancing the game's immersive experience.
• Online EducationAI virtual teachers provide real-time video Q&A and personalized tutoring.