SoulX-FlashTalk - Soul App's open-source real-time digital human generation model

What is SoulX-FlashTalk?

SoulX-FlashTalk is the first 14-parameter real-time digital human generation model open-sourced by the AI team under Soul App, achieving a sub-second latency of 0.87 seconds and a high frame rate of 32fps. The model employs bidirectional streaming distillation and a multi-step self-correction mechanism to achieve stable generation for unlimited duration, full-body motion interaction, and multi-language support. It is suitable for scenarios such as 24/7 live streaming, virtual customer service, and game NPCs. The model has already ranked among the top 5 on HuggingFace's I2V trend list, providing an open-source solution for commercial-grade real-time digital human applications.

Main functions of SoulX-FlashTalk

• Real-time audio and video generationBased on a 14B large model, it achieves a sub-second latency of 0.87 seconds and a high frame rate output of 32fps, meeting the real-time interactive requirements of live streaming.
• Audio-driven digital humansIt supports receiving voice or audio input, accurately driving the virtual avatar's lip movements, facial expressions, and body movements in sync.
• Full-body motion synthesisSupports full-body dynamic generation and highly precise hand gesture representation.
• Ultra-long stable generationThe self-correction mechanism ensures consistent identity, stable image, and lossless image quality during long-term generation.
• Multilingual supportThe model uses a Chinese-optimized speech encoder and a Chinese-English bilingual subtitle encoder, supporting cross-language digital human driving.
• Infinite Stream GenerationIt supports 24/7 continuous live streaming, and the system runs stably without crashing or lagging.
• Multi-style imageIt is compatible with various visual styles, including cartoon and live-action, to meet the image customization needs of different application scenarios.

The technical principles of SoulX-FlashTalk

• Two-way flow distillationBy preserving the intra-block bidirectional attention mechanism during the streaming generation process, the spatiotemporal relevance is effectively maintained, while the training process is significantly simplified. The model can converge with only 1,000 steps of supervised fine-tuning and 200 steps of distillation. Compared with traditional methods, it achieves a 23-fold improvement in training efficiency, laying the foundation for the real-time deployment of large models.
• Delay-aware spatiotemporal adaptationAs the first-stage training strategy, it is specifically optimized for low-resolution input, short frame sequences, and dynamic aspect ratio bins, so that the large model with 14B parameters can first adapt to the needs of fast inference, reduce the computational burden while maintaining the generation quality, and resolve the contradiction between the number of parameters of the large model and the inference speed.
• Multi-step review self-correction mechanismIt is used to ensure the stability of infinite-duration generation. It can detect and correct accumulated errors in real time during the generation process, prevent errors from snowballing over time, and ensure that identity characteristics are consistent, the picture is stable and smooth, and the visual quality is lossless when generating long videos, thus achieving true "infinite streaming" output.
• 3D VAE latent space compressionBased on the WAN2.1 architecture, it performs efficient latent space encoding and decoding of high-resolution video, significantly reducing the computational burden of real-time generation; combined with the full 3D attention and multimodal cross-attention mechanism of the 14B DiT generator, and the conditional encoder layer for multi-dimensional encoding of speech, images and text, it constructs a complete end-to-end real-time digital human generation system.

SoulX-FlashTalk project address

• Project official websitehttps://soul-ailab.github.io/soulx-flashtalk/
• GitHub repository: https://github.com/Soul-AILab/SoulX-FlashTalk
• HuggingFace model libraryhttps://huggingface.co/Soul-AILab/SoulX-FlashTalk-14B

SoulX-FlashTalk Application Scenarios

• 24/7 AI Live Streaming RoomDigital human e-commerce anchors can achieve 24/7 uninterrupted live streaming, read and reply to bullet comments in real time, greatly reducing labor costs while maintaining a natural and smooth live streaming experience.
• AI Virtual Mentor and Intelligent Customer ServiceIt can be applied in scenarios such as bank tellers and online education, providing a face-to-face interactive experience similar to video calls, and supporting real-time voice Q&A and emotional feedback.
• Mass production of high-quality short videos and short dramasA complete digital human video can be generated directly from just one audio clip, without the need for motion capture equipment or post-production. The output quality of long videos is stable and consistent, greatly improving content production efficiency.
• Real-time NPCs in the gameThe model supports voice-driven, non-scripted dialogue, enabling real-time linkage between emotions and actions, providing players with a more immersive and dynamic interactive experience.