SoulX-LiveAct - Soul App's open-source real-time digital human generation framework

What is SoulX-LiveAct?

SoulX-LiveAct is an open-source real-time digital human generation framework from the Soul App AI team, solving the stability challenge of streaming AR diffusion models. Core innovations include Neighbor Forcing technology, which aligns diffusion steps between adjacent frames to ensure image consistency; and a ConvKV Memory mechanism that achieves constant GPU memory usage, supporting generation for hours or even unlimited durations. It requires only dual H100/H200 GPUs to achieve 20 FPS real-time inference with a latency of only 0.94 seconds. SoulX-LiveAct is suitable for scenarios such as live streaming, virtual customer service, and podcasts, marking a new stage in the development of open-source digital human technology, making it suitable for production environments.

Main functions of SoulX-LiveAct

• Real-time portrait animation generationIt generates high-fidelity digital human videos in real time based on audio and text conditions, achieving precise lip synchronization, natural facial expressions, and coordinated body movements.
• Hourly/Unlimited video lengthBy using a constant video memory mechanism, it breaks through the duration limitations of traditional models and can stably generate continuous video streams of hours or even infinite duration.
• Controllable editing of emotions and actionsIt supports flexible control of head posture, gestures, and facial expressions via text commands, such as heart gestures, covering the face, laughing, etc., while maintaining consistency in identity and accurate lip-sync.
• Low-latency streaming inferenceWith just two H100/H200 cards, it can achieve 20 FPS real-time output with an end-to-end latency of only 0.94 seconds, meeting the needs of real-time interactive scenarios such as live streaming and virtual customer service.

The technical principles of SoulX-LiveAct

• Neighbor ForcingTraditional AR diffusion models use different diffusion steps in adjacent frames, leading to inconsistent distribution and image jitter. Neighbor Forcing forces adjacent frames to be generated with the same number of diffusion steps, using the latent variables of the previous frame as the conditional input of the current frame, so that the generation process is in a consistent noise space, eliminating the step alignment problem and achieving stable temporal coherence.
• ConvKV MemoryThe bottleneck of video memory generation for long videos stems from the linear growth of KV cache with the number of frames. ConvKV Memory adopts a "short-term precision + long-term compression" strategy: retaining the high-precision KV cache of the most recent frame to ensure continuity, compressing historical frames into fixed-length memory through 1D convolution (compression ratio 5:1), and resetting RoPE position encoding.
• End-to-end performance optimizationThe system employs adaptive FP8 precision to reduce computational load, combines sequential parallelism to fully utilize multi-GPU computing power, and reduces memory access overhead through operator fusion. These three approaches combined achieve 20 FPS real-time inference, requiring only 27.2 TFLOPs per frame, reducing computational cost by 30%-45% compared to similar methods.

Key information and usage requirements for SoulX-LiveAct

• Project PositioningSoul App AI Lab's open-source real-time interactive digital human generation framework solves the stability and duration limitations of AR diffusion models in streaming generation, supporting video synthesis for hours or even unlimited durations.
• Core Breakthrough – Neighbor ForcingAlign adjacent frames with the same diffusion step to eliminate image jitter caused by inconsistent distribution.
• Core Breakthrough – ConvKV MemoryConstant video memory usage, breaking through the time bottleneck.
• Core Breakthrough – Real-time Performance20 FPS streaming inference, 0.94 seconds latency.
• Real-world performance – resolution: 512×512 or 720×416.
• Real-world performance – Frame rate20 FPS.
• Real-world performance – latency0.94 seconds.
• Real-world performance – computational cost27.2 TFLOPs/frame.
• Recommended configuration – GPU: 2× NVIDIA H100 or H200.
• Recommended Configuration – EnvironmentPython 3.10, CUDA supported.
• Recommended configuration – critical dependencies: SageAttention (FP8 attention), vLLM (FP8 GEMM), LightVAE.
• Consumer-grade graphics cards – Applicable models: RTX 4090/5090 single card.

SoulX-LiveAct's core advantages

• Neighbor Forcing TechnologyBy aligning adjacent frames with the same diffusion step, the image jitter caused by inconsistent distribution in traditional AR diffusion models is eliminated, ensuring a stable and consistent generation process.
• ConvKV Memory MechanismThe system employs a "short-term precision + long-term compression" strategy to compress historical KV cache to a fixed length, achieving constant video memory usage, breaking through the duration bottleneck, and supporting hourly or even infinite duration generation.
• Real-time streaming inferenceThe model only requires two H100/H200 cards to achieve 20 FPS real-time output with an end-to-end latency of only 0.94 seconds, meeting the needs of real-time interactive scenarios such as live streaming.
• Low computing costEach frame requires only 27.2 TFLOPs, reducing computational cost by 30%-45% compared to similar methods, while balancing high quality and high efficiency.
• Long-term consistencyMaintaining stable character identities, preserving key details, and ensuring precise lip-syncing in hour-long videos are crucial to avoid issues such as identity shifts and accessories appearing and disappearing unnoticed.

How to use SoulX-LiveAct

• Environmental preparationUse conda to create and activate a Python 3.10 environment named liveact.
• Install basic dependenciesInstall the dependencies in requirements.txt via pip and install the sox audio processing tool via conda.
• Install SageAttentionClone the SageAttention repository and switch to version v2.2.0, then run setup.py to install and enable FP8 attention acceleration.
• Install the QKV operator fusion version (optional): Clone the SageAttentionFusion repository and install it to further improve operator fusion performance.
• Install vLLMInstall vLLM version 0.11.0 via pip, which provides support for FP8 GEMM matrix operations.
• Install LightVAEClone the LightX2V repository and run setup_vae.py to install the video codec components.
• Download model weightsDownload the SoulX-LiveAct model file from Hugging Face or ModelScope to your local directory.
• Download audio encoder: Obtain the chinese-wav2vec2-base audio feature extraction model.
• Dual-card H100/H200 real-time inferenceSet environment variables and run torchrun to start dual-card distributed inference, specify the model path, audio encoder path, input JSON file, and enable 20 FPS streaming audio generation.
• Inference that supports action/facial editingUsing a 512×512 resolution and a 24 FPS frame rate, the example_edit.json file containing editing instructions is loaded to achieve controllable facial expression generation.
• RTX 4090/5090 consumer-grade graphics cards runIn single-card mode, FP8 KV cache, memory block offloading, and T5 text encoder CPU offload are enabled to reduce memory usage on consumer-grade graphics cards.
• Prepare input dataEdit the JSON configuration file to specify generation parameters such as reference image path, driving audio path, and emotional action text prompts.
• Start real-time streaming generationAfter executing the inference command, the system outputs a digital human video stream with lip-sync and coordinated facial expressions in real time based on the audio input.

SoulX-LiveAct project address

• Project official websitehttps://soul-ailab.github.io/soulx-liveact/
• GitHub repository: https://github.com/Soul-AILab/SoulX-LiveAct
• HuggingFace model libraryhttps://huggingface.co/Soul-AILab/LiveAct
• arXiv technical paper: https://arxiv.org/pdf/2603.11746

SoulX-LiveAct's Competitive Comparison

Comparison Dimensions	InfiniteTalk	Live Avatar	OmniAvatar	SoulX-LiveAct
Reasoning efficiency
Throughput	25 FPS	20 FPS	–	20 FPS
Delay	3.20 s	2.89 s	–	0.94 s
Number of GPUs	8	5	–	2
TFLOPs per frame	50.2	39.1	–	27.2
Long-term generation capability
Video memory usage	linear growth	linear growth	linear growth	constant
Maximum duration	Limited by video memory	Limited by video memory	Limited by video memory	unlimited
Identity Consistency	Late drift	Gradually drifting	Severe drift	Maintain stability
Lip-sync	Late mismatch	Gradual mismatch	Severe mismatch	Continuous accuracy
Accessories/Texture Consistency	Appearing and disappearing	Loss of details	Serious loss	Continuous stability

Application scenarios of SoulX-LiveAct

• Live streaming scenariosThe model can generate digital human anchors in real time, supporting 24/7 uninterrupted live streaming. Lip movements and voice are accurately synchronized, and facial expressions are natural and rich. It is suitable for e-commerce sales, entertainment live streaming, knowledge sharing and other scenarios.
• Virtual Customer ServiceThe model can provide 24/7 online service, the digital human image is stable and consistent, it supports long-term dialogue and interaction, reduces enterprise labor costs, and improves service experience.
• Podcast/Conversation ShowUsed for producing two-person conversations and interview programs, it generates natural facial expressions and body language in real time, and the guest's image is controllable and editable, quickly producing high-quality content.
• FaceTime/Video CallIt can be used in B2B scenarios such as virtual social networking, online education, and remote conferencing, with a latency as low as 0.94 seconds and smooth and natural interaction.