LongCat-Video-Avatar 1.5 - Meituan's open-source digital human video generation model

What is LongCat-Video-Avatar 1.5?

LongCat-Video-Avatar 1.5 is an open-source audio-driven digital human video generation framework from Meituan's LongCat platform. The model is built upon the 13.6B-parameter LongCat-Video video generation model, upgrading the audio encoder from Wav2Vec2 to Whisper-Large-v3 for more accurate lip-sync. It uses Step Distillation technology to compress inference to 8 steps, supports 480P/720P resolution and INT8 quantization, and possesses production-grade physical plausibility and temporal stability in scenarios such as single-person and multi-person interactions, singing performances, and stylized animations.

Main features of LongCat-Video-Avatar 1.5

• Single audio-driven video generationInput a single audio clip and text or a reference image to generate a highly dynamic, lip-synced video of a person speaking.
• Multi-audio interactive generationIt supports dual audio stream input, enabling natural interactive scenarios such as two-person dialogue and alternating speaking.
• Video continuation and long video generationIt natively supports Video Continuation, which can continuously generate minute-long videos based on existing clips without color drift.
• Stylization and Animation AdaptationIt can be generalized to anime characters, animal figures, and complex real-world conditions, and supports 3D animation styles.
• Singing and performance generationSupports music-driven dynamic facial expressions and full-body/half-body stable performance.
• Multi-task basic modelThe underlying LongCat-Video platform uniformly supports text-based video, image-based video, and video continuation.

Technical Principles of LongCat-Video-Avatar 1.5

• Whisper-Large-v3 audio codecReplacing Wav2Vec2 with Whisper-Large-v3 extracts more refined speech features and significantly improves lip-sync accuracy.
• Step Distillation: Rapid ReasoningBy reducing the number of inference steps to 8 through distillation sampling, the generation speed is significantly accelerated while maintaining quality.
• Coarse-to-fine spacetime generation strategyCoarse-to-Fine generation is used along the time and space axes, combined with Block Sparse Attention, to achieve efficient inference at 720p/30fps.
• Multi-Reward GRPO Reinforcement LearningBased on Group Relative Policy Optimization and multi-reward model training, it optimizes text alignment, visual quality, and motion coherence.
• INT8 Quantization and Context ParallelismSupports INT8 quantization to reduce memory usage and is compatible with single-card/multi-card contextual parallel inference.

How to use LongCat-Video-Avatar 1.5

• Environmental preparationClone the repository and create a Conda environment, then install PyTorch 2.6.0, FlashAttention-2, and project dependencies.
• Download ModelDownload via HuggingFace CLI LongCat-Video-Avatar-1.5 Weighting to local ./weights Table of contents.
• Single audio generationPrepare a JSON file containing the audio path, text prompts, and reference images, and run it. run_demo_avatar_single_audio_to_video.py, specify --model_type avatar-v1.5 --use_distill --use_int8 Enable 1.5 model, distillation inference and INT8 quantization.
• Multi-audio generationPrepare a JSON file containing two audio clips and their corresponding character information, and run it. run_demo_avatar_multi_audio_to_video.pyIt supports two dual audio modes: Merge and Concatenation.
• Video continuationAdd to single/multi-audio scripts --num_segments The model will automatically segment and continue writing based on the parameters, generating a longer time series of continuous videos.
• WebUI Experience:implement streamlit run ./run_streamlit.py Launch the visual interface to perform interactive generation and parameter adjustment.

The core advantages of LongCat-Video-Avatar 1.5

• Leading lip-sync accuracyThe Whisper-Large-v3 audio encoder delivers greater lip-sync accuracy and smoother facial expression transitions.
• Long video identity consistencyMaintain a stable identity and consistent full-body movements in long-duration speaking scenes and hand-object interaction scenarios.
• Extremely fast reasoning speedThe 8-step distillation inference significantly shortens the generation time, making it suitable for business scenarios with higher real-time requirements.
• Open source and commercially usableThe model weights and inference code are both open source under the MIT license, allowing for free deployment and secondary development.
• Memory friendlyINT8 quantization and multi-GPU parallel support lower the hardware barrier.

The project address for LongCat-Video-Avatar 1.5

• Project official website: https://meigen-ai.github.io/LongCat-Video-Avatar-1.5-Page/
• GitHub repository: https://github.com/meituan-longcat/LongCat-Video
• HuggingFace model libraryhttps://huggingface.co/meituan-longcat/LongCat-Video-Avatar-1.5

Comparison of LongCat-Video-Avatar 1.5 with similar competing products

Application scenarios of LongCat-Video-Avatar 1.5

• AI-powered voiceover and e-commerce marketingMerchants can input product explanation audio and reference images to generate digital human-based product promotion videos with accurate lip movements and natural expressions in batches, significantly reducing the cost of real people appearing on camera.
• Online education and virtual instructorsEducational institutions can convert course audio into virtual lecture videos with stable and consistent identities, supporting long-duration lectures and gesture interactions, thereby improving content reuse rates.
• Virtual Customer Service and Corporate ImageEnterprises can deploy dedicated digital human customer service representatives, which, combined with multi-audio stream functionality, enable multi-round dialogue visualization for scenarios such as website reception and intelligent outbound call follow-up.
• Short video and social media creationCreators input singing or performance audio and quickly generate highly dynamic, stylized (animation, realism, etc.) virtual avatar short videos, adapted for platforms such as Douyin and Kuaishou.