LongCat-Video-Avatar - Meituan's open-source digital human video generation model

What is LongCat-Video-Avatar?

LongCat-Video-Avatar is an audio-driven character animation model developed by the Meituan LongCat team. The model can generate ultra-realistic, lip-synced long videos, maintaining consistency in character identity and natural movement. LongCat-Video-Avatar supports multiple generation modes, including Audio-to-Text Video (AT2V), Audio-to-Text-to-Image Video (ATI2V), and video continuation. By decoupling audio signals from motion, avoiding repetitive content, and reducing VAE error accumulation, it achieves high-quality, long-duration video generation, suitable for actor performances, singer animations, podcasts, sales presentations, and multi-person interactive scenarios.

The main functions of LongCat-Video-Avatar

• Multi-mode video generationIt supports audio-to-text video (AT2V), audio-to-text-image video (ATI2V), and video continuation, meeting diverse needs in different scenarios.
• Natural Dynamics and Identity ConsistencyThe model can maintain consistency in character identity, generate natural facial expressions, synchronized lip movements and body movements, and maintain natural and smooth dialogue behavior in multi-person interactive scenarios.
• High-quality video generationBy decoupling audio signals from motion, rigid behavior during silence is avoided, pixel degradation is reduced, and the stability and consistency of long videos are ensured.
• Diverse application scenariosIt is suitable for scenarios such as actor performances, singer showcases, podcasts, and sales presentations, providing high-quality video generation solutions for different fields.

The technical principles of LongCat-Video-Avatar

• Decoupling speech and action (Disentangled Unconditional Guidance)By distinguishing between speech signals and overall movements, the model can generate natural body movements even in silent segments, avoiding static behavior caused by over-reliance on speech signals and achieving more natural dynamic performance.
• Reference Skip AttentionThis mechanism selectively introduces reference image information, which can maintain the consistency of the character's identity, prevent the "copy and paste" phenomenon caused by excessive leakage of reference images, and balance visual fidelity and action diversity.
• Cross-Chunk Latent StitchingBy reducing redundant VAE decoding-encoding loops in autoregressive generation, pixel degradation is reduced, cumulative errors in long video generation are avoided, and the coherence and consistency of the video are ensured.
• A unified framework based on diffusion models (DiT-based framework)It adopts an architecture based on the diffusion model, which can generate ultra-realistic long videos and supports multiple generation modes, including audio-to-text to video (AT2V), audio-to-text-to-image to video (ATI2V), and video continuation.
• Multi-stream audio input supportIt supports single-stream or multi-stream audio input and uses L-ROPE (Learnable Relative Positional Encoding) technology to bind audio and visual information, adapting to complex multi-person interactive scenarios.

LongCat-Video-Avatar project address

• Project official website: https://meigen-ai.github.io/LongCat-Video-Avatar/
• GitHub repositoryhttps://github.com/MeiGen-AI/LongCat-Video-Avatar
• HuggingFace model libraryhttps://huggingface.co/meituan-longcat/LongCat-Video-Avatar

Application scenarios of LongCat-Video-Avatar

• Film and television productionIt is used to generate natural facial expressions and synchronize lip movements of actors, reduce special effects costs, and improve the realism of film and television characters.
• Music and EntertainmentGenerate vivid physical movements and stage performances for singers and virtual idols, enhancing the visual effects of music videos and virtual performances.
• Content creation and educationGenerate high-quality videos for broadcasters and teachers to enhance the appeal and interactivity of podcasts, video blogs, and online education.
• Business and SalesThe model can generate natural and smooth product demonstrations and virtual customer service videos, improving sales performance and brand image.
• Multi-person interactive sceneThe model supports multi-person dialogue and interaction, maintaining natural communication dynamics, and is suitable for meetings, interviews, and social entertainment.