LanDiff - A text-to-video generation hybrid framework jointly developed by Zhejiang University and Dark Side of the Moon

What is LanDiff?

LanDiff is an innovative hybrid framework for high-quality text-to-video (T2V) generation. It combines the advantages of autoregressive language models (LLM) and diffusion models, effectively overcoming the limitations of single methods in semantic understanding and visual quality through a coarse-to-fine generation approach. It achieved a high score of 85.43 on the VPench T2V benchmark, surpassing several open-source and commercial models, including 13B's Hunyuan Video.

LanDiff's main functions

• High-efficiency semantic compressionThe semantic tagger compresses 3D visual features into 1D discrete representations with a compression ratio of up to 14,000 times while retaining rich semantic information.
• High-quality video generationBased on the streaming diffusion model, semantic tags are refined into high-fidelity videos, supporting long video generation and reducing computational costs.
• Semantic consistency and causal modelingIt relies on the autoregressive generation of language models to ensure that the video and text are highly consistent and temporally coherent, avoiding the temporal inconsistency problem of traditional diffusion models.
• Flexible control and customizationIt supports control over frame rate, motion score, and other conditions, and can generate videos of specific length and dynamic characteristics, balancing high-quality visual effects with semantic accuracy.
• High-efficiency computing and optimizationIt employs video frame grouping technology to reduce temporal redundancy and uses an efficient Transformer structure and attention mechanism to reduce computational resource consumption.

LanDiff's technical principles

• Coarse-to-fine generative paradigmLanDiff employs a two-stage generation process:
  • Phase 1 (Coarse-grained generation)Semantic tags are generated based on language models (LLM). These tags represent the high-level semantic structure of the video and provide a blueprint for subsequent video generation.
  • Second stage (fine-grained generation)The semantic tags are refined into high-fidelity videos through a diffusion model, and details are gradually added to ultimately generate high-quality videos.
• Semantic taggerThis method compresses 3D visual features into a compact 1D discrete representation, achieving a compression ratio of up to 14,000 times. Inspired by the MP4 video coding algorithm, video frames are divided into keyframes (IFrames) and non-keyframes (PFrames). Keyframes are fully encoded, while non-keyframes only capture temporal changes, significantly reducing temporal redundancy.
• Language ModelTextual features are extracted using a pre-trained T5-XXL model, and the video is converted into a 1D discrete tag sequence. Additional control conditions (such as frame number and motion score) are introduced to enhance the controllability of the generated results. Semantic tags are generated through an autoregressive approach to ensure semantic consistency and temporal coherence of the generated content.
• Flow diffusion modelIt adopts an architecture similar to MMDiT, using a video tagger decoder to decode semantic tags into semantic features, which are then used as conditional guidance for the diffusion model to generate videos.

LanDiff's project address

• Project official website:https://landiff.github.io/
• arXiv technical paper:https://arxiv.org/pdf/2503.04606

Application scenarios of LanDiff

• Video productionLanDiff can quickly generate high-quality video content, helping video creators, advertising companies, and media organizations to efficiently produce video ads, short films, animations, and more.
• Virtual Reality (VR) and Augmented Reality (AR)Generate virtual scenes and character animations to enrich the content ecosystem of VR and AR applications.
• Educational VideosBased on the teaching syllabus or description of knowledge points, educational videos are generated to help students understand complex concepts more intuitively.
• Social media contentGenerate personalized, engaging video content for social media platforms to enhance brand influence and user engagement.