LinGen - A text-to-video generation framework developed by Meta in collaboration with Princeton University

What is LinGen?

LinGen is a novel text-to-video generation framework jointly developed by Princeton University and Meta. Based on the linearly complex MATE module (including MA-branch and TE-branch), the framework replaces the quadratic complexity self-attention module in traditional Diffusion Transformers, enabling efficient generation of high-resolution, minute-long videos on a single GPU. LinGen significantly reduces computational costs while maintaining high-quality video output, outperforming existing state-of-the-art models in both video quality and generation efficiency, paving new avenues for long video generation and real-time interactive video applications.

LinGen's main functions

• High-resolution video generationIt supports the generation of high-resolution videos (such as 512p and 1024p) to meet the needs of high-quality content creation.
• Long-duration video generationIt supports generating videos with durations in minutes, breaking through the limitation of traditional models that can only generate short videos (10-20 seconds).
• linear computational complexityBased on the MATE module with linear complexity, it significantly reduces computational costs, making video generation more efficient and suitable for running on a single GPU.
• High-quality video outputThe generated video is comparable to existing state-of-the-art models in terms of visual quality and text alignment, while maintaining consistency between frames.
• Real-time interactive video generationLinGen enables real-time interactive video generation and editing, suitable for various dynamic content creation scenarios.

LinGen's technical principles

• MA-branch (Multi-scale attention branch):
  • Bidirectional Mamba2 moduleMamba2 is an efficient linear-complexity sequence model based on a bidirectional design that captures bidirectional dependencies in sequences.
  • Rotary Major Scan (RMS)The 3D video token tensor is rearranged based on different scanning methods (such as spatial row priority, spatial column priority, temporal row priority, and temporal column priority) to enhance short-range correlation and reduce computational latency.
  • Review TokensAdding an average-pooled token sequence before sequence processing provides a global overview of the entire sequence and enhances long-distance correlation.
• TE-branch (Time Attention Branch)By dividing the 3D video token tensor into small windows and computing self-attention within each window, TESA can capture the correlation between spatially adjacent and temporally intermediate tokens. The windows move alternately between different layers, expanding the receptive field and enhancing video consistency.
• linear complexityBased on the MATE module design, LinGen's computational complexity is linearly related to the number of pixels in the generated video, rather than the traditional quadratic relationship. This allows LinGen to significantly reduce computational costs and improve generation efficiency while maintaining high-quality output.
• Training strategyLinGen employs a progressive training strategy, first pre-training on low-resolution text-to-image tasks, and then gradually increasing video resolution and length for further pre-training. During the text-to-video pre-training stage, it combines text-image pair training to improve the consistency of generated videos. Fine-tuning on high-quality video datasets further enhances the quality of the generated videos.

LinGen's project address

• Project official websitehttps://lineargen.github.io/
• GitHub repositoryhttps://github.com/jha-lab/LinGen
• arXiv technical paper: https://arxiv.org/pdf/2412.09856

Application scenarios of LinGen

• Content creationIt can quickly generate high-quality video content, such as advertisements, movies, and TV series, significantly shortening the creation cycle and cost.
• Entertainment industryGenerate cutscenes and background videos in the game to enhance the game's visual effects and immersion.
• Education and TrainingGenerate educational videos, such as course explanations and experimental demonstrations, to enhance the fun and interactivity of teaching; generate training videos to help employees quickly understand and master knowledge, thereby improving training effectiveness.
• Ad videoQuickly generate advertising videos to meet the needs of different advertising scenarios and improve the efficiency and effectiveness of advertising production.
• Artistic CreationGenerate art videos, providing artists with new creative tools and inspiring creativity.