AnimateDiff-Lightning - A model launched by ByteDance for quickly generating high-quality videos.

What is AnimateDiff-Lightning?

AnimateDiff-Lightning is a high-quality video generation model recently launched by researchers at ByteDance. It utilizes progressive adversarial diffusion distillation technology to achieve rapid video generation. This model aims to address the main challenges of speed and computational cost in existing video generation models while maintaining high-quality generated videos. Compared to the original AnimateDiff, it generates videos more than ten times faster!

AnimateDiff-Lightning official website entry

• Hugging Face model address:https://huggingface.co/ByteDance/AnimateDiff-Lightning
• Hugging Face Demo running:https://huggingface.co/spaces/AP123/AnimateDiffLightning
• arXiv research paper:https://arxiv.org/html/2403.12706v1

Features of AnimateDiff-Lightning

• Fast video generationAnimateDiff-Lightning can quickly generate videos with few-step inference, significantly reducing the time required for video generation and making it suitable for scenarios that require instant or rapid content generation.
• High-quality outputDespite its fast generation speed, AnimateDiff-Lightning is still able to maintain high quality of video content, ensuring that the generated video has high visual clarity and detail.
• Style compatibilityThrough cross-model distillation technology, AnimateDiff-Lightning can adapt to a variety of different base models, which means it can generate video content with different art styles (such as realism, animation, cartoon, etc.).
• Video generation with different aspect ratiosAnimateDiff-Lightning can generate videos with different aspect ratios and supports a variety of video formats and layout requirements.
• Video to video conversionAnimateDiff-Lightning not only supports text-to-video generation, but also video-to-video style conversion, providing users with a high degree of creative freedom.
• Compatibility with control moduleAnimateDiff-Lightning can be used in conjunction with image control modules (such as ControlNet) and camera motion modules (such as Motion LoRA) to further enhance the control over video content.

How AnimateDiff-Lightning works

The AnimateDiff-Lightning method is an improvement on the AnimateDiff model, and it enhances the speed and quality of video generation through cross-model diffusion distillation. The key steps and components of the AnimateDiff-Lightning method are as follows:

1. Model and Data Preparation

• Select a base modelAnimateDiff-Lightning has selected several popular base models, including realistic and anime-style models, such as Stable Diffusion v1.5, RealisticVision v5.1, epiCRealism, ToonYou Beta 6, IMP v1.0, and Counterfeit v3.0.
• Data generationThe distillation process was supported by generating a large amount of video clip data using the AnimateDiff model and a selected base model. This data included realistic videos generated from the WebVid-10M dataset and anime-style videos generated using text prompts.

2. Cross-model distillation

• Shared motion moduleThe AnimateDiff model consists of a frozen image base model and a shared motion module. During distillation, only the weights of the motion module are updated, while the weights of the image base model remain unchanged.
• Multi-GPU trainingDifferent base image models are loaded on different GPUs, while the same motion module is initialized. This allows for distillation on multiple base models simultaneously, improving training efficiency.

3. Stream Conditional Video Discriminator

• Extended discriminatorTo handle different flows from multiple base models, the discriminator is extended to be flow-conditional, meaning it can learn and evaluate different flow trajectories based on the indices of the base models.
• Training the discriminatorThe discriminator is trained together with the diffusion and distillation models to ensure that the generated videos match the original models in style and quality.

4. Distillation process

• Progressive distillationAnimateDiff-Lightning performs distillation in the order of 128 → 32 → 8 → 4 → 2, using mean squared error (MSE) and classifier free guidance (CFG) for initial distillation, and then using adversarial loss for subsequent steps.
• gradient accumulationDue to GPU memory limitations, each GPU can only process one batch, so gradient accumulation is used to achieve a larger effective batch size.

5. Evaluation and Testing

• Qualitative assessmentBy comparing with the original AnimateDiff and AnimateLCM models, we demonstrate the quality of videos generated by AnimateDiff-Lightning at different inference steps.
• Quantitative assessmentThe FVD (Fréchet Video Distance) metric was used to quantitatively compare videos generated on different base models to demonstrate the advantage of AnimateDiff-Lightning in maintaining the original quality.