Next-Frame Diffusion - An autoregressive video generation model jointly developed by Peking University and Microsoft
Next-Frame Diffusion (NFD) is an autoregressive video generation model jointly developed by Peking University and Microsoft Research. It combines the high-fidelity generation capabilities of diffusion models with the causality and controllability of autoregressive models. Next-Frame...
What is Next-Frame Diffusion?
Next-Frame Diffusion (NFD) is an autoregressive video generation model jointly developed by Peking University and Microsoft Research. It combines the high-fidelity generation capabilities of diffusion models with the causality and controllability of autoregressive models. Next-Frame Diffusion achieves efficient frame-level generation based on block-wise causal attention and a diffusion transformer. The model achieves real-time video generation exceeding 30 FPS while maintaining video quality and coherence. Techniques such as consistency distillation and speculative sampling are introduced to further improve sampling efficiency. The model performs exceptionally well in large-scale motion-conditional video generation tasks, significantly outperforming existing methods.
The main functions of Next-Frame Diffusion
- Real-time video generationIt supports real-time video generation of over 30 FPS on high-performance GPUs, making it ideal for interactive applications that require fast response times, such as games, virtual reality, and real-time video editing.
- High-fidelity video generationIt generates high-fidelity video content in a continuous space, and can capture details and textures better than traditional autoregressive models.
- Action condition generationIt generates corresponding video content based on the user's real-time operations, and has extremely high flexibility and controllability in interactive applications.
- Long-term video generationIt supports generating video content of any length, making it suitable for applications that require long-term continuity, such as storytelling or simulated environments.
The technical principle of Next-Frame Diffusion
- Block-wise Causal AttentionThe core of the model is a block causal attention mechanism, combining intra-frame bidirectional attention and inter-frame causal dependencies. Within each frame, the model performs bidirectional self-attention to capture intra-frame spatial dependencies. Between frames, the model maintains causality, with each frame only depending on previous frames, ensuring coherence and causal consistency in the generated data.
- Diffusion Model and Diffusion TransformerThe model is based on the principle of diffusion modeling, generating video frames in a progressive denoising process. The Diffusion Transformer is a key component of NFD, leveraging the powerful modeling capabilities of the Transformer architecture to handle the spatiotemporal dependencies of video.
- Consistency DistillationTo accelerate the sampling process, a consistency distillation technique is introduced, extending the Simplified Consistency Model (sCM) from the image domain to the video domain, significantly improving the generation speed while maintaining the high quality of the generated content.
- Speculative SamplingBy leveraging the consistency of action inputs in adjacent frames, future frames can be generated in advance. If a change in action input is detected later, the speculatively generated frames are discarded, and generation restarts from the last verified frame, significantly reducing inference time and improving the efficiency of real-time generation.
- Action ConditioningNFD (Network Functions) controls the direction and content of video generation based on action-conditional input. Action input can be user commands, control signals, or other forms of conditional information, and the model generates corresponding video frames based on the input.
Next-Frame Diffusion project address
- Project official website:https://nextframed.github.io/
- arXiv technical paper:https://arxiv.org/pdf/2506.01380
Application scenarios of Next-Frame Diffusion
- Game developmentThe model generates a dynamic game environment in real time based on player actions, enhancing the gaming experience.
- Virtual Reality (VR) and Augmented Reality (AR)The model generates virtual scenes in real time, providing an immersive experience for VR and AR applications.
- Video content creationThe model generates high-quality video content suitable for advertising, film, and television production.
- Autonomous driving and roboticsGenerate behaviors and scenarios for autonomous vehicles or robots in different environments for use in training and testing.
- Education and trainingIt generates virtual experimental environments to help students conduct scientific experiments and learn.