BlockDance - A diffusion model acceleration method jointly launched by Fudan University and ByteDance

What is BlockDance?

BlockDance is a new method developed by Fudan University in collaboration with ByteDance's intelligent content creation team for accelerating diffusion models. BlockDance is based on identifying and reusing structurally similar spatiotemporal features (STSS) in adjacent time steps, reducing redundant computation and improving inference speed by up to 50%. BlockDance focuses on structural features in the post-denoising stage, avoiding image quality degradation caused by reusing low-similarity features. BlockDance introduces BlockDance-Ada, which dynamically allocates computational resources based on reinforcement learning, adjusting the acceleration strategy according to the complexity of different instances to further optimize content quality and inference speed.

BlockDance's main functions

• Accelerating the reasoning processBy reducing redundant computation, the inference speed of DiTs can be increased by 25% to 50%, improving the efficiency of the model in practical applications.
• Maintain generation qualityWhile accelerating the process, it maintains consistent generation results with the original model, ensuring the visual quality, detail, and adherence to cues in images and videos.
• Dynamic resource allocationBased on BlockDance-Ada, it dynamically adjusts the allocation of computing resources according to the complexity of different generation tasks, achieving a better balance between speed and quality.
• Wide applicabilityIt supports seamless application to various diffusion models and generation tasks, such as image generation and video generation, and has strong versatility.

BlockDance's technical principles

• Feature similarity analysisIn the denoising process of the diffusion model, there is a high degree of similarity between features at adjacent time steps, especially in the shallow and middle layers of the model. These layers are primarily responsible for generating the structural information of the image, which is relatively stable in the early stages of the denoising process. BlockDance identifies structurally similar spatio-temporal (STSS) features based on feature similarity analysis as a key factor for acceleration.
• Caching and Reuse MechanismsThe denoising process is divided into a "caching step" and a "reuse step." In the caching step, the model saves the feature outputs of certain modules from the current step. In the subsequent reuse step, the model directly uses the previously cached features, skipping the repeated calculation of modules and saving computational resources.
• Dynamic Decision Network (BlockDance-Ada)Introducing BlockDance-Ada, a lightweight decision network based on reinforcement learning. It dynamically determines which steps should be cached and which can be reused based on the complexity of the current generation task (e.g., image structural complexity, number of objects, etc.). This dynamic adjustment mechanism allows BlockDance to achieve a better speed-quality balance across different generation tasks and models.
• Reinforcement learning optimizationThe decision network is trained using the policy gradient method in reinforcement learning. A reward function is designed to balance the trade-off between image quality and computational efficiency. The reward function includes image quality rewards (such as visual aesthetics and adherence to cues) and computational rewards (such as the proportion of reused steps). By maximizing the expected reward, the decision network learns optimal caching and reuse strategies, achieving maximum speedup while maintaining generation quality.

BlockDance's project address

• arXiv technical paper:https://arxiv.org/pdf/2503.15927

BlockDance Application Scenarios

• Image generationAccelerate the generation of high-quality images for artistic creation, game design, and other applications, while maintaining visual quality.
• Video generationImprove the speed of tasks such as video creation and animation production, and maintain visual and temporal consistency.
• Real-time interactionUsed in real-time applications such as VR and AR to quickly respond to user input and improve the experience.
• Massive content generationIt efficiently generates large numbers of images and videos, reducing computational costs and improving efficiency.
• Resource-constrained environmentIt can run efficiently in resource-constrained scenarios such as mobile devices and edge computing without requiring additional training.