CLEAR - A linear attention mechanism developed by the National University of Singapore, which speeds up the generation of 8K images by 6.3 times.

What is CLEAR?

CLEAR, a novel linear attention mechanism introduced by the National University of Singapore, improves the efficiency of pre-trained Diffusion Transformers (DiTs) in generating high-resolution images. By restricting the attention of each query to a local window, CLEAR achieves linear complexity with image resolution, reducing computational costs. Experiments show that after 10,000 iterations of fine-tuning, CLEAR reduces attention computation by 99.5% while maintaining similar performance to the original model, and achieves a 6.3x speedup when generating 8K images. CLEAR supports zero-shot generalization across models and plugins, as well as multi-GPU parallel inference, enhancing the model's applicability and scalability.

CLEAR's main functions

• linear complexityBy using a local attention mechanism, the complexity of pre-trained DiTs is reduced from quadratic to linear, making it suitable for high-resolution image generation.
• Efficiency improvementIt significantly reduces computational load and time latency when generating high-resolution images, thus accelerating the image generation process.
• Knowledge transferWith minimal fine-tuning, knowledge can be effectively transferred from the pre-trained model to the student model while maintaining the quality of the generated data.
• Cross-resolution generalizationCLEAR demonstrates excellent cross-resolution generalization ability and can handle image generation tasks of different sizes.
• Cross-model/plugin generalizationAttention layers trained with CLEAR can generalize to other models and plugins with zero samples, without the need for additional adaptation.
• Multi-GPU Parallel InferenceCLEAR supports multi-GPU parallel inference, optimizing the efficiency and scalability of large-scale image generation.

CLEAR's technical principles

• Local attention windowThis approach restricts each query to a local window, allowing interaction only with key-value pairs within that window, thus achieving linear complexity.
• Circular window designUnlike the traditional square sliding window, CLEAR uses a circular window to consider all key values within the Euclidean distance of each query.
• Knowledge distillationDuring fine-tuning, CLEAR uses knowledge distillation of the objective, based on flow matching loss and prediction/attention output consistency loss, to reduce the difference between the linearized model and the original model.
• Multi-GPU Parallel Inference OptimizationCLEAR leverages the locality of local attention to reduce communication overhead during multi-GPU parallel inference, thereby improving the efficiency of large-scale image generation.
• Maintain original functionalityAlthough each query only accesses local information, by stacking multiple Transformer blocks, each token can gradually capture the overall information, similar to the operation of a convolutional neural network.
• Sparse attention implementationAs a sparse attention mechanism, it can be implemented efficiently on GPUs and utilizes underlying optimizations.

CLEAR project address

• GitHub repository:https://github.com/Huage001/CLEAR
• arXiv technical paper:https://arxiv.org/pdf/2412.16112

Application scenarios of CLEAR

• Digital Media CreationIt enables artists and designers to quickly generate high-resolution images and artworks, improving creative efficiency.
• Virtual Reality (VR) and Augmented Reality (AR)In VR and AR applications, it is used to generate high-resolution virtual environments and objects in real time, enhancing the user experience.
• Game developmentThis allows game developers to generate high-quality game assets and backgrounds, reducing development time and resource consumption.
• Film and video productionIn film and video production, it is used to generate high-resolution special effects images and animations, improving the efficiency of post-production.
• Advertising and MarketingMarketers can quickly generate attractive advertising images and visual content to attract potential customers.