FreeScale - An inference framework that requires no fine-tuning, enhancing the generation capabilities of diffusion models and achieving 8K resolution images for the first time.

What is FreeScale?

FreeScale, developed by Nanyang Technological University, Alibaba Group, and Fudan University, is a fine-tuning-free inference framework that enhances the ability of pre-trained diffusion models to generate high-resolution images and videos. By processing and fusing information at different scales, FreeScale effectively addresses the problem of repetitive patterns caused by increased high-frequency information when generating ultra-high-resolution content. FreeScale is the first to achieve 8K resolution image generation, significantly improving the quality and fidelity of the generated content while reducing inference time, thus surpassing existing methods.

FreeScale's main functions

• High-resolution visual generationFreeScale can generate high-quality images and videos with resolutions up to 8K, expanding the capabilities of visual diffusion models in high-resolution generation.
• No fine-tuning requiredUnlike traditional methods that require fine-tuning, FreeScale can achieve high-resolution output without requiring additional adjustments or training to the pre-trained model.
• Processing high-frequency informationFreeScale effectively manages high-frequency information by extracting and fusing information at different scales, reducing repetitive patterns and artifacts in the generated content.
• Multi-scale information fusionBy combining information from different receptive field scales, FreeScale optimizes the generation of local and global details, thereby improving the overall quality of visual content.
• Flexible control of detail levelUsers can adjust the level of detail in different areas as needed to achieve more precise control over visual effects.

FreeScale's technical principles

• Customized self-cascaded upsamplingStarting with pure Gaussian noise, we gradually denoise, generate images using the training resolution, and obtain higher resolution images based on upsampling.
• Constrained dilated convolutionTo expand the receptive field of convolutions and reduce local repetition issues, FreeScale uses dilated convolutions in specific network layers.
• Scale fusionDuring the denoising process, the self-attention layer is adjusted to have both global and local attention structures. High-frequency details from global attention and low-frequency semantics from local attention are fused based on Gaussian blur.
• Frequency component extraction and fusionBased on extracting and fusing the required frequency components, the high-resolution generation quality is optimized, and the problem of repetitive patterns caused by high-frequency information is reduced.
• control of detailAdjusting the level of generated details, the level of newly generated details is controlled based on the scaling cosine decay factor, thereby enabling differentiated processing of details in different semantic regions.

FreeScale's project address

• Project official website:haonanqiu.com/projects/FreeScale
• GitHub repository:https://github.com/ali-vilab/FreeScale
• arXiv technical paper:https://arxiv.org/pdf/2412.09626

FreeScale application scenarios

• High-quality image generationIn the fields of artistic creation and digital entertainment, it generates high-resolution artwork, game textures, and 3D model textures.
• Video content productionIn film and video production, it generates high-resolution video content, improves video quality, and reduces post-production costs and time.
• Virtual Reality (VR) and Augmented Reality (AR)In VR and AR applications, it generates high-resolution virtual environments and objects, enhancing the user experience.
• Advertising and MarketingCreate compelling advertising images and videos to enhance the visual impact and appeal of your ads.
• Social media contentSocial media users generate high-resolution images and videos for personal branding or content sharing.