DemoFusion - A free and open-source image resolution enhancement framework

What is DemoFusion?

DemoFusion is a technology framework designed for low-cost high-resolution image generation. It extends existing open-source generative AI models (such as Stable Diffusion) to make blurry, low-resolution images much clearer (up to 4x, 16x, or even higher resolution) without additional training or excessive memory requirements. DemoFusion employs progressive enhancement, skip residuals, and dilatation sampling mechanisms to achieve higher-resolution image generation, providing a solution similar to Magnific AI for users with limited resources.

DemoFusion official website entrance

• Official project homepage:https://ruoyidu.github.io/demofusion/demofusion.html
• Arxiv research paper:https://arxiv.org/abs/2311.16973
• GitHub repository:https://github.com/PRIS-CV/DemoFusion
• Hugging Face runtime address:
  • Image to Image version:https://huggingface.co/spaces/radames/Enhance-This-DemoFusion-SDXL
  • Text to Image version:https://huggingface.co/spaces/fffiloni/DemoFusion
• Replicate runtime address:
  • Image to Image version:https://replicate.com/lucataco/demofusion-enhance
  • Text to Image version:https://replicate.com/lucataco/demofusion
• Google Colab runtime address:https://colab.research.google.com/github/camenduru/DemoFusion-colab/blob/main/DemoFusion_colab.ipynb

DemoFusion Features

• High-resolution image generationDemoFusion can extend the image generation capabilities of pre-trained GenAI models (such as SDXL) to higher resolutions, such as from 1024×1024 pixels to 4096×4096 pixels or higher, without requiring additional training of the model.
• Progressive upsamplingBy progressively increasing the image resolution, DemoFusion allows users to gradually refine image details during the generation process while maintaining the overall image quality and semantic consistency.
• Global semantic consistencyBy using skip residuals and dilatation sampling mechanisms, DemoFusion can maintain global semantic consistency when generating high-resolution images, avoiding repetition and structural distortion in local regions.
• Rapid iterationDue to its progressive upsampling feature, DemoFusion allows users to quickly preview low-resolution results during the generation process, enabling rapid iteration and adjustment of the image layout and style before waiting for the high-resolution image to be generated.
• No additional hardware requiredDemoFusion can run on consumer-grade hardware such as the RTX 3090 GPU, meaning users can generate high-resolution images without an expensive hardware investment.
• Easy to integrateDemoFusion, as a pluggable framework, can be easily integrated with existing AI generative models, enabling researchers and developers to quickly apply high-resolution image generation capabilities to their projects.
• Rich application scenariosDemoFusion is not only suitable for artistic creation, but also for various fields that require high-resolution images, such as game development, film production, virtual reality, etc.

How DemoFusion works

DemoFusion works based on several key steps and mechanisms that work together to generate high-resolution images. The following is its main workflow:

1. Initialization:
   • DemoFusion begins with a low-resolution image generated using a pre-trained latent diffusion model, such as SDXL.
2. Progressive Upscaling:
   • Starting with a low-resolution image, DemoFusion iteratively increases the image's resolution. This process involves upsampling the image at the current resolution to a higher resolution, then introducing noise through a diffusion process, and finally restoring the image through a denoising process. This process is repeated, each time at a higher resolution, to gradually increase the image's detail.
3. Skip Residual:
   • During denoising, DemoFusion utilizes the noise inversion representation from previous iterations as jump residuals. This helps preserve the global structure of the image during generation while allowing for optimization of local details.
4. Dilated Sampling:
   • To enhance the global context of each denoising path, DemoFusion introduces dilatation sampling. This means that a global representation is obtained in the latent space through dilatation sampling, and these global representations are then used to guide local denoising paths to generate image content with global consistency.
5. Fusing Local and Global Paths:
   • In each iteration step, DemoFusion combines the local denoising path (the local latent representation obtained through dilatation sampling) and the global denoising path (the global latent representation obtained through skip residuals) to generate the final high-resolution image.
6. Decoding:
   • Finally, a decoder transforms the final latent representation back into the image space to obtain a high-resolution output image.

These steps and mechanisms of DemoFusion work together to enable it to efficiently generate high-resolution images with rich detail and good global consistency without additional training.

How to use DemoFusion

1. Access the DemoFusion Replicate or Hugging Face runtime address.
2. Upload the image you want to enlarge or use the sample image.
3. Enter the prompt word, describe the image, or screen.
4. Adjust the Seed value and set the Demofusion parameter.
5. Finally, click "Run" to execute the process and wait for the image to be enlarged to a high resolution.