ResAdapter - A diffusion model resolution adapter from ByteDance

What is ResAdapter?

ResAdapter, developed by researchers at ByteDance, is a resolution adapter designed for diffusion models (such as Stable Diffusion). It allows these image generation models to generate images with arbitrary resolutions and aspect ratios while maintaining their original style domain. Since diffusion models typically learn only the ability to generate images at specific resolutions during training, they may fail to produce high-quality results (such as distorted images or abnormal limbs) when users attempt to generate images beyond the training resolution range. ResAdapter aims to solve this problem by expanding the resolution range and aspect ratio of the images generated by the model without altering its original style domain.

ResAdapter official website entry

• Official project homepage:https://res-adapter.github.io/
• GitHub repository:https://github.com/bytedance/res-adapter
• Hugging Face Model:https://huggingface.co/jiaxiangc/res-adapter
• arXiv research paper:https://arxiv.org/abs/2403.02084

Features of ResAdapter

• Resolution InterpolationThis allows the model to generate images at a resolution lower than its training resolution, resulting in smaller images while maintaining detail and quality.
• Resolution extrapolationThis enables the model to generate images at a resolution higher than its training resolution. This is crucial for applications requiring high-resolution output, such as printing and large-size displays.
• Domain ConsistencyWhen generating images at different resolutions, ResAdapter ensures that the style of the images remains consistent with the style domain during training, avoiding style distortion or inconsistency that may occur when the resolution is changed.
• Plug and PlayResAdapter is designed to be easily integrated into existing diffusion models without requiring major modifications to the model architecture, and can be quickly applied to a variety of different models and application scenarios.
• compatibilityResAdapter is not only compatible with the basic diffusion model, but can also be used in conjunction with other image generation-related modules such as ControlNet, IP-Adapter, and LCM-LoRA to achieve more complex image generation tasks.

How ResAdapter works

• Analysis Model StructureFirst, analyze the UNet architecture of diffusion models (such as Stable Diffusion) to determine which layers are resolution-sensitive. Typically, convolutional layers are resolution-sensitive because their receptive fields are fixed.
• Insert ResCLoRA: Resolution-based convolutional LoRA (ResCLoRA) is inserted into the convolutional layers of the downsampler and upsampler in the UNet architecture. ResCLoRA dynamically adjusts the receptive field of the convolutional layer by adding a low-rank matrix, enabling it to adapt to input images of different resolutions.
• Introducing ResENormTo address the resolution extrapolation problem, resolution extrapolation normalization (ResENorm) is introduced. ResENorm trains only the group normalization layers in the UNet block to adapt to the statistical distribution of high-resolution images, while maintaining the model's adaptability to the original style domain.
• Multi-resolution trainingDuring training, image datasets at different resolutions are used. This mixed-resolution training strategy allows the ResAdapter to learn the ability to generate images at different resolutions while avoiding impact on the original style domain.
• Integration into diffusion modelAfter training, ResAdapter, as a plug-and-play module, can be integrated into diffusion models of any style. This means that regardless of the style domain the original model focuses on, ResAdapter can extend the resolution range of the generated images.
• Generate imageDuring the inference phase, the diffusion model integrated with ResAdapter can generate images of arbitrary resolution according to the user's needs. The model generates high-quality images through an iterative denoising process based on the input text prompts or conditions (such as image conditions provided by ControlNet).