SUPIR - A high-fidelity AI image restoration and enhancement model

What is SUPIR?

SUPIR (Scaling-UP Image Restoration) is a groundbreaking image inpainting and enhancement method that leverages the large-scale generative model StableDiffusion-XL (SDXL) and model extension techniques. Through deep learning and multimodal methods, it achieves high-quality restoration of low-quality images. This method supports fine-grained control over image restoration via text prompts, allowing adjustments to various aspects of the restoration based on user input, such as object textures and the semantic content of the scene. SUPIR was jointly developed by researchers from the Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shanghai AI Lab, University of Sydney, Hong Kong Polytechnic University, Tencent ARC Lab, and the Chinese University of Hong Kong.

SUPIR's official website entrance

• Official project homepage:https://supir.xpixel.group/
• GitHub source code repository:https://github.com/Fanghua-Yu/SUPIR
• arXiv research paper:https://arxiv.org/abs/2401.13627

Main functions of SUPIR

• High-quality image restorationSUPIR can restore low-quality images to a high-quality version that is close to the original state, and can handle image degradation caused by various reasons (such as compression, noise, blur, etc.).
• Repair various types of imagesSUPIR can effectively restore various types of degraded images, including landscapes, faces, animals, game scenes, old movies, and old photos. It can enhance image details, restore clarity and realism, and give images a new lease on life.
• Text prompts guide repairSUPIR allows users to guide the image restoration process with text prompts, specifying particular aspects of the restoration, such as repairing a blurred object in the image, changing the material texture of an object, or adjusting the image restoration effect based on advanced semantics.
• Negative quality warningSUPIR uses negative quality cues to improve the perceived quality of images. By telling the model which image features are undesirable (e.g., "painting effect, cartoonish, blurry, messy, low quality"), the model can avoid these features during the restoration process, thereby improving the overall quality of the image.

How SUPIR works

1. Generative PriorsSUPIR uses StableDiffusion-XL (SDXL) as its generative prior, a large pre-trained image generation model with 2.6 billion parameters. The generative prior is the knowledge of the image data distribution learned by the model, used to guide the image generation and reconstruction process.
2. Datasets and text annotationsTo train SUPIR, researchers collected a dataset containing 20 million high-resolution, high-quality images, each with detailed descriptive text annotations. These annotations provide additional information about the image content, enabling the model to better understand and reconstruct the images.
3. adapterTo effectively apply the SDXL model, researchers designed and trained an adapter with over 600 million parameters. The adapter's role is to identify content in low-quality images and finely control the generation process at the pixel level.
   
4. Text prompt controlSUPIR can guide image restoration based on user-provided text prompts. These prompts can be specific descriptions of the image content or high-level semantic descriptions of the desired image quality.
5. Negative quality hints and training samplesTo improve the perceived quality of images, SUPIR introduces negative quality cues to help the model understand unwanted image features. Researchers generated images corresponding to these negative quality cues using SDXL and incorporated them into the training data to ensure the model could learn these negative quality concepts.
6. Resume guided samplingTo prevent image distortion during the generation process, SUPIR employs a novel recovery-guided sampling method that selectively guides the prediction results to approach low-quality images during diffusion to maintain the consistency of the recovery.
7. Model training and sampling settingsSUPIR used a synthetic degradation model during training, which was trained for 10 days on 64 Nvidia A6000 GPUs. During testing, the model was able to process images of 1024×1024 pixels.

Application scenarios of SUPIR

• Restoration of old photos:SUPIR can be used to restore old, damaged, or faded photographs, restoring their original colors and details, allowing precious memories to be preserved and passed down.
• Blur Image Enhancement:For photos that are blurry due to shaky hands, focus errors, or motion blur, SUPIR can improve image sharpness through advanced algorithms, making the originally blurry image sharper.
• Noise Removal:Photos taken under high ISO or low light conditions often contain noise. SUPIR can effectively identify and remove this noise while preserving important image details.
• Color correction and enhancement:SUPIR can identify and correct color distortion problems, such as oversaturation or color deviation, making the colors of the image more realistic and vivid.