BrushNet - A high-quality image restoration model launched by Tencent.

What is BrushNet?

BrushNet is a plug-and-play image inpainting model based on a diffusion model, developed by researchers at Tencent PCG's ARC Lab and the University of Hong Kong. It effectively handles masked regions in images through a decomposed two-branch architecture. One branch focuses on extracting pixel-level features from the masked image, while the other branch is responsible for image generation. This design allows BrushNet to finely integrate key masking information in a layered manner into the inpainting process, thereby generating high-quality inpainting results while maintaining the coherence of the original image content.

Compared to previous image restoration methods (such as Blended Latent Diffusion, Stable Diffusion Inpainting, HD-Painter, PowerPaint, etc.), BrushNet's image restoration capabilities demonstrate superior consistency in terms of style, content, color, and cue alignment.

BrushNet official website entrance

• Official project homepage:https://tencentarc.github.io/BrushNet/
• GitHub repository:https://github.com/TencentARC/BrushNet
• arXiv research paper:https://arxiv.org/abs/2403.06976

BrushNet Features

• Repair different types of imagesBrushNet can repair images in different scenes, such as human, animal, indoor and outdoor scenes, and can also repair images of different styles, such as natural images, pencil drawings, anime, illustrations, watercolors, etc.
• Pixel-level repairBrushNet can identify and process masked areas in an image, performing precise repair on each pixel to ensure a visually seamless integration between the repaired area and the original image.
• Preserve unmasked areasBy employing layered control and specific blur fusion strategies, BrushNet is able to preserve unmasked areas during the restoration process, avoiding unnecessary alterations to the original image content.
• Compatibility with pre-trained modelsAs a plug-and-play model, BrushNet can be combined with various pre-trained diffusion models (such as DreamShaper, epiCRealism, MeinaMix, etc.) to leverage their powerful generative capabilities to complete the repair task.
• Flexibility and controlUsers can control the scale and detail of the repair by adjusting the model's parameters, including the size of the repair area and the level of detail in the repair content.

How BrushNet works

BrushNet uses a diffusion model to perform image inpainting tasks through an innovative two-branch architecture.

Here is a brief overview of how BrushNet works:

1. dual-branch architectureAt its core, BrushNet is a decomposed two-branch architecture, where one branch focuses on processing the features of the masked image, while the other branch is responsible for generating the rest of the image.
2. Masked image feature extractionIn the masking branch, the model uses a variational autoencoder (VAE) to encode the masked image to extract its latent features. These features are then used to guide the image restoration process.
3. Pre-trained diffusion modelIn the generation branch, the model uses a pre-trained diffusion model to generate image content. This model has learned how to recover a sharp image from noise.
4. Feature fusionThe extracted masking image features are gradually fused into the pre-trained diffusion model, which allows for detailed control of the restoration process in a hierarchical manner.
5. Denoising and GenerationDuring backdiffusion, the model iteratively recovers a clear image from the noise through denoising steps. Each step considers the features of the masked image to ensure that the repaired area is visually consistent with the rest of the original image.
6. Fuzzy fusion strategyTo better preserve details in the unmasked areas, BrushNet employs a blurring and blending strategy. This means that when blending the masked and generated regions, a blurred mask is used to reduce hard edges and unnatural transitions.
7. Output repaired imageUltimately, the model outputs a repaired image where the masked areas are filled naturally and coherently, while the original content of the unmasked areas is preserved.