PixelHacker - An image restoration model jointly developed by Huazhong Technology and VIVO

What is PixelHacker?

PixelHacker is an image inpainting model jointly developed by Huazhong University of Science and Technology and VIVO AI Lab. Based on the Latent Categories Guidance (LCG) paradigm, it encodes the foreground and background separately, and injects features into the denoising process using linear attention, achieving structural and semantic consistency. The model is pre-trained on a large-scale dataset (containing 14 million image-mask pairs) and fine-tuned on several open-source benchmark datasets (such as Places2, CelebA-HQ, and FFHQ), demonstrating excellent inpainting performance.

PixelHacker's main features

• High-quality image restorationIt performs exceptionally well on various datasets, including natural scenes and facial images, generating realistic restoration content.
• Structural and semantic consistencyEnsure that the repaired area transitions naturally with the surrounding environment in terms of texture, shape, and color, and logically conforms to the scene.
• Adaptable to various scenariosIt is highly adaptable to different types of images (such as landscapes, faces, and complex backgrounds) and supports multiple resolutions and mask types.

The technical principles of PixelHacker

• Potential Category Guidance (LCG)The method divides the image into foreground and background parts, and encodes latent features in two fixed-size embeddings. This avoids dependence on specific category labels. During training, image-mask pairs are constructed using various masks (such as random brush masks, object semantic masks, scene semantic masks, etc.) and assigned to the foreground or background embeddings.
• Diffusion Model ArchitectureThe model is based on a diffusion model as its generation framework. The diffusion model generates image content through a progressive denoising process, producing high-quality and diverse images. During the denoising process, latent features of the foreground and background are injected into the model using linear attention.
• Training and fine-tuningPre-trained on a massive dataset containing 14 million image mask pairs, PixelHacker learns rich image distribution and semantic information. Building upon this pre-training, PixelHacker is fine-tuned on several open-source benchmark datasets (such as Places2, CelebA-HQ, and FFHQ) to adapt to specific tasks and data distributions, further improving performance.
• Multi-step interactionIn each step of denoising, linear attention is used to interact the latent features of the foreground and background with the current image features, ensuring that the generated content is consistent with the surrounding environment in terms of structure and semantics.
• Free Classifier Guidance (CFG)During the inference phase, the classifier-free guidance technique is used to adjust the guidance scale to balance the diversity and consistency of the generated content.

PixelHacker project address

• Project official website:https://hustvl.github.io/PixelHacker/
• GitHub repository:https://github.com/hustvl/PixelHacker
• arXiv technical paper:https://arxiv.org/pdf/2504.20438

Application scenarios of PixelHacker

• Restoring old photosAutomatically fills in scratches, stains, or missing parts to restore the integrity of photos.
• Remove irrelevant objectsRemove unwanted elements (such as pedestrians and clutter) from the image to keep the background natural.
• Creative content generationIt helps artists and designers quickly generate high-quality images for advertising, posters, or artistic creation.
• Medical Image RestorationIt fills in missing or damaged areas in medical images, improving diagnostic accuracy.
• Cultural heritage protectionRestoring images of cultural relics and ancient books, filling in missing parts, and contributing to cultural heritage preservation.