ColorFlow - An image sequence colorization model jointly developed by Tsinghua University and Tencent.
ColorFlow is an image sequence colorization model jointly developed by Tsinghua University and Tencent ARC Labs. It can colorize images while preserving the individual identities within the sequence with fine precision. It is based on retrieval enhancement, context learning, and super-resolution techniques...
What is ColorFlow?
ColorFlow is an image sequence colorization model jointly developed by Tsinghua University and Tencent ARC Labs. It can colorize images while maintaining the individual identities within the sequence with fine precision. Based on retrieval enhancement, contextual learning, and super-resolution technologies, ColorFlow ensures color consistency between the colorized black-and-white image sequence and the reference image, making it suitable for industrial applications such as comics and animation production. In the field of image sequence colorization, ColorFlow demonstrates superior performance compared to existing technologies by improving CLIP-IS, reducing FID, increasing PSNR and SSIM scores, and enhancing AS, providing a new colorization standard for the art industry.
ColorFlow's main functions
- Retrieval-Augmented Pipeline (RAP)Extract the color image patch that is most relevant to the input black and white image from the reference image pool to guide the colorization process.
- In-context Colorization Pipeline (ICP)It accurately retrieves color identities based on powerful contextual learning and employs a dual-branch design for coloring.
- Guided Super-Resolution Pipeline (GSRP)Upsampling low-resolution color output produces high-resolution color images, enhancing detail recovery and improving output quality.
ColorFlow's technical principles
- Retrieval-AugmentedThe pre-trained CLIP image encoder generates embeddings of the input and reference images, calculates cosine similarity to identify the most similar reference image patch, and uses it for subsequent colorization training.
- In-context coloringAn auxiliary branch, "Colorization Guider," is introduced to integrate conditional information. Based on the U-Net diffusion model, features are gradually integrated to achieve pixel-level conditional embedding. The pre-trained diffusion model is fine-tuned using the lightweight LoRA (Low-Rank Adaptation) method to preserve its coloring capabilities.
- Super-ResolutionThis addresses downsampling issues during the colorization process, reducing structural distortion. It combines high-resolution monochrome images with low-resolution color output to enhance detail recovery and improve output quality.
- Self-Attention MechanismIn the diffusion model, a self-attention mechanism is used to place the reference image and the grayscale image on the same canvas, extract features, and input them layer by layer into the diffusion model for coloring.
- Timestep Shifted SamplingAdjust the sampling strategy to emphasize sampling at higher time steps and enhance the effect of the coloring process.
- Screenstyle AugmentationRandom linear interpolation is performed on grayscale images and ScreenVAE output to enhance the style adaptability of the input image and improve colorization performance.
ColorFlow's project address
- Project official website:zhuang2002.github.io/ColorFlow
- GitHub repository:https://github.com/TencentARC/ColorFlow
- HuggingFace model library:https://huggingface.co/TencentARC/ColorFlow
- arXiv technical paper:https://arxiv.org/pdf/2412.11815
- Experience the demo online:https://huggingface.co/spaces/TencentARC/ColorFlow
Application Scenarios of ColorFlow
- Comic and cartoon coloringThis tool converts black-and-white comics or cartoon images into color versions, making it ideal for comic artists and animators to quickly color line art during the creative process.
- Restoration of old photosAdding color to historical black and white photographs helps restore the original appearance of old photos and increases the aesthetic value of historical images.
- Film and video post-productionUsed for colorizing black and white films, or to provide specific color grading effects for modern filmmaking.
- Artistic CreationArtists add color to black and white artworks or explore different color schemes during the creative process.
- Education and LearningIn the field of education, it serves as a tool to help students understand the impact of color on images and learn basic concepts of image processing and computer vision.