MT-Color - A controllable image coloring framework jointly developed by Shanghai Jiao Tong University and Bilibili.

What is MT-Color?

MT-Color is a controllable image colorization framework based on a diffusion model, jointly developed by Shanghai Jiao Tong University and Bilibili. It achieves accurate instance-level image colorization based on user-provided instance-aware text and masks. The framework uses a pixel-level mask attention mechanism to prevent color overflow, instance masks and text guidance modules to address color binding errors, and a multi-instance sampling strategy to enhance instance-awareness. MT-Color has built the GPT-Color dataset, providing high-quality instance-level annotations and supporting more refined image colorization tasks. MT-Color outperforms existing methods in color accuracy and visual quality, generating images that are more consistent with human visual perception.

Main functions of MT-Color

• Precise instance-level shadingBased on the instance mask and text description provided by the user, accurately color different objects in the image to ensure that the color of each object is consistent with its description.
• Prevent color overflowBased on a pixel-level mask attention mechanism, it effectively avoids the erroneous spread of color between different objects and maintains clear color boundaries.
• High-quality color generationThe generated color images are rich and natural in color, and have a high resolution (512×512) with clearer details.
• Flexible user controlIt supports users to finely control the image coloring process using text descriptions and masks to meet different coloring needs.
• Dataset supportWe have built a dedicated GPT-Color dataset, providing high-quality instance-level annotations to support more refined image colorization tasks.

MT-Color's technical principles

• Pixel-level mask attention mechanismThe grayscale image features extracted by ControlNet are aligned with the latent features of Stable Diffusion using a pixel-level cross-attention mechanism to prevent pixel information exchange between different instances and alleviate color overflow issues. A cross-attention mask is constructed using a segmentation mask to ensure that pixel information does not interfere with each other between different instances.
• Instance mask and text bootstrap moduleExtract the mask and text representation of each instance, fuse them with latent features based on a self-attention mechanism, and form a self-attention mask based on the instance mask to prevent instance text from guiding the coloring of other areas and reduce color binding errors.
• Multi-instance sampling strategyDuring the sampling process, each instance region is sampled separately, and the results are merged to further enhance the coloring effect of instance perception.
• GPT-Color datasetBased on pre-trained visual language models (such as GPT-4 and BLIP-2), high-quality instance-level annotations are automatically generated on existing image datasets, including fine-grained text descriptions and corresponding segmentation masks, providing rich training data for model training.

MT-Color project address

• arXiv technical paper:https://arxiv.org/pdf/2505.08705

Application scenarios of MT-Color

• Historical photo restorationAdding vibrant colors to black and white historical photographs enhances their visual appeal and historical feel.
• Film and television post-productionRepair and enhance black-and-white or low-quality color lenses to improve visual effects.
• Artistic CreationIt helps artists add color to black and white sketches or illustrations, bringing their creative ideas to life.
• Medical Image ProcessingAdding color to medical images enhances their diagnostic value.
• EducationAdd color to black and white images in teaching materials to enhance the learning experience.