DDColor - An AI image colorization framework launched by Alibaba that turns black and white images into color.

What is DDColor?

DDColor is an open-source AI image colorization framework developed by researchers at Alibaba DAMO Academy. It can transform black and white images into full-color images with a single click. This method uses a dual-decoder architecture (pixel decoder and color decoder) to automatically colorize grayscale images, resulting in more realistic and vivid color images. It aims to solve the multimodal uncertainty and high ill-posedness problems existing in traditional image colorization methods.

How DDColor works

DDColor works based on an end-to-end deep learning model, which implements the image colorization process through two main decoder components. The core workflow of DDColor is as follows:

1. Feature extractionFirst, a pre-trained image classification model (such as ConvNeXt) is used as the encoder to extract high-level semantic features from the input grayscale image. These features include information about the image's structure, texture, and objects.
2. Pixel DecoderThe features extracted by the encoder are fed into the pixel decoder, which consists of a series of upsampling layers that progressively restore the spatial resolution of the image. Each upsampling layer is connected to the corresponding layer of the encoder via a shortcut connection to preserve detail while restoring the spatial structure.
3. Color decoderThe color decoder receives multi-scale visual features from the pixel decoder and generates color queries. These color queries are learned and used to represent the colors of different regions in the image. The color decoder matches the color queries with image features through a cross-attention mechanism, thereby generating colors that match the image content.
4. Cross-attention and self-attention mechanismsIn the color decoder, the cross-attention layer is used to establish the association between color queries and image features, while the self-attention layer further refines these color queries, making them more accurately reflect the semantic content of the image.
5. Color richness lossTo enhance the color richness of generated images, DDColor introduces a color richness loss function based on the standard deviation and mean of the color plane. This loss function encourages the model to generate more colorful and vivid images.
6. Fusion and OutputFinally, the outputs of the pixel decoder and color decoder are combined through a fusion module to generate the final color image. This fusion process is achieved through a simple dot product operation, followed by a 1×1 convolutional layer to generate the final AB (hue and saturation) channels.
7. Training and optimizationDuring training, the entire network optimizes the model by minimizing pixel loss, perceptual loss, adversarial loss, and color richness loss. These loss functions work together to ensure that the generated images are visually realistic while maintaining semantic consistency.

DDColor's official entry point

• Official GitHub project:https://github.com/piddnad/DDColor
• ModelScope runtime address:https://www.modelscope.cn/models/iic/cv_ddcolor_image-colorization/summary
• Replicate runtime address:https://replicate.com/piddnad/ddcolor

How to use DDColor

1. Visit DDColor's ModelScope community or the Replicate runtime address.
2. Upload your black and white image or select a sample photo.
3. Click "Execute Test" and wait for the image to be colored.