Diffutoon - Alibaba's AI-powered framework for converting videos into cartoon styles

What is Diffutoon?

Diffutoon, developed by researchers from Alibaba and East China Normal University, is an AI framework for converting videos into cartoon/anime styles. Based on a diffusion model and editable cartoon coloring technology, it can transform realistic videos into anime-style animations. This technology achieves high-resolution and long-duration video rendering by breaking down tasks into stylization, consistency enhancement, structure guidance, and coloring. Diffutoon also features content editing capabilities, allowing adjustments to video details based on text prompts. It maintains a high degree of visual quality and consistency during video processing, achieving efficient and high-quality animation processing.

Diffutoon Features

• Cartoon video renderingDiffutoon uses a diffusion model to transform realistic videos into cartoon or anime-style videos, achieving a visually flattened and stylized effect. The transformation involves not only changes in color and texture but also artistic manipulation of elements such as light and shadow, and outlines, to mimic the visual effects of hand-drawn animation.
• High resolution supportDiffutoon is capable of handling high-resolution video, supporting resolutions of at least 1536×1536 pixels, ensuring clarity and detail even on enlarged or high-definition display devices, making it suitable for high-quality video production and presentation needs.
• Video editingUsers can edit video content using text prompts, and Diffutoon can recognize and adjust specific parts of the video based on these prompts. The editing features support modifications to the appearance and attributes of characters and scene elements, such as changing clothing colors and adjusting character expressions.
• Inter-frame consistencyThrough specific algorithms and technologies, Diffutoon ensures that each frame in a video sequence maintains consistency in style and content, avoiding issues such as flickering, color abrupt changes, or content inconsistencies that may occur during video playback, thus improving the viewing experience.
• Structure retentionDuring the video stylization process, Diffutoon is able to identify and preserve key structural information of the video, such as the outlines of characters and the edges of objects, ensuring that the main content and shape of the video remain clearly discernible even after stylization.
• Automatic coloringDiffutoon features an automatic coloring function that automatically selects appropriate colors for filling based on the video content and style requirements. Automatic coloring not only improves production efficiency but also ensures color harmony and visual appeal, resulting in a more aesthetically pleasing final video.

Diffutoon's official website entrance

• Official project homepage:https://ecnu-cilab.github.io/DiffutoonProjectPage/
• GitHub repository:https://github.com/modelscope/DiffSynth-Studio
• arXiv technical paper:https://arxiv.org/abs/2401.16224

Diffutoon's technical principles

1. Application of diffusion modelDiffutoon utilizes a diffusion model as the core technology for image synthesis. By learning the distribution characteristics of images and videos in the dataset, it achieves the transformation from a high-dimensional latent space to image data.
2. Multi-module noise reductionDiffutoon has built a multi-module denoising model that combines technologies such as ControlNet and AnimateDiff to address controllability and consistency issues in video.
3. Stylization, consistency enhancement, structural guidance, and coloringDiffutoon breaks down the cartoon coloring problem into four subproblems, each solved by a specific model:
   • StylizationAnime stylization is achieved using a personalized Stable Diffusion model.
   • Consistency EnhancementMaintain content consistency between video frames by inserting AnimateDiff-based motion modules into UNet.
   • Structure guidanceUse the ControlNet model to extract and preserve structural information, such as contours, from the video.
   • ColoringAnother ControlNet model is used for colorization to improve video quality, even if the input video resolution is low.
4. Sliding window methodThe sliding window method is used to iteratively update the potential embeddings of each frame, which helps to process long videos and maintain the coherence between frames.
5. Edit BranchIn addition to the main cartoon coloring pipeline, Diffutoon also includes an editing branch that generates editing signals based on text prompts, which are then provided to the main pipeline as color video.
6. Efficient attention mechanismsBy introducing Flash Attention, GPU memory usage is reduced, improving the efficiency of processing high-resolution videos.
7. Classifier free guidanceThe visual quality is optimized by using a classifier-guided mechanism and text prompts.
8. DDIM SchedulerThe DDIM (Denoising Diffusion Implicit Models) scheduler is used to control the video generation process, balancing generation quality and speed.
9. Post-processing methodsPost-processing techniques such as FastBlend are employed to further enhance the long-term consistency and visual effects of the video.