ToonCrafter - An open-source cartoon animation video frame interpolation tool from Tencent and other companies.

What is ToonCrafter?

ToonCrafter is an open-source tool for generating intermediate frames in cartoon animation videos, developed by researchers from Tencent AI Lab, the Chinese University of Hong Kong, and City University of Hong Kong. It breaks through the limitations of linear motion assumptions in traditional cartoon animation production, employing innovative generative interpolation technology to automatically generate intermediate dynamic frames using only two keyframe images, creating smooth animation effects. Compared to traditional animation methods that require frame-by-frame drawing, ToonCrafter significantly improves animation production efficiency, reduces animators' workload, shortens production time, and maintains the quality and creativity of the animation.

ToonCrafter Features

• Generative Cartoon Interpolation:ToonCrafter uses a deep learning model to automatically calculate and generate intermediate frames from two given keyframe images, achieving smooth transitions and dynamic effects in cartoon animation. This interpolation method not only fills in the gaps between frames but also simulates complex motion patterns, such as character movements and scene changes.
• Detail preservation and enhancement:ToonCrafterUtilizing advanced dual-reference 3D decoder technology, this ensures that image details are preserved and even enhanced during the generation of new frames. This mechanism is particularly suitable for cartoon animation, as it typically contains sharp lines and vibrant colors, requiring the avoidance of blurring and distortion of details during interpolation.
• Supports sketch-guided animation:ToonCrafter provides a sketch encoder that allows users to guide the generation of animations through simple sketch input. Users can specify aspects of the animation, such as the character's movement trajectory or a particular style of emphasis, to achieve personalized animation effects.
• Obstruction Area Handling:ToonCrafter effectively identifies and handles occlusion in animations, such as when one character or object partially or completely occludes another. The tool can reasonably infer the movement and changes of the occluded area, generating frames that conform to visual logic.
• Multiple application scenarios:ToonCrafter has a wide range of applications. It can be used to generate complete cartoon animation videos, as well as to generate animations from cartoon sketches and line drawings, and to color and stylize existing animations, providing diversity in animation production.
  

ToonCrafter's official website entrance

• Official project homepage:https://doubiiu.github.io/projects/ToonCrafter/
• GitHub repository:https://github.com/ToonCrafter/ToonCrafter
• Hugging Face Demo:https://huggingface.co/spaces/Doubiiu/tooncrafter
• arXiv technical paper:https://arxiv.org/abs/2405.17933v1

ToonCrafter's technical principles

1. Generative interpolation frameworkToonCrafter employs a novel generative interpolation method. Unlike traditional correspondence-based interpolation methods, it does not rely on explicit inter-frame correspondences but instead generates frames by learning the latent representations of video data.
2. Domain Adaptation (Toon Rectification Learning):Through its domain adaptation strategy, ToonCrafter can adapt the motion priors of real-world videos to the domain of cartoon videos, solving the domain difference problem and avoiding the accidental synthesis of non-cartoon content.
3. Dual Reference 3D Decoder:By employing a dual-reference 3D decoder, ToonCrafter is able to compensate for the loss of detail caused by latent spatial compression. This decoder injects detailed information from the input image into the latent representation of the generated frame through a hybrid attention residual learning (HAR) mechanism.
4. Hybrid Attention Residual Learning (HAR):During the decoding process, HAR injects the features of the input image into the shallow layer of the decoder through a cross-attention mechanism, while using residual learning in the deep layer to enhance the recovery of details.
5. Pseudo-3D Convolution:To enhance temporal coherence, ToonCrafter introduced pseudo-3D convolution in the decoder, which helps improve consistency between time-series frames.
6. Sketch Encoder:It provides a standalone sketch encoder that allows users to control the motion and style of the generated animation by inputting sketches, increasing the interactivity and controllability of the generation process.
7. Diffusion Models:ToonCrafter is based on a diffusion model, a generative model that progressively adds noise to data and then learns the inverse process to remove the noise and recover the data. In video generation, this allows for the generation of consecutive video frames from random noise.
8. Iterative denoising process:As each frame is generated, ToonCrafter refines the generated image step by step through an iterative denoising process, recovering a clear frame from the noise.
9. End-to-end training and optimization:The components of ToonCrafter are trained and optimized end-to-end to ensure the seamless operation of the entire interpolation process and the quality of the final generated video.
10. Multimodal input support:In addition to start and end frames, ToonCrafter also supports multimodal inputs such as sketches and reference images to enhance the expressiveness and control of generated animations.

How to use and experience ToonCrafter

Method 1: Online Demo Experience

1. Visit the online demo for Hugging Face.https://huggingface.co/spaces/Doubiiu/tooncrafter
2. Add an input image to Input Image
3. Enter the prompt word and adjust the Seed value, step size, FPS, etc.
4. Finally, click the Generate button to generate the video.

Method 2: Local code deployment

Developers can choose to deploy and run ToonCrafter locally. The specific steps are as follows:

1. Get code:Visit ToonCrafterProject PageAlternatively, you can use the Git command to clone or download the code to your local machine.git clone https://github.com/ToonCrafter/ToonCrafter.git
2. Environmental preparationInstall the required Python environment and dependencies using Anaconda.

   conda create -n tooncrafter python=3.8.5
   conda activate tooncrafter
   pip install -r requirements.txt

3. Download pre-trained model: Download pre-trained modelToonCrafter_512 modelandmodel.ckptPut incheckpoints/tooncrafter_512_interp_v1/model.ckpt.
4. Run from command line sh scripts/run.sh
5. Run the local Grado UI: You can also choose to run the local Grado UI and execute the command. python gradio_app.py
6. Then follow the steps above on the online version to input the image and set the parameters to generate the image.