AniDoc - A 2D animation coloring AI model that automatically converts sketch sequences into color animations based on a video diffusion model.
AniDoc is a simplified 2D animation coloring AI model jointly developed by the Hong Kong University of Science and Technology, Ant Group, Nanjing University, Zhejiang University, and the University of Hong Kong. Based on a video diffusion model, it automatically converts sketch sequences into color animations, following a reference...
What is AniDoc?
AniDoc is a simplified 2D animation coloring AI model jointly developed by the Hong Kong University of Science and Technology, Ant Group, Nanjing University, Zhejiang University, and the University of Hong Kong. Based on a video diffusion model, it automatically converts sketch sequences into colored animations while adhering to reference character designs. The model uses correspondence matching technology to handle differences between the character design and the sketches, achieving accurate color and style transfer. AniDoc supports binarized sketch input, improves model robustness through background enhancement, and reduces the need for manual drawing of intermediate frames using a sparse sketch training strategy. AniDoc improves color accuracy and inter-frame consistency, reduces the labor costs of animation production, accelerates content creation processes, and is suitable for animation and various digital art fields.
AniDoc's main functions
- Automatic coloringAutomatically converts sketch sequences into color animations, reducing the amount of manual coloring work.
- Corresponding matchBased on correspondence matching technology, it handles inconsistencies between the reference character design and the sketch, such as differences in angle, scale, and pose.
- Background EnhancementBy employing background enhancement strategies, the model's ability to distinguish between foreground and background is improved, thereby enhancing the coloring quality.
- Sparse sketching trainingUse sparse sketch input to reduce the drawing work of intermediate frames, and use keyframe interpolation to generate smooth animation.
- Multi-style adaptationIt adapts to reference images of different styles, providing a consistent coloring effect for different sketches of the same character.
AniDoc's Technical Principles
- Video diffusion model: The ability to capture temporal dynamics and visual coherence based on a video diffusion model.
- Corresponding guidance coloring: By using an explicit mapping mechanism, color and style information from the reference character design is integrated into the sketch, improving color accuracy and consistency.
- Binarization and Background EnhancementBinarize the conditional sketch and use a background enhancement strategy to improve the robustness of the model.
- Sparse sketching trainingA two-stage training strategy is adopted: first, the ability to color is learned, and then the interpolation ability is learned by removing intermediate sketches, thereby reducing the dependence on dense sketches.
- 3D U-Net ArchitectureThis includes a denoising 3D U-Net designed for video generation, which integrates reference image information into the coloring process.
AniDoc's project address
- Project official website:yihao-meng.github.io/AniDoc
- GitHub repository:https://github.com/yihao-meng/AniDoc
- arXiv technical paper:https://arxiv.org/pdf/2412.14173
AniDoc Application Scenarios
- Animation ProductionIn the traditional 2D animation production process, the coloring of character line art is completed automatically, reducing the time and labor intensity of animators manually coloring.
- Animation IndustryFor animation production companies, this means improving the efficiency of animation production, reducing costs, and maintaining consistency in character colors across different scenes.
- Game developmentIn game development, it enables the rapid generation of game character animations, especially in situations requiring a large amount of character animation resources.
- Digital art creationArtists and designers can experiment with and explore different color schemes, quickly preview the coloring effects of sketches, and accelerate the creative process.
- Education and trainingIn animation education and training, it serves as a teaching tool to help students understand the animation coloring process and practice animation production techniques.