Sketch2Anim - A 2D sketch to 3D animation framework developed by the University of Edinburgh and other institutions.
Sketch2Anim is an automation framework developed by the University of Edinburgh in collaboration with Snap Research and Northeastern University. It can directly convert 2D sketch storyboards into high-quality 3D animations. Based on conditional motion synthesis technology, it uses 3D key poses,...
What is Sketch2Anim?
Sketch2Anim is an automated framework developed by the University of Edinburgh in collaboration with Snap Research and Northeastern University. It can directly convert 2D sketch storyboards into high-quality 3D animations. Based on conditional motion synthesis technology, it precisely controls animation generation using 3D key poses, joint trajectories, and action words. The framework comprises two core modules: a multi-conditional motion generator and a 2D/3D neural mapper. Sketch2Anim generates natural and fluid 3D animations, supports interactive editing, and greatly improves the efficiency and flexibility of animation production.
The main functions of Sketch2Anim
- Automatic conversionQuickly convert 2D sketch storyboards into 3D animations, reducing the tedious steps of manual production.
- High-quality animationGenerates smooth, natural 3D animations and supports smooth transitions between multiple segments.
- Interactive editingUsers can adjust the generated animation in real time, such as modifying key poses or trajectories, improving design flexibility.
- Improve efficiencyAutomated processes significantly improve animation production efficiency and support rapid design iteration.
The technical principles of Sketch2Anim
- Multi-condition motion generatorThis approach is based on a motion diffusion model, combining the trajectory control network (ControlNet) and a keypose adapter. The ControlNet embeds joint trajectories into the motion diffusion model to control global dynamic motion. The keypose adapter further refines local static poses based on the ControlNet, ensuring that the generated animation accurately matches the input key poses.
- 2D-3D Neural MapperIncludes 2D and 3D key pose and trajectory encoders. Aligns 2D sketches and 3D motion constraints in a shared embedding space, enabling direct 2D-to-3D animation control.
- Motion Fusion and OptimizationBased on the deterministic DDIM inverse process and guided denoising, multiple animation clips are smoothly transitioned to generate a complete animation. The generated animation is further optimized using a classifier-guided, second-order optimization method to better match the input 2D conditions.
- Data processing and augmentationThe model was trained using the HumanML3D dataset, which contains 14,646 actions and 44,970 corresponding motion annotations. Diverse 2D key poses and trajectories were generated based on camera view augmentation, joint perturbations, and body proportion perturbations, improving the model's generalization ability.
Sketch2Anim project address
- Project official website:https://zhongleilz.github.io/Sketch2Anim/
- arXiv technical paper:https://arxiv.org/pdf/2504.19189
Application scenarios of Sketch2Anim
- Film and animationUsed for pre-production previews in film and television production to improve efficiency.
- Game development: Accelerate the design of game action sequences and the production of story animations.
- AdvertisingTransform creative sketches into 3D animated advertisements to enhance visual effects and attract viewers.
- Education and TrainingTo help students better understand complex knowledge.
- VR/AR applicationsIt can directly convert 2D sketches into immersive 3D animations, enriching virtual scenes.