VidSketch - A video animation generation framework developed by Zhejiang University

What is VidSketch?

VidSketch is an innovative video generation framework developed by the State Key Laboratory of CAD&CG and the School of Software at Zhejiang University. It generates high-quality video animations based on hand-drawn sketches and simple text prompts. VidSketch dynamically adjusts the guidance intensity of sketches based on a "hierarchical sketch control strategy," adapting to users with different drawing skills. It enhances the spatiotemporal consistency of videos through a "spatiotemporal attention mechanism," solving the problem of inter-frame coherence. VidSketch lowers the technical barrier to video creation, allowing ordinary users to easily create high-quality animations and meet diverse artistic needs.

VidSketch's main functions

• High-quality video animation generationIt supports the generation of high-quality, spatiotemporally coherent video animations from hand-drawn sketches and simple text descriptions.
• Adapt to different drawing skillsIt automatically assesses the level of abstraction in a sketch and dynamically adjusts the guidance intensity during the generation process to suit the drawing skills of users ranging from beginners to professionals.
• Enhanced spatiotemporal consistencyImprove the continuity and smoothness between video frames, avoiding tearing or flickering issues.
• Diverse style supportIt supports multiple visual styles (such as realism, fantasy, pixel art, etc.) to meet the aesthetic needs of different users.
• Low barrier to entry for creationNo professional video editing skills are required; ordinary users can easily create high-quality video animations.

VidSketch's technical principles

• Video generation based on diffusion modelThe framework uses Video Diffusion Models (VDMs). Diffusion models are based on the generation of images or videos by progressively removing noise. VidSketch extends this process to video generation and introduces spatiotemporal consistency control.
• Hierarchical sketch control strategyBased on quantitative analysis of the sketch's continuity, connectivity, and texture details, the system assesses the sketch's level of abstraction and dynamically adjusts the guidance intensity during the generation process accordingly. This ensures that even highly abstract user-provided sketches maintain high quality and consistency in the generated videos.
• Spatiotemporal attention mechanism:
• To address the issue of inter-frame continuity in video, VidSketch introduces a spatiotemporal attention mechanism. Attention weights are calculated based on the query (Q) of the current frame and the key/value pairs (K/V) of the previous, second, and first frames, enhancing spatiotemporal consistency between video frames and preventing tearing or flickering in the generated video.
• Collaborative guidance of sketches and textUser-input hand-drawn sketches and text prompts work together to guide video generation. The sketches provide visual structure, while the text prompts provide semantic information. The combination of the two ensures that the generated video conforms to the visual layout of the sketches and satisfies the content described in the text.

VidSketch project address

• Project official website:https://csfufu.github.io/vid_sketch
• GitHub repository:https://github.com/CSfufu/VidSketch
• arXiv technical paper:https://arxiv.org/pdf/2502.01101v1

Application Scenarios of VidSketch

• Creative DesignDesigners can quickly realize their design ideas by generating creative videos using hand-drawn sketches and simple text.
• Teaching aidsTeachers use sketches and teaching content to generate teaching animations, helping students understand knowledge more intuitively.
• Advertising productionMarketers can use sketches and product descriptions to generate ad videos, efficiently creating engaging advertising content.
• social mediaContent creators use sketches and creative text to generate personalized videos, enhancing the interactivity of social media.
• Animation and FilmAnimators use sketches to quickly generate animation prototypes, validate ideas, and improve production efficiency.