SketchVideo - Kuaishou, in collaboration with several universities, launched a sketch-based video generation and editing framework.

What is SketchVideo?

SketchVideo is a sketch-based video generation and editing framework developed by the University of Chinese Academy of Sciences, the Hong Kong University of Science and Technology, and the Kuaishou Keling team. It draws sketches on keyframes and combines them with text prompts to achieve fine-grained control over the spatial layout and motion of the video. The framework is based on a DiT video generation model and features an efficient sketch control network, including sketch control blocks and an inter-frame attention mechanism, which propagates sparse keyframe sketch conditions to all video frames. SketchVideo supports fine-grained editing of both real and synthetic videos. Based on a video insertion module and potential fusion technology, it ensures spatial and temporal consistency between new content and the original video, preserving details in unedited areas.

SketchVideo's main functions

• Video generationVideo generation based on sketches and text.
• Video editing: Draw sketches on keyframes to modify video content.
• Dynamic controlSupports motion interpolation and extrapolation.
• Details preservedPreserve details of unmodified areas during editing.
• High-efficiency generationMemory optimization enables rapid generation of high-quality videos.

The technical principles of SketchVideo

• Sketch Conditional NetworkBased on the DiT (Diffusion-based Transformer) video generation model, a specialized sketch conditional network is designed. The sketch conditional network contains multiple sketch control blocks and predicts the residual features of skipped DiT blocks. The sketch control blocks are uniformly distributed within the DiT blocks, injecting control signals into features at different levels.
• Inter-frame attention mechanismBased on an inter-frame attention mechanism, sketch conditions on keyframes are propagated to all video frames. The relationship between the hidden features of all frames and the hidden features of the control frames is calculated to achieve spatiotemporal propagation of sketch features.
• Video Insertion ModuleIn video editing tasks, a video insertion module was designed to analyze the relationship between the input sketch and the original video. The module generates new content that is spatially and temporally consistent with the original video, ensuring seamless integration between the edited and original videos.
• Potential fusion technologyDuring inference, noisy latent codes for the input video are generated based on DDIM (Denoising Diffusion Implicit Models). These latent codes are then replaced in the unedited areas, preserving the details of the original video and ensuring that the edited video appears visually natural and coherent.
• Hybrid training strategyThis approach employs a hybrid training strategy, combining image and video data for training. In the first phase of training, image and video data are used to accelerate convergence and address the issue of limited video data. In the second phase, video data is used to further optimize temporal coherence.

SketchVideo's project address

• Project official website:http://geometrylearning.com/SketchVideo
• GitHub repository:https://github.com/IGLICT/SketchVideo
• arXiv technical paper:https://arxiv.org/pdf/2503.23284

Application scenarios of SketchVideo

• Film and AdvertisingQuickly generate creative videos and effects previews, optimize the production process, and save time and costs.
• Education and Training: To assist in the production of teaching videos and training materials, thereby improving teaching effectiveness.
• Game developmentQuickly generate level previews and character animations to improve development efficiency.
• Personal creationEasily create personalized short videos and lower the barrier to entry for creators.
• Architectural DesignGenerate dynamic previews of architectural and interior designs to enhance client communication.