VideoMaker - A zero-sample custom video generation framework jointly developed by Zhejiang University, Tencent, and Huawei.

What is VideoMaker?

VideoMaker is an innovative project jointly developed by Zhejiang University, Tencent, and Huawei Noah's Ark Lab. It's a zero-shot custom video generation framework based on the Video Diffusion Model (VDM). Unlike traditional methods, VideoMaker can directly extract and inject thematic features from reference images without requiring additional models, enabling one-click generation of personalized video content. The framework leverages the inherent capabilities of VDM for fine-grained feature extraction and uses a spatial self-attention mechanism for feature injection, ensuring both the diversity and thematic consistency of the generated videos. While maintaining video diversity, VideoMaker ensures a high degree of consistency with the thematic features in the reference images, bringing great convenience and flexibility to personalized video creation.

VideoMaker's main functions

• Fine-grained feature extractionVideoMaker can directly leverage the inherent capabilities of the Video Diffusion Model (VDM) to extract richly detailed subject features from provided reference images.
• Feature InjectionThrough VDM's spatial self-attention mechanism, VideoMaker can effectively inject extracted thematic features into each frame of video during the video generation process, ensuring that the video content maintains a high degree of consistency with the reference image.
• Video content generationWhile maintaining consistency with the theme in the reference image, VideoMaker also ensures the diversity and dynamism of the generated videos, avoiding monotonous and repetitive content.
• No additional training requiredVideoMaker does not require complex retraining or parameter tuning of VDM; simple fine-tuning is all that is needed to activate the model’s inherent power and achieve customized video generation.
• High fidelityVideoMaker generates videos with high fidelity, accurately presenting the appearance and dynamic characteristics of both people and objects in the video.

VideoMaker's technical principles

• Video Diffusion Model (VDM)VDM is the core of VideoMaker. It learns the distribution of video data and generates videos by progressively denoising.
• Feature extractionVideoMaker directly inputs reference images into VDM and uses VDM's pre-trained knowledge to perform fine-grained feature extraction. These features are highly compatible with VDM's knowledge system.
• Spatial self-attention mechanismVideoMaker achieves feature injection by modifying the spatial self-attention calculation in VDM. This mechanism allows the model to bidirectionally interact with the generated content and thematic features when generating each frame of video, ensuring the accurate representation of thematic features.
• Guiding information to identify lossesTo distinguish between reference information and generated content during training, VideoMaker designed a loss function to help the model more accurately identify and process reference information.
• Training and Reasoning ParadigmDuring the training phase, VideoMaker effectively injects topic information by fine-tuning the spatial self-attention layer and motion block parameters of the VDM. During inference, the output corresponding to the reference information is directly discarded to ensure that the generated video is free from noise interference.

VideoMaker project address

• Project official website:https://wutao-cs.github.io/VideoMaker/
• arXiv technical paper:https://arxiv.org/pdf/2412.19645

Application scenarios of VideoMaker

• Film and television productionIn the pre-production of movies or TV series, VideoMaker can be used to generate specific scenes and actions to help directors and producers preview and plan.
• Virtual idolsGenerate diverse video content for virtual idols to enhance their interaction and influence with fans.
• Product ShowcaseBusinesses can use VideoMaker to showcase how their products look in different environments, such as how furniture looks in different decorating styles, or how cars perform under different road conditions.
• Customized advertisingCustomize personalized advertising videos based on the target customer group to enhance the appeal and marketing effectiveness of the ads.
• Instructional VideosTeachers can create engaging teaching videos, such as demonstrations of physical and mechanical principles, displays of chemical experimental phenomena, and recreations of historical events, to enhance the learning experience.
• Animation and game designGame designers can quickly generate preliminary character animation videos by inputting character sketches and motion design concept art, and evaluate the smoothness and visual effects of character movements.