VACE - A video generation and editing framework launched by Alitongyi

What is VACE?

VACE (Video Creation and Editing) is a one-stop video generation and editing framework launched by Alibaba Tongyi Labs. It integrates multiple video tasks (such as reference video generation, video-to-video editing, and masking editing) into a unified model, enabling efficient content creation and editing. The core of VACE lies in the Video Condition Unit (VCU), which integrates multiple modal inputs such as text, images, videos, and masks into a unified conditional unit, supporting flexible combinations of various tasks. The open-source Wan2.1-VACE-1.3B supports 480P resolution, while Wan2.1-VACE-14B supports both 480P and 720P resolutions. Experiments show that VACE achieves performance comparable to task-specific models across multiple tasks, providing broader application possibilities and opening new paths for video content creation.

Main functions of VACE

• Text to video generationGenerate video based on text prompts.
• Reference to video generation: Generate video by combining text and reference images.
• Video extensionGenerate new beginnings or endings based on existing video clips.
• Video to video editingPerform overall style transformation on the input video (such as colorization and stylization).
• Masking video editingEditing is performed in a specified area, such as inpainting or outpainting.
• Main body removal and reconstructionRemove a specific subject from the video and fill the background.
• Task Combination and InnovationThis approach combines various tasks, such as reference generation and subject replacement, and pose control and video extension. Video generation is controlled based on conditions such as pose, depth, and optical flow.

VACE's technical principles

• Video Condition Unit (VCU)VCU is the core input interface of VACE, used to integrate inputs from multiple modalities (such as text, images, videos, and masks). It passes inputs to the model based on a unified format, supporting flexible combinations of various tasks.
• Context Adapter StructureBased on the Context Adapter, concepts from different tasks (such as editing areas and reference content) are injected into the model. Formal representations of time and space dimensions are provided to adapt to different task requirements.
• diffusion modelIt is built on a diffusion model (such as Diffusion Transformer) and generates high-quality video content by progressive denoising.
• Multimodal input processingIt supports multiple input modalities such as text, images, videos, and masks, mapping them to a unified feature space based on a specific encoder. For example, the video VAE (Variational Autoencoder) handles video input, while segmentation and masking operations handle local editing tasks.
• Training and optimization strategiesBased on a stepwise training strategy, it starts with basic tasks (such as repair and expansion) and gradually expands to complex tasks (such as combined tasks). It supports full model fine-tuning and context adapter fine-tuning, the latter of which converges faster and supports plug-in functionality.

VACE's project address

• Project official website:https://ali-vilab.github.io/VACE-Page/
• GitHub repository:https://github.com/ali-vilab/VACE
• HuggingFace model library:https://huggingface.co/collections/ali-vilab/vace
• arXiv technical paper:https://arxiv.org/pdf/2503.07598

VACE application scenarios

• Creative video generationQuickly generate creative video content such as advertisements and animations from text or images.
• Video Restoration and EnhancementRepairing old videos, filling in missing parts of the footage, or improving the style of the video.
• High-efficiency video editingIt enables complex editing tasks such as subject replacement and animation addition.
• Video extensionGenerate new clips for short videos to extend the video content.
• Interactive video creationGenerate personalized videos based on user input (such as poses or sketches).