UniEdit - A unified video editing framework that requires no training or optimization

What is UniEdit?

UniEdit is an innovative video editing framework developed by researchers from Zhejiang University, Microsoft Research, and Peking University. It allows users to edit the motion and appearance of videos without needing to fine-tune models. The framework's core advantage lies in its ability to simultaneously handle editing tasks in both the temporal (e.g., motion changes) and spatial (e.g., stylization, object replacement, background modification) dimensions of video.

UniEdit official website entrance

• Official project homepage:https://jianhongbai.github.io/UniEdit/
• arXiv research paper:https://arxiv.org/abs/2402.13185
• GitHub repository:https://github.com/JianhongBai/UniEdit(Source code is pending release)

UniEdit Features

• Video object action editingUniEdit can edit the actions of objects in a video based on text commands. For example, it can edit the action of a raccoon playing guitar in a video to make it eat an apple or wave.
• Video stylizationThis framework supports style transfer for videos, meaning users can change the visual style of a video through text descriptions without altering the video's content, such as converting a video to an oil painting style.
• Video background modificationUniEdit allows users to modify and change the background in a video. For example, you can place the people in the video in a completely different scene.
• Video object replacementThe UniEdit framework supports the replacement of rigid and non-rigid objects, allowing users to replace static or dynamic objects in videos.
• No training or fine-tuning requiredUniEdit requires no additional training or fine-tuning, greatly simplifying model deployment and use, allowing users to quickly get started with video editing.

How UniEdit works

1. Inversion processing:UniEdit first performs an inversion process on the input video, converting it into a random noise representation. This process typically involves an inversion using a pre-trained diffusion video generation model (such as LaVie), which gradually transforms the video from a noisy state back to its original state through a series of denoising steps.
2. Generate edit path:Building upon the inversion process, UniEdit uses a pre-trained UNet model for denoising to generate the edited video. This process is performed under conditions provided by target text prompts to ensure that the generated video content matches the user's editing intent.
3. Assisted Reconstruction Branch:To preserve the unedited content of the source video, UniEdit introduces an auxiliary reconstruction branch. This branch starts with the same inverse noise but performs denoising under the given source video text cues to reconstruct the original video frames. Features from the reconstruction process are injected into the spatial self-attention layer of the main editing path to maintain content consistency.
4. Assisted Motion Reference Branch:To enable motion/action editing, UniEdit introduces an auxiliary motion reference branch that generates motion features given a target text cue. These features are then injected into the main editing path via a temporal self-attention layer to guide motion changes in the video.
5. Content retention and motion injection:In the main editing path, UniEdit preserves the content of the source video by replacing the value features of the spatial self-attention layer. Simultaneously, it achieves motion editing by injecting motion features into the temporal self-attention layer.
6. Space structure control:In appearance editing, UniEdit preserves the spatial structure of the source video by replacing the query and key features of the spatial self-attention layer in the main editing path. This helps maintain the layout and position of objects when changing the video style or appearance.
7. Text-guided editing:Users provide text descriptions to guide the video editing process. UniEdit parses these text descriptions and transforms them into guiding signals for video editing, thereby achieving the desired editing effect for the user.