InstructAV2AV - An open-source audio and video co-editing model developed by Zhiyuan Technology in collaboration with Peking University. - AiBoss

What is InstructAV2AV?

InstructAV2AV is an audio-visual co-editing model jointly developed by the Beijing Academy of Artificial Intelligence and Peking University. Users only need a single natural language command to simultaneously edit video footage and corresponding audio during the end-to-end generation process, eliminating the need for manual masking or step-by-step processing. The model supports dialogue modification, character replacement, object insertion and deletion, while maintaining background and ambient sound and ensuring audio-visual synchronization.

The main functions of InstructAV2AV

Voice modification for identity preservation: Replace only the dialogue while preserving the character's appearance.
Audio and video instance replacementReplace the specified character in the image along with their voice and lines.
Instance InsertionAdd new objects to the screen and automatically generate synchronized sounds.
Instance removal: Deletes the specified object and its corresponding sound in the audio track.
Attribute combination editingCharacter appearance, speech content, and voice can be modified independently and combined freely.

The technical principle of InstructAV2AV

Source latent variable splicingThe latent variables and noise of the source video and source audio are spliced along the channel dimension, so that the source content becomes the structural condition of the entire denoising process, constraining the model to retain the background, irrelevant objects and ambient sound, and avoiding "changing one target and redrawing the whole world".
SIGA Gated AttentionSource-Instruction Gated Attention allows each token to interact with both source and instruction features simultaneously. Through learnable soft gating, it determines how much to modify and how much to retain position by position, achieving a dynamic balance between content preservation and instruction execution.
Dual-stream interaction architectureBased on the Ovi symmetric two-stream diffusion Transformer, after the video and audio branches are modeled for their own attention, they exchange features through bidirectional cross-modal attention, so that visual motion and sound events are mutually constrained and kept synchronized.
Two-stage training: Disable cross-modal attention and train single-modal editing capabilities separately, then restore the complete architecture for joint fine-tuning to learn the fine-grained correspondence between sound and image.
Automated data pipelineThe InsAVE-80K dataset is constructed through three stages: material selection, editing and target synthesis, and five-dimensional automatic evaluation plus manual review, thus solving the problem of scarce paired training data.

How to use InstructAV2AV

Cloning codeClone the InstructAV2AV code repository from GitHub to your local machine.
Configure environmentConfigure the Python environment and install the required dependencies according to the repository instructions.
Download weightDownload the InstructAV2AV model weights from Hugging Face.
Prepare materialsPrepare a video clip with sound as input.
Input commandEnter a natural language editing command to describe what you want to modify.
Running inferenceRun the inference script and wait for the model to generate the edited video and audio.
Save resultsView the output and save the edited audio and video files.

The project address for InstructAV2AV

Project official website：https://hjzheng.net/projects/InstructAV2AV/
GitHub repository：https://github.com/suimuc/InstructAV2AV
HuggingFace model library：https://huggingface.co/suimu/InstructAV2AV
arXiv technical paper：https://arxiv.org/pdf/2605.18467

Comparison of InstructAV2AV with similar competitors

Comparison items	InstructAV2AV	Ovi
Product Positioning	Instruction-driven audio and video co-editing model	Audio and video joint generation model
Core Competencies	Modify the visuals and audio based on the original video.	Generate video and audio from scratch
Input method	Source video + source audio + natural language commands	Text prompts (generated from scratch)
Background Preservation	Precisely preserve irrelevant background and ambient sounds	No source content retention involved
Audio-visual synchronization	Bidirectional cross-modal attention ensures post-edit synchronization	Synchronous generation, scene editing is not supported.
Technical Architecture	Based on the modified Ovi dual-stream diffusion Transformer, source latent variable splicing and SIGA gating were added.	Symmetrical two-stream diffusion Transformer
Applicable Scenarios	Dialogue modification, character replacement, object insertion and removal	Creative generation of text to audio and video

Application scenarios of InstructAV2AV

Post-production of film and televisionFilm and television post-production staff use it to replace actors' lines and synchronize lip movements, reducing the cost of reshoots.
Short video productionShort video creators can use it to modify the characters and voices in their footage with just one sentence, improving their creative efficiency.
Virtual HumanThe virtual human operations team uses it to adjust the appearance, tone of voice, and content of speech of digital humans.
Advertising CreativityAdvertising professionals use it to generate promotional videos in batches with different characters and lines.
Interactive content productionIt is used by game developers and educators to dynamically generate multimedia materials with synchronized audio and video.