ControlFoley - Xiaomi's open-source controllable video and audio effects generation model - AiBoss

What is ControlFoley?

ControlFoley is an open-source, controllable video audio generation model from Xiaomi, addressing the controllability challenges in the V2A (Video-to-Action) domain. The model uniformly supports three types of video dubbing tasks: text-guided, text-controlled, and reference audio-controlled. Through its self-developed spatiotemporal audio-video encoder CAV-MAE-ST, time-timbre decoupling, and modal robust training, it achieves comprehensive improvements in semantic alignment, time synchronization, and audio quality. The model has achieved state-of-the-art (SOTA) performance on multiple benchmarks, and its code and model weights are publicly available.

ControlFoley's main functions

TV2A (Text-guided Video Dubbing)Generate synchronized sound effects based on video and text prompts, with text supplementing the semantic meaning of the visuals and sounds.
TC-V2A (Text-controlled video dubbing)When there is a semantic conflict between text and video, the text intent should be followed first, while maintaining time synchronization.
AC-V2A (Reference Audio Control Video Dubbing)The timbre style is controlled based on the reference audio without disrupting the video rhythm.

ControlFoley's technical principles

Joint visual coding: Self-developed CAV-MAE-ST spatiotemporal audio and video encoder, focusing on the spatiotemporal correspondence of audio and video, enhancing the understanding of action rhythm and time synchronization, and combining with CLIP to take into account both semantics and synchronization.
Time-timbre decouplingSuppress timing information in the reference audio, preserve global timbre characteristics, and avoid the reference audio interfering with video synchronization.
Modal robust trainingRandomized modal dropout + unified multimodal representation alignment adapts to various input combinations; semantic consistency is improved through REPA alignment targets.

How to use ControlFoley

Obtain open source resourcesVisit the GitHub repository to download the code and model weights, or try it out directly using the official online demo.
Environment configurationInstall the dependencies according to the repository instructions, and configure the Python runtime environment and necessary audio and video processing libraries.
Select task modeChoose one of three task types based on your creative needs: TV2A (text guidance), TC-V2A (text control), or AC-V2A (reference audio control).
Prepare input conditionsImport video files; if TV2A is selected, add text prompts; if TC-V2A is selected, enter text commands that conflict with the screen; if AC-V2A is selected, upload reference audio files.
Perform generative reasoningRun the model inference script, and ControlFoley will generate sound effects synchronized with the video based on joint visual encoding and time-timbre decoupling mechanism.
Export and Post-processingAfter acquiring the generated audio, the final audio track is output through VAE Decoder and Vocoder, and then combined with the video to complete the dubbing.

ControlFoley's core advantages

Unified frameworkA single model covers three types of tasks: text guidance, text control, and reference audio control, eliminating the need to switch between multiple tools.
Precise synchronizationOur self-developed CAV-MAE-ST spatiotemporal encoder enhances audio and video timing understanding, and its audio-visual alignment accuracy surpasses that of open-source competitors.
Tone decouplingThe time-timbre decoupling technology ensures that the reference audio only affects the timbre style and does not interfere with the original rhythm of the video.
Robust controlRandomized modal dropout and uniform representation alignment training enable the model to output stably under single-modal or multimodal inputs.
Open source SOTAIt leads in semantic alignment and sound quality across multiple benchmarks, including VGGSound-Test and Kling-Audio-Eval.

ControlFoley's project address

Project official website: https://yjx-research.github.io/ControlFoley_web_page/
GitHub repositoryhttps://github.com/xiaomi-research/controlfoley
HuggingFace model libraryhttps://huggingface.co/YJX-Xiaomi/ControlFoley
arXiv technical paperhttps://arxiv.org/abs/2604.15086

Comparison of ControlFoley's similar products

Comparison Dimensions	ControlFoley	MMAudio	HunyuanVideo-Foley
Task Coverage	Unified support for three types of controllable tasks: TV2A, TC-V2A, and AC-V2A.	Mainly supports basic TV2A video dubbing.	It primarily supports basic video dubbing for TV2A.
Text conflict handling	Strong: In conflict scenarios, DeSync is only 0.36-0.38, prioritizing text intent.	Weakness: Text is easily obscured by visual information.	Weakness: Limited text control capabilities
Reference audio control	Supports time-timbre decoupling without disrupting synchronization.	Not supported	Not supported
Audio-visual synchronization	Advantages: CAV-MAE-ST enhances spatiotemporal correspondence.	good	good
Open source status	The code, weights, and technical reports are all open source.	open source	open source

Application scenarios of ControlFoley

Short video creationAdd customized sound effects to silent footage to match the creator's intent and avoid biases caused by the model's automatic guessing.
Animation and GamesGenerate specific styles of impact sounds or ambient sounds for character actions, such as replacing a normal knocking sound with a drumbeat.
Post-production of film and televisionThe sound effects and timbre style of the entire film are unified based on the reference audio materials to maintain the consistency of sound for the brand or series of works.
Advertising and MarketingQuickly generate synchronized voiceovers that match the brand's tone based on text commands, enhancing rhythm and emotional expression.
Self-media live streaming: To supplement live stream clips or derivative videos with multimodal, controllable, immersive audio, thereby improving the completeness of the content.