Xiaomi open-sources ControlFoley, a controllable video and audio generation model.

Xiaomi's large-scale model application team has open-sourced ControlFoley, a controllable video audio generation model that uniformly supports three types of tasks: text-guided video dubbing, text-controlled video dubbing, and reference audio-controlled video dubbing. Through joint visual encoding, temporal-timbre decoupling, and modal robust training, the model addresses issues such as weak text control and difficulty in synchronizing reference audio in existing solutions, achieving state-of-the-art (SOTA) performance on multiple benchmarks, including VGGSound-Test.