MOFA-Video - Tencent's open-source controllable AI-generated video model

What is MOFA-Video?

MOFA-Video is an open-source model for controllable image-to-video generation, developed by researchers at Tencent AI Labs and the University of Tokyo. This technology utilizes a generative motion field adaptor to animate images and generate videos. Based on a pre-trained Stable Video Diffusion model, MOFA-Video achieves fine-grained control over motion during video generation using sparse control signals such as hand trajectories, facial marker sequences, or audio. MOFA-Video can not only use these control signals individually but also combine them to create more complex animations in a zero-shot manner, providing a novel and highly controllable solution for image-to-video animation.

MOFA-Video Features

• Track control animationUsers can manually draw trajectories on images to guide MOFA-Video in generating corresponding video animations. This feature is particularly suitable for scenarios requiring precise control over object or camera movement.
• Facial keypoint animationThe system uses facial key point data, such as markers obtained through facial recognition technology, to generate realistic facial expressions and head movement animations.
• Hybrid Control AnimationMOFA-Video combines trajectory control and facial key point control to achieve synchronized animation of facial expressions and body movements, creating complex multi-part animation effects.
• Audio-driven facial animationBy analyzing audio signals, MOFA-Video can generate facial animations synchronized with speech or music, such as lip-syncing.
• Video-driven facial animationUsing reference videos, MOFA-Video enables facial movements in still images to mimic those in videos, achieving dynamic facial expression reproduction.
• Zero-sample multimodal controlMOFA-Video supports zero-shot learning, meaning that different control signals can be combined and used without additional training, which greatly improves the flexibility and diversity of animation generation.
• Long video generation capabilityBy employing a periodic sampling strategy, MOFA-Video can generate longer video animations than traditional models, breaking through the frame rate limitation.
• User interface operationMOFA-Video provides a simple and easy-to-use user interface based on Gradio, allowing users to intuitively generate animations without needing professional programming skills.

MOFA-Video official website entrance

• Official project homepage:https://myniuuu.github.io/MOFA_Video
• GitHub repository:https://github.com/MyNiuuu/MOFA-Video
• Trajectory-based image animation Grado demo and model checkpoints:https://huggingface.co/MyNiuuu/MOFA-Video-Traj
• Gradio demo and blended control image animation checkpoints:https://huggingface.co/MyNiuuu/MOFA-Video-Hybrid

How MOFA-Video works

1. Sparse control signal generationDuring the training phase, the system generates sparse control signals using sparse motion sampling techniques. These signals may be trajectory-based animation control points, facial keypoint sequences, or other forms of motion indicators.
2. MOFA-Adapter DesignThe MOFA-Adapter is the core of the system; it's a specially designed network structure for converting sparse control signals into a dense motion field. This component includes:
   • S2D networkTransform sparse motion cues into dense motion fields.
   • Reference image encoderExtract multi-scale features from the reference image for subsequent motion field generation.
   • Feature Fusion EncoderThe motion field generated by the S2D network is combined with the features of the reference image encoder.
3. Multi-scale feature extractionThe reference image encoder processes the input reference image to extract multi-scale feature representations, which will be used for guidance and deformation in the subsequent video frame generation process.
4. The generation and application of sports fieldsThe S2D network generates dense motion fields based on sparse control signals. These motion fields are then used to spatially deform multi-scale features to simulate motion effects in videos.
5. Pre-trained SVD modelThe MOFA-Adapter is combined with a pre-trained Stable Video Diffusion (SVD) model to guide the generation of video frames using conditional features obtained from the MOFA-Adapter.
6. Space deformationUsing the generated motion field, the system spatially deforms the multi-scale features of the reference image to ensure that objects and scene elements in the video frame move according to the predetermined motion trajectory.
7. Video frame generationThe deformed features in the feature space are used to generate video frames. This process involves sampling from the latent space and progressively removing noise to reconstruct a clear video frame.
8. Multimodal control signal integrationMOFA-Video can process control signals from different sources and integrate them into a unified generation process to achieve complex animation effects.
9. Zero-shot learningOnce trained, the MOFA-Adapter can work together in different control domains without additional training, enabling fine-grained control over video generation.
10. Long video generation strategyTo generate longer videos, MOFA-Video employs a periodic sampling strategy, which addresses the issues of coherence and computational complexity in long video generation by grouping and overlapping frames in the latent space.