FlexiAct - An action transfer model jointly launched by Tsinghua University and Tencent
FlexiAct is a novel action transfer model jointly developed by Tsinghua University and Tencent ARC Labs. Given a target image, FlexiAct can transfer actions from a reference video to the target subject, even in situations with significant spatial structural differences...
What is FlexiAct?
FlexiAct is a novel motion transfer model jointly developed by Tsinghua University and Tencent ARC Labs. Given a target image, FlexiAct can transfer actions from a reference video to a target subject, achieving accurate motion adaptation and appearance consistency in heterogeneous scenes with significant spatial differences or cross-domain challenges. The model introduces a lightweight RefAdapter module and a Frequency-Aware Action Extraction (FAE) module to overcome the limitations of existing methods in terms of layout, viewpoint, and skeletal structure differences, maintaining identity consistency. FlexiAct performs exceptionally well in motion transfer for both humans and animals, demonstrating broad application prospects.
Main functions of FlexiAct
- Cross-subject action transferSupports transferring actions from one character to another, or from a character to an animal.
- Maintain consistent appearanceDuring the migration process, ensure that the appearance of the target subject (such as clothing, hairstyle, etc.) remains consistent with the original target image.
- Flexible spatial structure adaptationIt can achieve natural motion transfer even when there are differences in layout, viewpoint and skeletal structure between the reference video and the target image.
FlexiAct's technical principles
- RefAdapter (Spatial Structure Adapter)RefAdapter is a lightweight image conditional adapter, primarily designed to address the spatial structural differences between reference videos and target images. During training, it randomly selects video frames as conditional images to maximize spatial structural diversity. This allows the model to adapt to different poses, layouts, and viewpoints while maintaining visual consistency. By injecting a small number of trainable parameters (such as the LoRA module), it achieves flexible spatial adaptation within the MMDiT layer of CogVideoX-I2V, avoiding the strict constraints of traditional methods.
- Frequency-aware motion extractionFAE is an innovative action extraction module that extracts actions directly during the denoising process, independent of a separate spatiotemporal architecture. FAE observes that at different time steps in denoising, the model pays varying degrees of attention to motion (low frequency) and appearance details (high frequency). In earlier time steps, the model focuses more on motion information; in later time steps, it focuses more on appearance details. FAE dynamically adjusts attention weights, prioritizing motion extraction in earlier time steps and focusing on appearance details in later time steps, achieving precise action extraction and control.
FlexiAct's project address
- Project official website:https://shiyi-zh0408.github.io/projectpages/FlexiAct/
- GitHub repository:https://github.com/shiyi-zh0408/FlexiAct
- HuggingFace model library:https://huggingface.co/shiyi0408/FlexiAct
- arXiv technical paper:https://arxiv.org/pdf/2505.03730
Application scenarios of FlexiAct
- Film and television productionQuickly generate realistic character movements and reduce shooting costs.
- Game developmentGenerate diverse actions for game characters to enhance the gaming experience.
- Advertising and MarketingGenerate virtual spokesperson actions to enhance the appeal of advertisements.
- Education and TrainingGenerate teaching and rehabilitation training movements to assist learning and recovery.
- Entertainment and InteractionIt supports users in creating interesting videos, enhancing their entertainment experience.