project
ACTalker - An end-to-end video distribution framework jointly launched by HKUST, Tencent, and Tsinghua University.
ACTalker is an end-to-end video diffusion framework for generating realistic speaker head videos. It supports single-signal or multi-signal control, such as audio and facial expressions. The core architecture includes a parallel Mamba structure, utilizing multiple branches to...
What is ACTalker?
ACTalker is an end-to-end video diffusion framework for generating realistic speaker head videos. It supports single-signal or multi-signal control, such as audio and facial expressions. Its core architecture includes a parallel Mamba structure, using multiple branches to control facial regions separately with different driving signals. Based on gating mechanisms and mask discarding strategies, it achieves flexible and natural video generation. On the CelebV-HQ dataset, ACTalker demonstrates excellent Sync-C and Sync-D scores of 5.317 and 7.869 respectively, and an FVD-Inc score of 232.374, showcasing good audio synchronization and video quality.
ACTalker's main functions
- Multi-signal control and single-signal controlACTalker supports both multi-signal and single-signal control, and can use various signals such as audio and facial expressions to drive the generation of the speaker's head video.
- Naturally Harmonious Video GenerationThe Mamba architecture enables drive signals to manipulate feature markers across both time and space in each branch, ensuring natural temporal and spatial coordination of the controlled video.
- High-quality video generationExperimental results show that ACTalker can generate natural and realistic facial videos. Under multi-signal control, the Mamba layer can seamlessly integrate multiple driving modalities and generate videos without conflict.
ACTalker's technical principles
- Parallel Mamba structureACTalker employs a parallel Mamba architecture, containing multiple branches. Each branch utilizes separate driving signals (such as audio and facial expressions) to control a specific facial region. This allows signals from different modalities to act simultaneously on the video generation process without interference, achieving multi-signal control.
- Gating mechanismGating mechanisms are applied to all branches, randomly enabled or disabled during training, and manually adjustable as needed during inference. This gating mechanism provides flexible control over video generation, allowing the selection of a single or multiple signals for driving the process under different circumstances.
- Mask-Drop StrategyACTalker introduces a mask discarding strategy, allowing each driving signal to independently control its corresponding facial region. During training, the strategy enhances the effectiveness of driving signals, improves the quality of generated content, and prevents control conflicts by randomly discarding feature labels unrelated to the controlled region.
- State-space modeling (SSM)To ensure natural temporal and spatial coordination in controlled videos, ACTalker employs State-Space Modeling (SSM). The model allows driving signals to manipulate feature markers across both temporal and spatial dimensions in each branch, achieving natural facial motion coordination.
- Video diffusion model basicsACTalker is built on a video diffusion model and introduces a multi-branch control module during the denoising process. Each Mamba branch processes a specific modality signal, and the influence weights of each modality are dynamically adjusted through a gating mechanism.
ACTalker's project address
- Project official website:https://harlanhong.github.io/publications/actalker
- Github repository:https://github.com/harlanhong/ACTalker
- HuggingFace model library:https://huggingface.co/papers/2504.02542
- arXiv technical paper:https://arxiv.org/pdf/2504.02542
ACTalker application scenarios
- Virtual streamerACTalker can generate natural and smooth speaking head videos through various signals such as audio and facial expressions, making virtual anchors more vivid and realistic, better interacting with the audience, and enhancing the audience's viewing experience.
- Remote conferencingIn remote meetings, ACTalker can generate natural speaking head videos using audio signals and participants' facial expressions. This solves the problem of lip-syncing caused by network latency, allowing participants to generate natural facial videos using audio and facial expressions even when the video signal is poor, enhancing the realism of remote communication.
- Online EducationIn online education scenarios, teachers can use ACTalker to generate natural speaking head videos. By controlling audio and facial expressions, they can make teaching videos more vivid and interesting, attract students' attention, and improve teaching effectiveness.
- Virtual Reality and Augmented RealityIn virtual reality (VR) and augmented reality (AR) applications, ACTalker can generate talking head videos that match the virtual environment or augmented reality scene.
- Entertainment and GamesIn the entertainment and gaming industries, ACTalker can generate natural speaking head videos for characters, enhancing their expressiveness and immersion.