MimicTalk - An open-source 3D digital head project launched by ByteDance in collaboration with Zhejiang University.

What is MimicTalk?

MimicTalk, jointly developed by Zhejiang University and ByteDance, is based on NeRF (Neural Radiation Field) technology. It can train personalized and expressive 3D speaking face models in a very short time, requiring only 15 minutes. MimicTalk improves training efficiency, significantly enhancing the quality of generated videos through efficient fine-tuning strategies and a face motion generation model with contextual learning capabilities. MimicTalk's code is an optimized version of the previous Real3D-Portrait project, supporting audio-driven generation of speaking avatars for specific individuals, making digital human videos highly similar to real people in terms of sensory perception.

MimicTalk's main functions

• Rapid Personalized TrainingMimicTalk can help users adapt to and train a new identity within 15 minutes, significantly improving training efficiency compared to traditional methods.
• High-quality video generationBased on fine-tuning and optimization, MimicTalk generates videos that surpass previous technologies in visual quality.
• Enhanced expressivenessMimicTalk can capture and mimic the dynamic speaking style of a target person, making the generated videos more vivid and expressive.
• Contextual learningThe model can learn the target person's speaking style from the context, improving the naturalness and realism of facial movements.
• audio driverIt supports using audio input to drive a 3D speaking avatar of a specific person, achieving synchronization between audio and facial movements.

MimicTalk's technical principles

• Person-Agnostic 3D Face Generation Model:
  • As a base model, it is a pre-trained, general-purpose 3D face generation model capable of processing single image inputs.
  • The model can generate realistic 3D faces for different characters.
• Static-Dynamic Hybrid Adaptation Pipeline:
  • Hybrid adaptation processes help models learn the static appearance (such as facial geometry and texture details) and dynamic features (such as facial expressions and muscle movements) of a specific identity.
  • Based on optimized 3D face representation (tri-plane) and low-rank adaptation (LoRA) technology, the model can quickly adapt to new identities.
• In-Context Stylized Audio-to-Motion Model (ICS-A2M):
  • The model is responsible for generating facial movements that match the target person's speaking style.
  • It learns and imitates the speaking style of reference videos based on context, without the need for explicit style representation.
• Flow Matching Model:
  • Advanced generative models used to generate expressive facial movements predict the velocity field of data points, guiding the data points to move from a simple prior distribution to the target distribution.
  • During training, the objective of Conditional Flow Matching (CFM) is used to optimize the accuracy of model predictions.
• Inference Process:
  • During the inference phase, MimicTalk uses the ICS-A2M model and a personalized renderer to generate high-quality videos of speaking faces.
  • The reasoning process involves combining audio input with reference videos of the target person to generate facial movements that mimic a specific speaking style.
• Data and training efficiencyMimicTalk's design prioritizes sample efficiency and training efficiency, enabling users to adapt to new identities with minimal data in a very short time.

MimicTalk's project address

• Project official website:mimictalk.github.io
• GitHub repository:https://github.com/yerfor/MimicTalk
• arXiv technical paper:https://arxiv.org/pdf/2410.06734

Application Scenarios of MimicTalk

• Virtual anchors and digital humansUse MimicTalk technology to create virtual anchors in news broadcasts, entertainment programs, and live online broadcasts, providing a more natural and engaging viewing experience.
• Video conferencing and remote collaborationIn remote work and online meetings, MimicTalk provides users with personalized virtual avatars, increasing interactivity and immersion.
• Virtual Reality (VR) and Augmented Reality (AR)In VR and AR applications, MimicTalk generates realistic virtual characters, enhancing the gaming experience and the quality of interaction.
• Social media and entertainmentUsers create their own virtual avatars, share them on social media, or interact with others in the virtual world.
• Customer service and chatbotsBased on MimicTalk technology, we will develop more human-like customer service robots to provide a more natural and friendly customer experience.