ARTalk - A 3D head animation generation framework developed by the University of Tokyo and other institutions.

What is ARTalk?

ARTalk is a novel voice-driven 3D head animation generation framework developed by the University of Tokyo and RIKEN. Based on an autoregressive model, it achieves real-time, highly synchronized lip movements and natural facial expressions and head poses. ARTalk uses a multi-scale motion codebook and sliding time window technology, combined with voice input, to generate high-quality animation sequences. ARTalk introduces a style encoder to adapt to unseen speaking styles, generating 3D animations with unique personalities. ARTalk outperforms existing technologies in lip synchronization accuracy, facial expression naturalness, and style consistency, offering real-time performance suitable for virtual reality, game animation, and human-computer interaction.

ARTalk's main functions

• Generate natural 3D facial animations in real time.Generates highly synchronized lip movements, facial expressions, and head poses from any audio clip, suitable for fields such as virtual reality, game animation, film production, and human-computer interaction.
• Personalized style adaptationBased on the sample motion sequence, style features are extracted to generate 3D animations with unique personal styles, which can perform well even for identities or styles not seen in training.
• Multiscale motion generationCapture motion details from coarse to fine, ensuring that the generated animation remains natural and coherent across different time scales.
• Low latency and high efficiencyBased on autoregressive models and sliding time window techniques, it enables rapid real-time animation generation, avoiding the high computational cost of diffusion models, and is suitable for real-time applications.

ARTalk's technical principles

• Multiscale VQ autoencoderBased on encoding motion sequences into multi-scale discrete codebooks, motion features at different time scales are captured, improving the compactness of motion representation. Causal masks are used to ensure the coherence of time series.
• Autoregressive generatorBased on the Transformer architecture, it combines the speech features of the current time window and the motion information of the previous window to generate multi-scale motion codebooks step by step, ensuring that the generated actions are closely aligned with the speech and maintain consistency in time.
• Style EncoderExtract style features from sample motion sequences to reduce the dimensionality of the complex mapping between speech and action, enabling the model to generate animations with personalized styles.
• Sliding Time WindowThe speech is segmented into time windows for processing to ensure real-time performance, and a cross-window autoregressive mechanism is used to avoid time discontinuities.
• FLAME modelAs the foundation of 3D facial representation, it transforms complex mesh motion into a low-dimensional parametric representation, simplifying the complexity of motion modeling while preserving facial and movement details.

ARTalk's project address

• Project official website:https://xg-chu.site/project_artalk/
• arXiv technical paper:https://arxiv.org/pdf/2502.20323

ARTalk Application Scenarios

• Virtual Reality (VR) and Augmented Reality (AR)Generate real-time facial animations for virtual characters to enhance immersion.
• Game developmentQuickly generate natural facial expressions and lip movements for NPCs or player characters to enhance the gaming experience.
• Animation ProductionGenerate high-quality 3D animations based on voice, improving production efficiency and reducing labor costs.
• Human-computer interactionGenerate realistic expressions and lip movements for smart assistants, making them more human-like.
• Online EducationIt assists in language learning by demonstrating standard pronunciation movements through animation, thereby improving learning outcomes.