EMAGE - A framework for generating full-body communicative gestures based on audio, jointly developed by Tsinghua University, Southeast University, and other institutions.

What is EMAGE?

EMAGE (Expressive Masked Audio-conditioned Gesture Modeling) is a framework for generating full-body syncopated gestures, developed by institutions such as Tsinghua University, the University of Tokyo, and Keio University. EMAGE can generate complete full-body gestures synchronized with audio, including facial expressions, body movements, gestures, and global motion, based on audio and partially masked gesture input. EMAGE achieves joint training of audio-to-gesture generation and masked gesture reconstruction using a masked audio-gesture transformer, effectively encoding audio and body gesture cues. EMAGE uses four pre-trained VQ-VAEs (Vector Quantized Variational AutoEncoders) to decode local facial and body movements, and a pre-trained global motion predictor to decode global translation.

EMAGE's main functions

• Generate gestures synchronized with audio.Based on the input audio signal, generate full-body gestures that match the audio rhythm and semantics.
• Accepts some predefined gesture inputsBased on partial gestures (such as specific actions or postures) provided by the user, it generates complete and natural gesture sequences.
• Generate diverse gesturesCombining multiple pre-trained VQ-VAEs (Vector Quantization Variational Autoencoders) generates a rich variety of gestures, avoiding monotonous and repetitive results.
• Supports full-body gesture generationIt supports the generation of facial and upper limb movements, as well as lower limb and global motion, achieving more natural and complete human animation effects.

EMAGE's technical principles

• Masked audio gesture changerDuring training, some gesture data is randomly occluded, and the model predicts the occluded parts based on audio and other unoccluded gesture information. This masking mechanism is similar to the BERT model in natural language processing, enhancing the model's understanding of local and global information. Training for audio-to-gesture generation and masked gesture reconstruction allows the model to better learn the mapping relationship between audio and gestures, while utilizing existing gesture information (even if partially occluded) to generate more accurate gestures.
• Audio feature fusionThis approach combines rhythmic features of audio (such as pitch variations and rhythmic strength) with content features (such as word embeddings from text transcription) to generate gestures that reflect both the rhythm of the audio and are semantically relevant. For example, when the word "walking" is mentioned, the model can generate a corresponding walking gesture.
• Pre-trained VQ-VAEThis approach involves modeling different body parts (such as the face, upper limbs, and lower limbs) using independent VQ-VAEs, with each VQ-VAE responsible for generating the movements of a specific part. This segmented modeling method better captures the unique relationship between different body parts and audio, improving the accuracy and diversity of generated gestures.
• Discrete Representation LearningBased on VQ-VAE, continuous gestures are converted into discrete representations, which facilitates model learning and generation. This discrete representation is similar to words in a language, allowing the model to process and generate complex gesture sequences more efficiently.
• Global motion predictorIn addition to generating local movements, a pre-trained global motion predictor estimates the body's global translation (such as walking, turning, etc.), making the generated gestures natural in local movements and the overall movement more coherent and realistic.

EMAGE's project address

• Project official website:https://pantomatrix.github.io/EMAGE
• GitHub repository:https://github.com/PantoMatrix/EMAGE
• arXiv technical paper:https://arxiv.org/pdf/2401.00374
• Experience the demo online:https://huggingface.co/spaces/H-Liu1997/EMAGE

Application scenarios of EMAGE

• Virtual Reality (VR) and Augmented Reality (AR)Generate natural gestures and expressions for virtual characters, enhancing the interactive experience in scenarios such as virtual social interaction, virtual meetings, and virtual games, making virtual characters more vivid and lifelike.
• Animation ProductionIt assists in the production of animated films, TV series, and dynamic advertisements by automatically generating character gestures and expressions based on dialogue content, reducing the workload of animators and improving production efficiency.
• Virtual Customer Service and Digital Humans: Create natural and fluid gestures and expressions for online customer service, intelligent assistants, etc., making them more approachable and realistic when communicating with users, thus improving the user experience.
• Education and TrainingIn the field of education, virtual teachers generate corresponding gestures based on the content being explained, helping students better understand the knowledge; in training scenarios, they generate realistic operation demonstration gestures to assist teaching.
• Entertainment industryIn game development, generating matching gestures and expressions based on character voices enhances the game's immersion and realism.