The Language of Motion - A unified multimodal language model developed by Fei-Fei Li's team at Stanford University.

What is the Language of Motion?

The Language of Motion, developed by Fei-Fei Li's team at Stanford University, is a multimodal language model that integrates verbal and nonverbal language from human actions. The model processes text, speech, and motion data to generate corresponding target modalities, crucial for creating virtual characters that communicate naturally. The Language of Motion demonstrates superior performance in shared speech and gesture generation tasks, requiring significantly less data for training compared to traditional models. The model can also perform novel tasks such as emotion prediction, recognizing emotions from actions. The Language of Motion has significant implications for applications in games, film, and virtual reality, advancing the development of technologies that enable natural communication between virtual characters and humans.

The main functions of the Language of Motion

• Multimodal input processingIt can accept text, voice and motion data as input and flexibly process data of multiple modalities.
• Action understanding and generationBased on input speech, text, or motion data, it understands and generates corresponding 3D human motion.
• Common speech and gesture generationGenerate gestures synchronized with voice to enhance the natural communication ability of virtual characters.
• Sentiment PredictionPredicting emotions from motion data can provide support for fields such as mental health and psychiatry.
• Edit gesture generationIt supports users in editing the movements of specific body parts based on voice or text prompts, enhancing the expressiveness of the movements.

The technical principles of the Language of Motion

• Modal taggingThe motions of the face, hands, upper body, and lower body are tokenized using a vector quantization variational autoencoder (VQ-VAE), converting continuous motion data into discrete tokens.
• Multimodal vocabularyThe goal is to combine tags from different modalities into a unified multimodal vocabulary, allowing language models to process inputs from different modalities.
• Encoder-decoder architectureA language model with an encoder-decoder structure takes mixed tokens as input and generates a sequence of output tokens.
• Generate pre-trainedBased on self-supervised learning, it aligns the relationships between different modalities, such as the correspondence between the movements of different parts of the body and audio-text alignment.
• Instructions follow trainingAfter pre-training, the model is fine-tuned using instruction templates, enabling it to execute specific downstream tasks based on natural language instructions.
• End-to-end trainingThe model is trained end-to-end in both pre-training and post-training phases to maximize the alignment between modalities.

The Language of Motion project address

• Project official website:languageofmotion.github.io
• arXiv technical paper:https://arxiv.org/pdf/2412.10523

Application scenarios of the Language of Motion

• Game developmentTo create more realistic and natural non-player characters (NPCs) in the game, enabling them to interact with players in a richer way through body language and gestures.
• Film and animation productionGenerate more natural and fluid 3D character movements in movies or animations, reducing the workload of manual animation production and improving production efficiency.
• Virtual Reality (VR)In a virtual reality environment, it provides a more realistic interactive experience, making the actions and reactions of virtual characters more similar to those of real humans.
• Augmented Reality (AR)In AR applications, the actions of virtual objects or characters are coordinated with the gestures and movements of users in the real world.
• social robotsTo provide social robots with more natural ways of communication, enhance robot-human interaction, and make them more effective in areas such as service, education, or companionship.