AB
AB
AiBoss

All Categories

project

Large Motion Model - A unified multimodal motion generation model jointly developed by SenseTime and Nanyang Technological University

Large Motion Model (LMM) is a unified multimodal motion generation model jointly developed by the S-Lab at Nanyang Technological University, Singapore, and the SenseTime research team. LMM can handle various motion generation scenarios, including text-to-motion, music-to-dance, and more...

What is Large Motion Model?

Large Motion Model (LMM) is a unified multimodal motion generation model jointly developed by the S-Lab at Nanyang Technological University, Singapore, and the SenseTime research team. LMM can handle various motion generation tasks, including text-to-motion and music-to-dance, demonstrating performance comparable to expert models on multiple benchmarks. The model is built upon a comprehensive MotionVerse dataset integrating datasets of different modalities, formats, and tasks. It employs an innovative ArtAttention mechanism and pre-training strategies to achieve precise control over body parts and broad knowledge generalization. LMM exhibits strong generalization capabilities when handling unseen tasks, providing a new perspective for future research on large-scale motion models.

The main functions of Large Motion Model

  • Multi-task motion generationIt can perform various motion generation tasks, such as text-to-motion, music-to-dance, and action-to-motion.
  • Dataset integrationCreate the MotionVerse dataset, which integrates datasets of different modalities, formats, and tasks to achieve a unified motion representation.
  • Precise controlIt employs the ArtAttention mechanism to support precise control of different body parts, thereby improving the precision of motion generation.
  • Generalization abilityIt demonstrates strong generalization ability and performs effective motion generation on a variety of unseen tasks.
  • Multimodal input processingIt can simultaneously process multiple modal inputs, such as text, music, and video, and generate corresponding motion outputs.

The technical principle of Large Motion Model

  • Unified Dataset (MotionVerse)Based on the unified dataset MotionVerse, which contains motion data from various tasks and modalities, and using the TOMATO notation to unify motion data in different formats.
  • Diffusion Transformer Backbone NetworkA Diffusion model based on the Transformer architecture is used to generate high-quality motion sequences using a Denoising Diffusion Probability Model (DDPM).
  • ArtAttention mechanismWe designed a novel attention mechanism called ArtAttention, which combines body part perception modeling to support the model in independently controlling and learning different body parts.
  • Pre-training strategyThe pre-training strategy employs random frame rates and multiple masking techniques to enhance the model's learning and generalization capabilities from different data sources.
  • Zero-shot learningThe zero-sample method is used to generate long sequence motions, allowing the model to generate motions without additional samples.

Project address for Large Motion Model

Application scenarios of Large Motion Model

  • Animation and game productionGenerate realistic character animations, reduce the time and cost of manual animation production, and improve the efficiency of animation production.
  • Virtual Reality (VR) and Augmented Reality (AR)In VR and AR applications, virtual character movements are generated to match the user's actions, enhancing immersion.
  • Film and video productionGenerate special effects for movies, such as simulating complex fight scenes or dance moves, to improve production efficiency.
  • Exercise analysis and trainingAnalyze athletes' movements, provide training suggestions, or generate standard movement templates.
  • Robotics: To teach robots to perform complex human actions and improve their application capabilities in service, medical or industrial fields.
Back