MoMask - A text-driven model for generating high-quality 3D human motion.

What is MoMask?

MoMask is an innovative 3D human motion generation tool that uses generative masking modeling to generate high-quality 3D human motion based on text descriptions. MoMask employs a hierarchical quantization scheme, representing human motion as multi-layered discrete motion markers, and combines masking Transformers and residual Transformers to generate motion sequences. The model performs exceptionally well on text-to-action generation tasks, achieving an FID of 0.045 (HumanML3D dataset), significantly outperforming other methods. MoMask can be seamlessly applied to related tasks, such as text-guided temporal restoration, without requiring additional fine-tuning.

MoMask's main functions

• Text-driven 3D motion generationUsers can generate corresponding 3D human motion animations by inputting simple text descriptions. For example, inputting "a person is running on a treadmill" will generate the corresponding animation.
• Motion editing and timing controlMoMask supports complex timing editing of generated actions, such as inserting, deleting, or replacing action segments, and can change the duration or details of the actions.
• High-precision motion generationBy employing multi-layer quantization and masking modeling techniques, MoMask can generate high-quality, high-precision, and coherent 3D motion sequences. On the HumanML3D dataset, the generated quality achieves an FID value of only 0.045, significantly outperforming other methods.
• Multi-platform support and ease of useMoMask supports local operation and provides Huggingface WebUI demos and Colab online demos. It can be used as a Blender plugin, making it easy for users to get started quickly.
• Motion evaluation and optimizationMoMask provides evaluation scripts that can be used to assess the quality and realism of generated motion, helping users optimize motion generation effects.

MoMask's technical principles

• Layered Quantification SchemeMoMask employs a hierarchical quantization scheme, representing 3D human motion as multiple layers of discrete motion tokens. First, vector quantization encodes the motion sequence into base-layer motion tokens. Then, iterative residual quantization progressively reduces quantization errors, generating higher-level residual tokens. This approach captures high-fidelity details of the motion.
• Mask TransformerIt is one of the core components of MoMask. During the training phase, the motion markers of the base layer are randomly masked, and these masked markers are predicted based on the text input. During the generation phase, starting from a completely empty sequence, the Mask Transformer iteratively fills in the missing markers to gradually generate a complete action sequence.
• Residual TransformerUsed to process residual labels in hierarchical quantization. After generating the base layer labels, the Residual Transformer predicts the residual labels for the next layer progressively based on the label sequence of the current layer. This allows for further optimization of action details and improves the quality of the generated actions.
• Generation processThe generation process of MoMask consists of two stages:
  • Mask Transformer generates base layer markersStarting from an empty sequence, the masked tags are predicted iteratively until a complete basal layer tag sequence is generated.
  • Residual Transformer generates residual tagsBased on the labels in the basal layer, the residual labels in higher levels are predicted layer by layer, ultimately generating high-quality 3D motion.

MoMask project address

• Project official website:https://ericguo5513.github.io/momask/
• Github repository:https://github.com/EricGuo5513/momask
• arXiv technical paper:https://arxiv.org/pdf/2312.00063

MoMask application scenarios

• Game developmentIn game development, MoMask can quickly generate various character animations, reducing the workload of manually creating animations and improving development efficiency.
• Animation ProductionAnimators can use MoMask to quickly generate complex motion sequences, enabling efficient dynamic scene editing and reducing manual labor.
• Virtual Reality (VR)In VR applications, MoMask can generate realistic virtual character movements based on natural language commands, enhancing the user's sense of immersion.
• Sports data analysisMoMask can be used to analyze athletes' movement trajectories, helping researchers better understand the patterns of human movement and providing training suggestions for athletes.
• Motion editing and repairMoMask supports text-based editing, allowing users to specify and edit specific parts of an action, such as changing its duration or details.