EMMA-X - Embossed Multimodal Motion Model from Singapore University of Technology and Design
EMMA-X is an embodied multimodal action model with 7 billion parameters developed by the Singapore University of Technology and Design. It was created by fine-tuning OpenVLA on evidenced chain-of-thought (CoT) inference data. EMMA-X incorporates hierarchical embodied datasets...
What is EMMA-X?
EMMA-X, developed by the Singapore University of Technology and Design, is an embodied multimodal motion model with 7 billion parameters, created by fine-tuning OpenVLA on equivocal chained reasoning (CoT) data. EMMA-X combines a hierarchical embodied dataset, including 3D spatial motion, 2D gripper positions, and equivocal reasoning, and introduces a novel trajectory segmentation strategy. Using the gripper's opening and closing states and the robot arm's motion trajectory, it enhances equivocal task reasoning and forward-looking spatial reasoning, achieving significant performance improvements in real-world robotic tasks, especially those requiring spatial reasoning.
Main functions of EMMA-X
- Enhanced spatial reasoningBased on the predicted future 2D position and 3D motion plan of the gripper, the robot's long-term task planning capability is optimized.
- Specific task planningThe model combines vision and task reasoning to generate action strategies that adapt to the environment, improving the robot's ability to perform complex tasks.
- Trajectory segmentationBy using the gripper state and the robotic arm's motion trajectory, the operation sequence is segmented into semantically similar action segments, enhancing task understanding and motion planning.
- Reduce hallucination problemsCombining visual images with task reasoning reduces errors and illusions during the task reasoning process.
- Hierarchical planning data generationGenerate 2D gripper position and 3D spatial motion for each operation segment, as well as specific reasoning to support the robot's decision-making process.
EMMA-X Technical Principles
- Hierarchical Embodied DatasetsBuilt on the BridgeV2 dataset, it contains 60,000 robot operation trajectories, each with detailed spatial reasoning and task reasoning information.
- Look-ahead Spatial ReasoningThe model predicts the future position and motion plan of the gripper, guiding the robot's immediate actions to align with long-term goals.
- Trajectory segmentation strategyThe operation sequence is dynamically segmented using the HDBSCAN algorithm and a custom distance metric method, combined with the motion trajectory of the end effector and the gripper state.
- Gemini Generates Task ReasoningThe Gemini model is used to generate sub-tasks and specific inferences for each segment, improving the accuracy of task understanding.
- EMMA-X architectureBased on OpenVLA adjustments, chain-like thinking training enhances spatial reasoning and scene understanding capabilities, predicting the robot's next action strategy.
EMMA-X project address
- GitHub repository:https://github.com/declare-lab/Emma-X
- HuggingFace model library:https://huggingface.co/declare-lab/Emma-X
- arXiv technical paper:https://arxiv.org/pdf/2412.11974
Application scenarios of EMMA-X
- Manufacturing AutomationRobots are used in tasks such as assembly, packaging, and quality control to improve the efficiency and flexibility of production lines.
- Logistics and warehousingIn warehouses, robots help pick, move, and sort goods, optimizing storage space and logistics processes.
- service industryIn catering or hotel services, robots perform complex tasks such as cooking assistance, room cleaning, and item delivery.
- Medical assistanceIn the medical field, robots perform delicate operations, such as assisting in surgery or processing samples in the laboratory.
- Home AutomationHome service robots perform cleaning, tidying, and other household chores, improving the convenience of life.