Magma - A multimodal AI foundational model developed by Microsoft Research in collaboration with universities in Washington, D.C., and other institutions.

What is Magma?

Magma is a new multimodal AI foundational model from Microsoft Research, providing general-purpose capabilities for multimodal AI agents. Magma can understand and execute tasks with multimodal inputs, covering both digital and physical environments. Pre-trained on massive amounts of visual-language and motion data, Magma combines linguistic, spatial, and temporal intelligence to perform complex tasks ranging from UI navigation to robot manipulation. In experiments, Magma demonstrated superior performance in both zero-shot and fine-tuned settings, outperforming existing dedicated models in robot manipulation and multimodal understanding tasks.

Magma's main functions

• Multimodal understandingIt can process data in multiple modalities, including images, videos, and text, and understand their semantic, spatial, and temporal information. It supports tasks ranging from simple image recognition to complex video understanding.
• Motion planning and executionIt breaks down complex tasks into a series of executable action sequences. It supports everything from UI navigation (such as web page operations and mobile application operations) to robot operations in the physical environment (such as grasping, placing, and moving objects).
• Environmental adaptabilityIt can adapt to a variety of downstream tasks with zero-shot capability, including UI navigation, robot manipulation, and multimodal understanding.

Magma's technical principles

• Pre-trained architectureThis approach uses convolutional networks (such as ConvNeXt) as visual encoders to process image and video data. The encoded visual information, along with language tags, is then fed into a large language model (LLM) to generate action sequences or language descriptions.
• Set-of-Mark (SoM)This involves labeling interactive visual objects in an image (such as buttons in a GUI or target locations on a robotic arm). Predicting the positions of these labels helps the model understand and execute actions (action grounding).
• Trace-of-Mark (ToM)This involves annotating the motion trajectories of objects in videos (such as the movement path of a robotic arm). Based on predicted future trajectories, it helps models understand and plan action sequences, enhancing their ability to understand temporal dynamics.
• Multimodal data fusionPre-training data includes images, videos, robot operation data, and data from multimodal understanding tasks. Based on SoM and ToM technologies, these different types of data are unified into a single pre-training framework, improving the model's versatility and adaptability.
• Zero-sample and fine-tuning capabilitiesThe pre-trained model can be directly applied to unseen tasks (zero samples), demonstrating strong generalization ability. Fine-tuning on a small amount of data can further improve performance and adapt to the needs of specific tasks.

Magma's project address

• Project official website:https://microsoft.github.io/Magma/
• GitHub repository:https://github.com/microsoft/Magma
• arXiv technical paper:https://www.arxiv.org/pdf/2502.13130

Application scenarios of Magma

• Web and mobile application operationsIt can automatically complete tasks such as searching, installing applications, and filling out forms.
• robot operationControlling robots to perform tasks such as grasping, placing, and moving objects.
• Video UnderstandingAnalyze the video content and answer related questions.
• Smart AssistantAs a virtual assistant, it understands instructions and completes interactive tasks.
• Education and Training: To assist in teaching, providing operational guidance and feedback.