Cosmos-Reason1 - A series of multimodal large language models launched by NVIDIA

What is Cosmos-Reason1?

Cosmos-Reason1 is a series of multimodal large language models from NVIDIA, capable of generating responses based on physical reality. Cosmos-Reason1 includes two models: Cosmos-Reason1-7B and Cosmos-Reason1-56B. The models are trained through four stages: visual pre-training, general SFT, physical AI SFT, and reinforcement learning. Combined with video input and text prompts, they output responses with long inference chains, demonstrating outstanding performance in physical commonsense and embodied reasoning benchmarks, significantly outperforming other similar models. The models define the ontology of physical commonsense and embodied reasoning, constructing corresponding benchmarks to evaluate the physical AI reasoning capabilities of multimodal LLMs.

Main functions of Cosmos-Reason1

• Understanding basic physicsTo understand the basic knowledge of the physical world, such as space, time and fundamental physical laws, and to judge the rationality of events.
• Embodied reasoningBased on common sense about physics, it generates reasonable decision-making and action plans for embodied agents (such as robots and autonomous vehicles).
• Long Chain ThinkingIt generates detailed reasoning processes based on chain-of-thought reasoning, improving the transparency and explainability of decision-making.
• Multimodal input processingIt supports video input, combines visual information and language commands to perform reasoning, and generates natural language responses.

The technical principles of Cosmos-Reason1

• Hierarchical Ontology: Defines a hierarchical ontology of physical common sense, covering three main categories: space, time, and fundamental physics, which are further subdivided into 16 subcategories.
• Two-dimensional ontology: Design a two-dimensional ontology for embodied reasoning, encompassing four key reasoning capabilities of five embodied agents.
• Multimodal architectureBased on a decoder-only multimodal architecture, the input video is processed by a visual encoder, aligned with text tag embeddings, and then input into the LLM.
• The model has four training phases:
  • Visual pre-trainingAlign visual and text modalities.
  • Supervised Fine-tuning (SFT)Improve the model's performance in general visual language tasks.
  • Physics AI SFTEnhance physical common sense and embodied reasoning abilities with specialized data.
  • Physics AI Reinforcement Learning (RL)): Further optimize the model's reasoning ability based on rule-based rewards.
• reinforcement learningDesign a rule-based reward mechanism based on multiple-choice questions, and improve the model's performance in physics common sense and embodied reasoning tasks based on reinforcement learning.

Cosmos-Reason1 project address

• Project official website:https://research.nvidia.com/labs/dir/cosmos-reason1/
• GitHub repository:https://github.com/nvidia-cosmos/cosmos-reason1
• HuggingFace model library:https://huggingface.co/collections/nvidia/cosmos-reason1
• arXiv technical paper:https://arxiv.org/pdf/2503.15558

Application scenarios of Cosmos-Reason1

• robot operationIt helps robots understand task objectives, generate operation plans, and complete complex actions such as grasping and assembling.
• autonomous drivingIt processes road video, predicts traffic dynamics, and generates safe driving decisions, such as avoidance and lane changing.
• Intelligent monitoringIt can monitor abnormal behaviors in video in real time, such as people falling or equipment malfunctions, and issue alarms in a timely manner.
• Virtual Reality (VR) / Augmented Reality (AR)Based on input from the virtual environment, it generates interactive responses to enhance user immersion.
• Education and Training: Based on video explanations of physical phenomena or operating procedures, to assist in teaching and vocational skills training.