Cosmos 3 Edge - NVIDIA's open-source 4B parameter world model

What is Cosmos 3 Edge?

Cosmos 3 Edge is a 4B-parameter open-source world model from NVIDIA, designed for real-time inference in robotics and edge AI. Based on the Nemotron architecture, it integrates autoregressive and diffusion dual Transformer towers, shares a multimodal attention layer, and can run locally on devices such as Jetson Thor, RTX GPUs, and DGX, achieving real-time control at 15 Hz. Developers can adapt the model to a specific robot or environment in approximately one day, enabling perception, inference, and action policy generation without relying on the cloud.

Key features of Cosmos 3 Edge

• World State UnderstandingBy processing visual and text tokens through autoregressive towers, it can parse objects, spatial relationships, and scene semantics in the edge environment in real time, providing a shared world representation for subsequent reasoning.
• Future state predictionThe algorithm uses a diffusion tower to denoise and generate visual, audio, and action tokens, simulating the visual result after performing a certain action.
• Action strategy generationThe system encodes different physical forms such as robotic arms, vehicles, and cameras into compact geometric vectors representing translation, rotation, and operation states. Each inference generates 32 consecutive actions, and outputs control commands at a frequency of 15 Hz.
• Real-time inference on the edgeAchieve high-throughput, memory-efficient inference on memory-constrained devices such as Jetson Thor, RTX PRO, and Jetson T2000/T3000, with closed-loop operation without cloud access.
• Rapid Domain AdaptationIt provides complete post-training scripts and DROID dataset policy checkpoints, allowing developers to fine-tune the base model to a specific robot or environment in about a day.

Technical Principles of Cosmos 3 Edge

• Dual-Tower Shared ArchitectureThe model consists of an autoregressive tower and a diffusion tower. The autoregressive tower uses causal attention to process visual and text tokens for scene understanding and reasoning; the diffusion tower uses bidirectional attention to process visual, audio, and action tokens for prediction, generation, and neural simulation. Each tower has its own independent LayerNorm and MLP, but shares a multimodal attention layer to align language, video, audio, and action information into a unified representation space.
• General action representationIt maps the actions of different physical systems into compact geometric vectors, uniformly encodes translation, rotation and operation states, and establishes a direct connection between pixel changes and physical motion, spatial relationships and control inputs.
• World Action Model (WAM)When used as a strategy, the model receives the current observation state and outputs the action to be performed and its expected visual result, directly linking world modeling with robot policy training. Actions can flow bidirectionally within the model: the effect of an action can be predicted, and the action can be driven by the effect.
• Edge optimization inferenceBased on the Nemotron architecture, the 4B parameter compact design combines a 4-step knowledge distillation and vLLM optimization framework to achieve up to 25x inference speedup while maintaining output quality, and is adapted to the memory and computing power constraints of edge computing platforms such as the Jetson series.

Follow us on WeChat and reply with "open source",join inAI open source project discussion group

How to use Cosmos 3 Edge

• Environmental preparationInstall the NVIDIA Container Toolkit and pull the official Docker image, or configure the Conda environment to install dependencies such as PyTorch, Diffusers, and Transformers.
• Download ModelDownload from Hugging Face nvidia/Cosmos3-Edge Weights, and also obtain post-training script and DROID dataset policy checkpoints.
• Post-training fine-tuningUsing a small H100 cluster or DGX Station, fine-tuning of a specific robot, sensor, or environment based on the open-source Cosmos Framework takes about a day.
• Deployment reasoningThe fine-tuned model is deployed to edge devices such as Jetson T2000/T3000, RTX PRO, or DGX, and 15 Hz real-time motion generation is achieved through vLLM or NIM microservices.

The core advantages of Cosmos 3 Edge

• Real-time inference on the edgeWith a lightweight design featuring 4 billion parameters, it achieves 15 Hz real-time control on Jetson Thor, eliminating cloud latency.
• Quick adaptationDevelopers can fine-tune the base model to a specific robot or environment in about a day, significantly reducing the deployment cycle.
• Unified Multimodal ArchitectureThe autoregressive tower and the diffusion tower share the same attention layer, uniformly processing language, video, audio, and actions to achieve a closed loop of "understanding → simulation → action".
• Open EcosystemThe model weights, training scripts, post-training schemes, and 4-step knowledge distillation checkpoints are all open source, supporting Hugging Face Diffusers and vLLM deployment.
• Benchmark LeadingAmong models of similar size (4B), VANTAGE-Bench ranks first in visual analysis and its robot policy learning reaches an industry-leading level.