AB
AB
AiBoss

All Categories

project

Aether - Shanghai AI Lab's open-source generative world model

Aether is an open-source generative world model from the Shanghai AI Lab, trained entirely on synthetic data. Aether is the first to deeply integrate 3D spatiotemporal modeling with generative modeling, possessing capabilities for 4D dynamic reconstruction, motion-conditional video prediction, and...

What is Aether?

Aether is an open-source generative world model from the Shanghai AI Lab, trained entirely on synthetic data. Aether is the first to deeply integrate 3D spatiotemporal modeling with generative modeling, possessing three core capabilities: 4D dynamic reconstruction, motion-conditional video prediction, and goal-oriented visual planning. Aether can perceive its environment, understand object positions and motion relationships, and make intelligent decisions. Aether demonstrates powerful zero-shot generalization capabilities in the real world, efficiently completing complex tasks using virtual data training, and providing robust spatial reasoning and decision support for embodied intelligent systems.

Aether's main functions

  • 4D dynamic reconstructionReconstruct a 3D scene model containing time and space from video to capture dynamic changes.
  • Motion Condition Video PredictionPredict future scene changes based on initial observations and movement trajectories.
  • Goal-oriented visual planningGenerate a reasonable path based on the initial and target scenarios to assist the intelligent system in planning the action route.

Aether's technical principles

  • Unified Multitasking FrameworkThis approach integrates dynamic reconstruction, video prediction, and action planning into a unified framework for optimization. Based on task-interleaved feature learning, it achieves collaborative optimization across different tasks, improving the model's stability and robustness.
  • Geometric perceptual modelingThe project introduces 3D spatiotemporal modeling to construct a geometric space and enhance the model's spatial reasoning capabilities. Using massive amounts of simulated RGBD data (color images and depth maps), a complete data cleaning and dynamic reconstruction workflow is developed, annotating rich action sequences.
  • Camera trajectory as motion representation: Camera trajectory is chosen as the representation of global motion. In navigation tasks, the camera trajectory directly corresponds to the navigation path; in robot operation, the motion of the handle camera can capture the 6D motion of the end effector.
  • Diffusion Model and Multimodal FusionBased on a pre-trained video diffusion model, Aether is post-trained using synthetic 4D data. It converts depth video into a scale-invariant normalized disparity representation and encodes camera trajectories as a scale-invariant ray map sequence aligned with the spatiotemporal framework of DiDiffusion Transformers (DiTs). By dynamically integrating cross-task and cross-modal conditional signals, Aether achieves the fusion and collaborative optimization of multimodal information.
  • Zero-shot generalization abilityTrained entirely on virtual data, it achieves zero-shot generalization to the real world. Based on combinations of different conditional inputs (such as observation frames, target frames, and motion trajectories) and a diffusion process, it enables unified modeling and generation for various tasks. This allows the model to perform exceptionally well in real-world scenarios even without real-world data.

Aether's project address

Aether's application scenarios

  • Robot NavigationIt helps robots plan their paths and avoid dynamic obstacles.
  • autonomous drivingReal-time reconstruction of road scenes to predict traffic dynamics.
  • Virtual RealityGenerate immersive virtual scenes to enhance the user experience.
  • Industrial robotsOptimize robot operation paths to improve production efficiency.
  • Intelligent monitoringAnalyze surveillance video to predict abnormal behavior.
Back