TesserAct - an AI 4D embodied world model that can predict the dynamic evolution of 3D scenes.

What is TesserAct?

TesserAct is an innovative 4D embodied world model that predicts the dynamic evolution of 3D scenes over time and responds to the actions of embodied agents. Trained on RGB-DN (RGB, depth, and normal) video data, it surpasses traditional 2D models by incorporating detailed shape, configuration, and temporal variations into its predictions. TesserAct's core strength lies in its spatiotemporal consistency, supporting novel perspective synthesis and significantly improving policy learning performance.

Main functions of TesserAct

• 4D scene generationTesserAct can generate video streams containing RGB (color images), depth maps, and normal maps, which together form a coherent 4D scene, helping AI systems understand the shape, position, and motion of objects.
• New Perspective SynthesisThe model supports generating scene images from different perspectives, which is very helpful for robot navigation and operation in complex environments.
• Spatiotemporal consistency optimizationBy introducing spatiotemporal continuity constraints, TesserAct ensures that the generated 4D scenes remain highly consistent in time and space, more closely resembling the physical laws of the real world.
• Robot operation supportThe TesserAct-based robot performs well in a variety of maneuvering tasks, especially in tasks requiring precise spatial understanding, with a success rate far exceeding that of methods that rely solely on 2D images.
• Cross-platform generalization capabilityTesserAct performs stably across different platforms and environments, and can adapt to a variety of complex scenarios.

The technical principles of TesserAct

• Dataset ExpansionTesserAct first expands the existing robot operation video dataset by adding depth and normal information to enrich the data content. It uses existing models to acquire depth and normal data, providing richer multimodal information for training.
• Fine-tuning the video generation modelOn the expanded dataset, TesserAct fine-tuned a video generation model that can jointly predict the RGB, depth, and normal information for each frame. This multimodal prediction capability allows the model to more comprehensively understand the shape, configuration, and temporal changes of the scene.
• Scene transformation algorithmTesserAct proposes an algorithm that can directly convert generated RGB, depth, and normal videos into high-quality 4D scenes. It ensures the temporal and spatial coherence of the 4D scenes predicted from embodied scenes and supports novel perspective synthesis and policy learning.
• Spatiotemporal consistency optimizationTesserAct introduces spatiotemporal continuity constraints to ensure that the generated 4D scenes remain highly consistent in time and space. This enables the model to more realistically reflect the dynamic changes of the physical world, providing embodied agents with a more accurate understanding of their environment.
• Inverse dynamics model learningTesserAct can generate high-quality 4D scenes and learn inverse dynamics models of embodied agents. This enables agents to more accurately predict the impact of their actions on the environment and performs better in complex tasks.

TesserAct's project address

• Project official website:https://tesseractworld.github.io/
• Github repository:https://github.com/UMass-Embodied-AGI/TesserAct
• HuggingFace model library:https://huggingface.co/anyeZHY/tesseract
• arXiv technical paper:https://arxiv.org/pdf/2504.20995

Application scenarios of TesserAct

• Robot operation tasksTesserAct helps robots better understand and predict dynamic changes in their environment by generating high-quality 4D scenes. For example, in tasks such as object grasping, sorting, and placement, TesserAct provides accurate spatial information, significantly improving the success rate of robot operations.
• Virtual environment interactionTesserAct supports new perspective synthesis and spatiotemporal consistency in 4D scene generation. For example, in virtual reality (VR) or augmented reality (AR) scenarios, TesserAct can provide users with a more realistic visual experience.
• Embodied Intelligence ResearchTesserAct provides a powerful tool for embodied intelligence research, helping researchers better understand how agents interact with their environment through perception and action.
• Industrial AutomationIn industrial automation scenarios, TesserAct can help robots perform tasks more effectively, such as object recognition and manipulation in dynamic environments. Its spatiotemporal continuity optimization capabilities enable it to adapt to complex working environments.