Scenethesis - An interactive 3D scene generation framework from NVIDIA
Scenethesis is an innovative framework from NVIDIA used to generate interactive 3D scenes from text. The framework combines Large Language Models (LLM) and visual perception techniques, achieving efficient generation through a multi-stage process, using LLM...
What is Scenethesis?
Scenethesis is an innovative framework from NVIDIA for generating interactive 3D scenes from text. The framework combines Large Language Models (LLM) and visual perception technology, achieving efficient generation through a multi-stage process. LLM is used for coarse layout planning, a visual module refines the layout to generate image guidance, an optimization module adjusts object poses to ensure physical plausibility, and a judgment module verifies the spatial coherence of the scene. Scenethesis can generate diverse indoor and outdoor scenes with high realism and physical plausibility, and is widely used in virtual content creation, simulation environments, and embodied intelligence research.
The main functions of Scenethesis
- Text-generated 3D sceneThe user inputs a text description, and AI automatically generates the corresponding 3D scene.
- Physical rationalityThe generated scene objects do not penetrate each other, can be placed stably, and conform to the rules of physics.
- User InteractionUsers can adjust the position and size of objects, and the scene is updated in real time.
- Diverse scenes and object librariesSupports a variety of indoor and outdoor scenes, with a rich selection of 3D models to choose from.
- Scenario verificationAutomatically checks scene quality and regenerates and optimizes as needed to ensure high-quality output.
Scenethesis's technical principles
- Rough layout planning (LLM module)After the user inputs a text description, the LLM parses the text to understand the scene's theme and key elements. The LLM selects objects relevant to the scene description from a predefined library of 3D models, generating a rough layout plan. The LLM then generates detailed hints describing the initial positions and relationships of objects in the scene, providing a foundation for subsequent visual refinement.
- Visual refinement (visual module)Based on prompts generated by the LLM, the vision module (such as a diffusion-based image generator) generates detailed images as visual guidance for scene layout. The generated images are segmented and depth estimated using pre-trained visual base models (such as Grounded-SAM and DepthPro), extracting a scene map including 3D bounding boxes of objects and spatial relationships. Based on the scene map and prompts, 3D object models matching the scene description are retrieved from a 3D model library.
- Physics optimization (Physics optimization module)Semantic correspondence matching techniques (such as RoMa) are used to align the poses of objects in the 3D model and the image guidance, ensuring that the position, size, and orientation of the objects are consistent with the image guidance. Signed distance field (SDF) technology is used to detect collisions between objects and adjust the position and size of the objects to avoid collisions.
- Scene verification (Scene verification module)The system evaluates the spatial coherence of the generated scene based on a pre-trained language model, ensuring that the positions and relationships of objects conform to common sense. If the evaluation results do not meet the requirements, the system will trigger a replanning and optimization process until the generated scene reaches a high-quality standard.
Scenethesis's project address
- Project official website:https://research.nvidia.com/labs/dir/scenethesis/
- arXiv technical paper:https://arxiv.org/pdf/2505.02836
Application scenarios of Scenethesis
- Virtual Reality (VR) / Augmented Reality (AR)Create immersive virtual environments, such as virtual tours or exhibition halls.
- Game developmentQuickly generate game levels and virtual worlds to improve development efficiency.
- Embodied IntelligenceProvide realistic virtual scenarios for AI training, such as home environments.
- Virtual content creationGenerate scene prototypes for movies, animations, or advertisements to accelerate the creation process.
- Education and trainingSimulated laboratory or training scenarios are used for teaching and skills practice.