SceneGen - A 3D scene generation framework developed by Shanghai Jiao Tong University

What is SceneGen?

SceneGen is a high-efficiency, open-source 3D scene generation framework developed by a research team at Shanghai Jiao Tong University. Starting from a single scene image and its corresponding object segmentation mask, it directly generates a 3D scene containing complete geometry, texture, and spatial layout through a single forward propagation. Its innovation lies in its end-to-end generation process, eliminating the need for time-consuming optimization processes or retrieval and assembly from an asset library, significantly improving generation efficiency. The core technology includes a module for aggregating local and global scene information, and a position prediction head that simultaneously predicts 3D assets and their relative spatial positions, ensuring the physical plausibility and visual consistency of the generated scene. The tool primarily targets VR/AR, Embodied AI, game development, and interior design, providing a powerful solution for rapidly building realistic virtual environments.

SceneGen's main functions

• Single Image to 3D Scene GenerationIt can directly generate a 3D scene containing complete geometry, texture and spatial layout from a single scene image and its corresponding object segmentation mask.
• High-efficiency end-to-end generationScene generation can be completed in a single forward propagation, without relying on time-consuming optimization processes or retrieval and assembly from an asset library, significantly improving generation efficiency.
• Local and global information aggregationAn aggregation module was introduced during the feature extraction process to effectively combine local details with global scene information, ensuring the rationality and consistency of the generated scene.
• Joint forecasting assets and locationsThrough a unique position head, it can simultaneously predict the 3D assets (objects) in the scene and their precise relative spatial positions.
• High precision and realismOn both synthetic and real datasets, the scenes it generates demonstrate superior performance compared to previous methods in terms of geometric accuracy, texture detail, and visual quality.

SceneGen's technical principles

• Input processing and feature extractionThe algorithm takes a single scene image and its corresponding object segmentation mask as input, and extracts asset-level (object) features and scene-level global features through a visual encoder and a geometric encoder, respectively.
• Local texture refinementThe pre-trained local attention module is used to optimize and enhance the texture details of each individual object, ensuring the visual quality of the generated assets.
• Global Feature FusionBy integrating local object features with scene-level layout information through the designed global attention module (aggregation module), spatial relationships and contextual dependencies between objects are captured, ensuring scene rationality and consistency.
• Joint Decoding and GenerationIt uses an off-the-shelf structural decoder to process the fused features, while predicting the precise relative spatial position of objects through a unique position prediction head, thus achieving one-time generation of geometry, texture and position.
• End-to-end optimizationThe entire process is completed through a single forward propagation, eliminating the need for iterative optimization or external asset retrieval, which significantly improves generation efficiency and achieves high accuracy and realism on both synthetic and real datasets.

SceneGen's project address

• Project official websitehttps://mengmouxu.github.io/SceneGen/
• Github repositoryhttps://github.com/mengmouxu/scenegen
• HuggingFace model libraryhttps://huggingface.co/haoningwu/scenegen
• arXiv technical paperhttps://arxiv.org/pdf/2508.15769

Application scenarios of SceneGen

• Game and film productionIt can quickly generate ready-to-use 3D scene resources based on concept art or reference photos, significantly reducing the time cost of scene modeling, which is especially beneficial to teams with limited resources, such as independent game developers.
• Virtual Reality (VR) and Augmented Reality (AR): To efficiently generate realistic and interactive 3D environments for VR/AR applications, meeting the urgent need for high-quality virtual scenes in fields such as Embodied AI.
• Real Estate and Interior DesignIt can quickly convert floor plans or real-life photos into interactive 3D displays, helping real estate agents, developers, or clients to more intuitively understand the spatial layout and design effects of a house.
• Simulation and training environment constructionIt provides efficient scene generation solutions for tasks such as autonomous driving and robot navigation that require extensive testing and training in high-fidelity virtual environments.