HouseCrafter - Northeastern University and Stability AI launch technology to transform 2D indoor scenes into 3D
HouseCrafter, an advanced technology developed by Northeastern University and Stability AI, automatically converts 2D floor plans into 3D interior scenes. Based on a network-scale image-trained 2D diffusion model, it generates consistent multi-view...
What is HouseCrafter?
HouseCrafter, an advanced technology developed by Northeastern University and Stability AI, automatically converts 2D floor plans into 3D interior scenes. Based on a network-scale image-trained 2D diffusion model, it generates consistent multi-view color (RGB) and depth (D) images. Image autoregressive batch generation ensures global consistency, reconstructing high-quality 3D scenes. It simplifies the creation of complex virtual environments, allowing users to easily edit scene layouts by moving furniture on the floor plan, and updating the generated 3D scene in real time. This gives HouseCrafter broad application potential in architecture, interior design, and real estate.
HouseCrafter's main functions
- Conversion from 2D drawing to 3D sceneConvert 2D floor plans into complete 3D interior scenes, such as house models.
- Multi-view image generationGenerate consistent multi-view RGB and depth (RGB-D) images of the scene at different locations.
- Autoregressive image generationThe previously generated image is used as a condition to guide the generation of new images at adjacent locations.
- Global consistencyThe consistency of generated images is ensured through a global planar graph and an attention mechanism.
- 3D scene reconstructionBased on TSDF fusion technology, a 3D mesh model is reconstructed from the generated RGB-D image.
- User interaction and editingThis feature allows users to move furniture and other elements on the floor plan to edit the scene layout, and the 3D view will update accordingly.
HouseCrafter's technical principles
- 2D diffusion modelA 2D diffusion model trained on a large-scale network image is adjusted to generate RGB-D images.
- Autoregressive batch generationBased on previously generated images, new images are generated in batches to ensure spatial coherence.
- Layout guidanceUse a floor plan as a global layout guide, and ensure that the generated image is consistent with the floor plan through a layout attention layer.
- Deep information fusion: Simultaneously consider RGB and depth information during image generation to improve geometric and semantic consistency.
- Attention mechanismUpdate the cross-attention layer based on geometric information from the reference depth to improve the quality of image generation.
- 3D Reconstruction AlgorithmBased on TSDF fusion technology, multi-view RGB-D images are converted into 3D meshes.
HouseCrafter's project address
- Project official websiteneu-vi.github.io/houseCrafter
- arXiv technical paper:https://arxiv.org/pdf/2406.20077
Application scenarios of HouseCrafter
- Architectural Design and PlanningIt helps architects and designers quickly convert floor plans into 3D models, enabling better space planning and design reviews.
- Interior DesignInterior designers can create and modify interior design plans, and preview furniture placement and decoration effects.
- Real Estate MarketingReal estate developers use 3D models generated by HouseCrafter to provide virtual house tours for customers, enhancing the appeal of marketing materials.
- Game developmentGame designers can quickly build complex 3D game environments.
- Virtual Reality (VR) and Augmented Reality (AR)It provides detailed 3D indoor scenes for virtual reality and augmented reality applications, enhancing the user experience.
- Film and animation productionQuickly generate 3D scenes needed for movies or animations, improving production efficiency.