WorldCraft - A 3D virtual world creation and customization system launched by HKUST

What is WorldCraft?

WorldCraft, developed by the Hong Kong University of Science and Technology, is a 3D world creation and customization system based on Large Language Model (LLM) proxies. It's used to create and customize realistic 3D virtual worlds. Based on natural language interaction, it allows users to easily generate complex indoor and outdoor scenes and fine-tune objects and layouts within them. WorldCraft consists of three core modules: ForgeIt, used for customizing individual objects, achieves precise geometric and texture control through procedural generation; ArrangeIt, responsible for scene layout generation, achieves a balance between functionality and aesthetics through layered numerical optimization; and the trajectory control module allows users to control object movement and camera trajectory through dialogue, generating animations and videos. WorldCraft is compatible with existing deep 3D generators, enriching scene assets and enabling non-professionals to efficiently realize their creative designs.

Main functions of WorldCraft

• Natural Language InteractionUsers can create and customize complex 3D scenes based on simple text commands or dialogues, without the need for professional 3D modeling skills.
• Customization of a single objectUsers can precisely control the geometry, texture, and material of objects to achieve personalized designs.
• Scene layout generationGenerate a reasonable object layout based on the user's design intent (such as functional and aesthetic requirements).
• Scene animation controlUsers can generate animations and videos based on natural language descriptions of the motion trajectories of objects or cameras.
• Compatibility and scalabilityThe system supports integration with existing depth 3D generators (such as Infinigen) to enrich scene assets and enhance visual effects.

The technical principles of WorldCraft

• Coordinator AgentManages the entire 3D scene generation process, breaking down complex tasks into multiple sub-tasks and coordinating other agent modules to complete them. Supports multiple rounds of user interaction with the system, allowing for real-time adjustments to the generated results.
• Single object generation and customizationThis tool utilizes LLM agents to operate procedural generators (such as Infinigen), generating 3D objects with precise geometry and PBR textures using natural language instructions. It employs an automated verification mechanism to build a dynamically growing manual that guides the agent through complex generation tasks, supporting multi-turn dialogue editing.
• Scene layout optimizationThe system models the scene layout problem as a hierarchical numerical optimization problem and solves for the optimal layout based on the simulated annealing algorithm. It allows users to describe their layout requirements in natural language, transforming design intent into optimization goals and constraints.
• Trajectory control moduleBased on the ChatCam extension, it allows users to describe the motion trajectories of objects or cameras using natural language. These trajectory descriptions are then translated into specific motion commands to generate scene animations.
• Deep 3D Generator IntegrationThe system is compatible with existing depth 3D generators, used to introduce artistic objects and enrich scene details. It enhances the visual quality and diversity of scenes through a combination of procedural and depth generation.

WorldCraft project address

• arXiv technical paper:https://arxiv.org/pdf/2502.15601

Application scenarios of WorldCraft

• Architectural Design and Interior DesignIt can quickly generate architectural concept models and virtual showrooms, customize interior layouts and styles based on natural language, optimize the design process, and enhance the customer experience.
• Film and EntertainmentIt creates virtual movie scenes, game levels, and complex scenes in animated films, reducing the cost of building physical sets and providing highly realistic virtual environments for virtual reality (VR) and augmented reality (AR) experiences.
• Education and TrainingDesign virtual laboratories, historical scenes, geographical environments, and medical training scenarios to enhance learning outcomes through immersive experiences and support distance education and professional skills training.
• Product Design and PresentationIt can quickly generate product concept models and virtual showrooms to showcase the designs of products such as furniture, automobiles, and electronic devices, and supports personalized customization and the creation of virtual advertising scenarios.
• Culture and ArtCreate virtual museums, art exhibitions, historical buildings, and cultural festival scenes to recreate cultural heritage and provide immersive spaces for artistic creation and cultural experiences.