LatticeWorld - A multimodal 3D world generation framework developed by NetEase, Tsinghua University, and others.

What is LatticeWorld?

LatticeWorld is a multimodal 3D world generation framework developed by NetEase, City University of Hong Kong, Beijing University of Aeronautics and Astronautics, and Tsinghua University. It combines large language models with the industrial-grade 3D rendering engine Unreal Engine 5 (UE5). It supports the rapid generation of large-scale interactive 3D worlds with highly dynamic environments, realistic physical simulation, and real-time rendering through text descriptions and visual instructions. LatticeWorld consists of three core modules: scene layout generation, environment configuration generation, and procedural rendering pipeline. Compared to traditional manual creation, it improves creation efficiency by over 90 times and produces high-quality results, making it promising for widespread application in games, film, and other fields.

Main functions of LatticeWorld

• Quickly generate 3D worldBased on text descriptions and visual instructions, it can quickly generate large-scale interactive 3D worlds with highly dynamic environments, realistic physical simulations, and real-time rendering, significantly improving efficiency compared to traditional manual creation.
• Supports generation of multiple scenariosIt supports the generation of various types of 3D scenes, such as suburbs and wilderness, to meet the needs of different users.
• Generate a dynamic interactive environmentIt can generate dynamic agent configuration information, including agent type, number, behavioral state and spatial location, making the generated 3D world more interactive and dynamic.
• Supports multimodal inputIt supports multiple input methods, such as text description and visual conditions, allowing users to choose the appropriate input method according to their needs, making the generated 3D world more in line with their requirements.

LatticeWorld's Technical Principles

• Multimodal input processingBy using various input methods such as text descriptions and visual conditions, the input information is converted into symbolic scene layouts and environment configuration parameters, providing a foundation for subsequent 3D world generation.
• Symbol sequence scene layout representationThe complex spatial scene layout is converted into a symbol matrix, where each symbol represents a specific asset type, such as F representing a forest area and W representing a water body. The symbolized matrix is serialized into a string format that can be directly processed by a large language model, effectively preserving the integrity of spatial relationships.
• Multimodal visual fusion mechanismHigh-dimensional visual feature representations are extracted using a pre-trained CLIP visual encoder, and these features are mapped to the word embedding space through a specially designed multi-layer CNN projection network. A carefully designed three-stage training paradigm is employed, including a CLIP fine-tuning stage, a continuous pre-training stage, and an end-to-end fine-tuning stage, to jointly optimize the visual instruction integration module and the layout generation model.
• Hierarchical scene attribute frameworkEstablish a hierarchical structure for two-layer scene attributes. The coarse-grained attribute layer controls global environmental features, such as terrain type, seasonal changes, and weather conditions. The fine-grained attribute layer covers a variety of detailed parameters, such as the material, density, location, and orientation of assets, to ensure semantic consistency of the scene and reduce parameter conflicts.
• Procedural rendering pipelineThe system utilizes a scene layout decoder and an environment configuration translation system to convert symbolic scene layouts and JSON-formatted environment configurations into input for the 3D rendering engine. The scene layout decoder performs precise processing of the symbolic scene layout, including converting it into a binary mask image, adjusting the mask image size, and creating natural transition areas. The environment configuration translation system is responsible for translating environment configuration parameters into the native attribute format of the specific rendering engine, precisely controlling detailed parameters such as the type, state, and spatial distribution of objects and agents.

LatticeWorld's project address

• arXiv technical paper: https://arxiv.org/pdf/2509.05263

Application scenarios of LatticeWorld

• Game development: Quickly generate game world prototypes, including elements such as terrain, buildings, and vegetation, to accelerate the development process.
• Film and television productionIt can quickly build complex virtual scenes, such as alien worlds and ancient cities, reducing the cost of building physical scenes.
• Virtual Reality (VR) and Augmented Reality (AR)Create immersive virtual environments, such as virtual tourism and virtual education, to provide an immersive experience.
• Urban PlanIt can quickly generate virtual city models, including elements such as streets, buildings, and parks, for preliminary research.
• Education and TrainingIt generates virtual laboratories, historical scenes, etc., providing an immersive learning experience.