Lyra 2.0 - NVIDIA's open-source explorable generative 3D world framework

What is Lyra 2.0?

Lyra 2.0 is an explorable generative 3D world framework from NVIDIA. Starting from a single image, the framework combines camera-controlled video generation with feedforward 3D reconstruction technology, constructing large-scale 3D scenes that can be persistently roamed through an iterative "retrieve-generate-update" loop. The system employs an independent 3D geometry cache for each frame for spatial memory retrieval and introduces a self-reinforcing training strategy to suppress temporal drift, achieving consistent 3D generation over hundreds of frames. The generated results from Lyra 2.0 can be reconstructed into high-fidelity 3D Gaussian Splatting and surface meshes, directly exported to physics engines such as NVIDIA Isaac Sim, providing an interactive simulation environment for embodied intelligent training.

Main features of Lyra 2.0

• Long-range 3D consistent video generationGenerates hundreds of frames of roaming video along a user-defined camera trajectory, supporting large-angle changes and area revisiting.
• Spatial memory retrievalA spatial cache is built based on the 3D geometry of each frame, and the historical frames most relevant to the target viewpoint are automatically retrieved as conditions.
• Time-drift-resistant generationBy employing a self-reinforcing training strategy, the model actively corrects error accumulation during autoregressive inference, maintaining long-term visual consistency.
• Interactive 3D ExplorerIt provides a GUI to visualize the accumulated point cloud, allowing users to plan trajectories to revisit explored areas or enter unobserved areas.
• Feedforward 3D ReconstructionThe generated video is reconstructed into a high-fidelity 3D Gaussian Splatting with surface mesh using a finely tuned feedforward model.
• Exporting Simulated AssetsIt supports direct export to physics engines such as NVIDIA Isaac Sim for robot navigation and interaction training.
• Accelerated Reasoning VersionIt provides a 4-step denoising student model based on distribution matching distillation, which improves inference speed by approximately 13 times.

The technical principles of Lyra 2.0

• Generative Reconstruction ParadigmCombining the visual fidelity of camera-controlled video diffusion models with feedforward 3D reconstruction technology, a single image + trajectory is transformed into a 3D output that can be rendered in real time.
• Decoupling geometry routing and appearance compositionMaintain an independent 3D cache (depth map + point cloud) for each frame, used only for retrieving historical frames and establishing dense 3D correspondences; actual pixel synthesis is still completed by the generation prior of the video diffusion model to avoid the propagation of rendering artifacts.
• Normalized coordinate warp injectionThe retrieved historical frames are forward-warped to the target viewpoint to generate a normalized coordinate map and a depth map. After position encoding and MLP processing, the maps are injected into the self-attention layer of DiT to provide a precise geometric alignment signal.
• Self-reinforcing anti-drift trainingThe historical latent variables in training are noisy using probability and replaced with approximate reconstruction through single-step denoising, forcing the model to recover the clean target even under imperfect conditions, thus reducing the training-inference distribution difference.
• FramePack context compression: Uses variable kernel patchification to compress time history, with fine-grained patching for near frames and coarse-grained patching for far frames, expanding the effective context window within a fixed token budget.
• Fine-tuning feedforward reconstructionBased on Depth Anything v3, the Gaussian prediction density at high resolution is improved and fine-tuned on Lyra 2.0 generated data to enhance robustness to generated artifacts.

How to use Lyra 2.0

• Cloning repositoryPull code from GitHub repository and configure environment dependencies according to the README.
• Download ModelGet Lyra 2.0 pre-trained weights from Hugging Face or the project page.
• Ready to inputProvides a single scene image, with optional text prompts to guide the generation style.
• Start the explorerRun an interactive GUI to load images and plan camera roaming paths.
• Iterative generationThe system automatically retrieves relevant historical frames from spatial memory and generates long-range videos segment by segment.
• Reconstructing 3D: Call the fine-tuned feedforward model to convert the video sequence into 3D Gaussian Splatting.
• Extract MeshThe script outputs a surface mesh, supporting the extraction of layered sparse meshes.
• Export DeploymentImport 3D assets into physics engines such as NVIDIA Isaac Sim for embodied intelligence training.

Key information and usage requirements for Lyra 2.0

• Project PositioningNVIDIA's open-source, explorable generative 3D world framework supports iterative construction of persistent, roamable, large-scale 3D scenes from a single image.
• Core technologiesThe video diffusion model is based on Wan 2.1 VAE + DiT and adopts an autoregressive loop of "retrieval-generation-update". Spatial forgetting is solved by independent 3D geometric caching for each frame and temporal drift is suppressed by self-reinforcing training.
• Input/OutputThe input is a single RGB image with optional text prompts and camera trajectory; the output is long-range camera control video, which can be further reconstructed into 3D Gaussian Splatting and surface mesh, and can be exported to a physics engine.
• Performance optimizationIt provides a 4-step accelerated model based on distribution matching distillation, which improves inference speed by approximately 13 times.
• Hardware environmentRequires an NVIDIA GPU (high VRAM is recommended to support long video generation and 3D reconstruction), and a CUDA environment is essential.
• Software Dependency: PyTorch, diffusers, transformers, FramePack, Depth Anything V3, OpenVDB, etc. Specific versions can be found in the repositories. requirements.txt As the standard.

Lyra 2.0's core advantages

• Global space persistenceBy using independent per-frame geometry caching and visibility retrieval, the spatial forgetting problem in long-range generation is solved, and the revisited regions maintain structural consistency.
• Long-term visual stabilitySelf-reinforcement training effectively suppresses the accumulation of autoregressive errors and significantly reduces color drift and geometric distortion compared to the baseline.
• High-quality 3D outputThe feedforward reconstruction model is fine-tuned for the generated data, tolerates slight inconsistencies across multiple views, and produces clean and coherent 3DGS and meshes.
• Interactive controllableUsers can plan camera trajectories of any length in real time, and the system progressively expands the scene, rather than generating a one-time black box.

Lyra 2.0 project address

• Project official website: https://research.nvidia.com/labs/sil/projects/lyra2/
• GitHub repositoryhttps://github.com/nv-tlabs/lyra
• HuggingFace model libraryhttps://huggingface.co/nvidia/Lyra-2.0
• arXiv technical paper: https://arxiv.org/pdf/2604.13036

Comparison of Lyra 2.0 with similar competing products

Application scenarios of Lyra 2.0

• Embossed Intelligent SimulationGenerate interactive 3D indoor/outdoor environments for robot training, replacing costly real-world scene data collection.
• Virtual World Construction: Quickly generate roamable game levels or metaverse scene prototypes from a single concept art.
• Architecture and Interior DesignGenerates 3D walkthroughs based on floor plans or renderings, allowing customers to immerse themselves in previewing the spatial layout.
• Film and television pre-productionIt provides directors with a rapid visualization solution from static concept art to dynamic scene walkthroughs.