4D-LRM - A 4D reconstruction model developed by Adobe in collaboration with the University of Michigan and other institutions.

What is 4D-LRM?

4D-LRM (Large Space-Time Reconstruction Model) is a novel 4D reconstruction model jointly developed by researchers from Adobe Research, the University of Michigan, and other institutions. The model can quickly and with high quality reconstruct dynamic scenes with arbitrary new views and temporal combinations based on sparse input views and arbitrary time points. Based on a Transformer architecture, the model predicts a 4D Gaussian primitive for each pixel, achieving a unified spatial and temporal representation, and exhibits high efficiency and strong generalization capabilities. 4D-LRM demonstrates excellent performance across various camera settings, especially under alternating canonical views and frame interpolation settings, where the model effectively interpolates time to generate high-quality reconstruction results.

Main functions of 4D-LRM

• High-efficiency 4D reconstruction4D-LRM can quickly and with high quality reconstruct dynamic scenes with any new view and time combination from sparse input views and arbitrary time points. On a single A100 GPU, it can reconstruct a 24-frame sequence in less than 1.5 seconds, demonstrating efficiency and scalability.
• Strong generalization abilityIt supports generalization to new objects and scenes. The model exhibits good performance across various camera settings, especially under alternating canonical views and frame interpolation settings, where it effectively interpolates time to generate high-quality reconstruction results.
• Supports any combination of view and time.It supports the generation of dynamic scenes with arbitrary views and time combinations, providing new possibilities for understanding and generating dynamic scenes.
• Wide range of applicationsIt supports expansion to 4D generation tasks, and can generate 4D content with higher fidelity by combining with models such as SV3D.

4D-LRM Technical Principles

• 4D Gaussian representation (4DGS)4D-LRM represents each object in a dynamic scene as a set of 4D Gaussian distributions. Gaussian distributions capture the spatial position and appearance of objects, as well as their temporal changes. Each 4D Gaussian distribution is defined by parameters such as spatial center, temporal center, spatial scale, temporal scale, rotation matrix, and color.
• Transformer architecture4D-LRM processes input images based on a Transformer architecture. The input image is first segmented into image patches, which are then encoded into multi-dimensional vectors and used as input to the Transformer. The Transformer processes the input based on a multi-head self-attention mechanism and a multilayer perceptron (MLP), ultimately predicting a 4D Gaussian primitive for each pixel.
• Pixel-aligned Gaussian rendering4D-LRM uses pixel-aligned Gaussian rendering technology to project the predicted 4D Gaussian distribution onto the image plane and synthesizes the final image based on alpha blending.
• Training and optimization4D-LRM is trained on large-scale datasets and optimizes model parameters by minimizing the difference between the reconstructed image and the real image. During training, the model learns a general space-time representation that enables it to generalize to new objects and scenes, generating high-quality reconstruction results under sparse input conditions.

4D-LRM project address

• Project official websitehttps://4dlrm.github.io/
• GitHub repositoryhttps://github.com/Mars-tin/4D-LRM
• HuggingFace model libraryhttps://huggingface.co/papers/2506.18890
• arXiv technical paper: https://arxiv.org/pdf/2506.18890

Application scenarios of 4D-LRM

• Video game and film productionIt efficiently reconstructs and renders dynamic scenes, suitable for modeling complex scenes such as character animation and scene changes, significantly improving the visual effects of games and movies, supporting real-time rendering and multi-view generation, and enhancing the audience's immersion.
• Augmented Reality (AR) and Virtual Reality (VR)It provides a realistic and immersive experience for AR and VR applications, supports real-time interaction, and allows users to move and observe freely in the virtual environment.
• Robots and autonomous drivingIt helps robots and autonomous driving systems better understand and predict environmental changes, providing accurate path planning information.
• Digital content creationIt reduces the workload of manual modeling and animation production, and provides a wealth of editing options for video editing.
• Scientific researchIt is used to reconstruct and analyze biomedical imaging data, such as heartbeats and respiratory movements, to help researchers understand the dynamic processes within organisms.