NutWorld - A video processing framework developed by the National University of Singapore, Nanyang Technological University, and Skywork AI.

What is NutWorld?

NutWorld is a video processing framework developed by the National University of Singapore, Nanyang Technological University, and Skywork AI. It efficiently converts everyday monocular videos into dynamic 3D Gaussian splatting. Based on the Spatiotemporally Aligned Gaussian (STAG) representation, NutWorld achieves spatiotemporally coherent modeling of videos in a single feedforward pass, overcoming the limitations of traditional methods under complex motion and occlusion. NutWorld combines depth and optical flow regularization techniques to effectively address spatial blur and motion uncertainty issues in monocular videos. NutWorld supports high-fidelity video reconstruction and provides real-time support for various downstream tasks, such as new view compositing, video editing, frame interpolation, and consistent depth prediction.

NutWorld's main functions

• High-efficiency video reconstructionConvert everyday monocular videos into dynamic 3D Gaussian splatting and reconstruct the video content with high fidelity.
• Real-time processing capabilityIt supports real-time processing, which is significantly better than traditional optimization methods.
• Support for multiple downstream tasks:
  • New View CompositionGenerate new perspectives from monocular video.
  • Video editingSupports precise frame-level editing and stylization.
  • Frame interpolationGenerate intermediate frames to improve video frame rate.
  • Consistent depth predictionProvides spatiotemporally coherent depth estimation.
  • Video object segmentationTarget segmentation is achieved based on the propagation object mask.
• Spatiotemporal coherenceBased on the structured spatiotemporally aligned Gaussian (STAG) representation, it ensures the temporal and spatial coherence of the video.

NutWorld's Technical Principles

• Spacetime Aligned Gaussian (STAG) RepresentationEach pixel in the video is associated with a 3D Gaussian distribution, constrained by spatiotemporal alignment. Each Gaussian distribution has attributes such as position, scale, color, and opacity, and temporal dynamics are captured based on a deformation field.
• Feedforward network architectureA Transformer-based encoder-decoder architecture maps input video frames to STAG representations. The encoder processes input frames using Transformer blocks, capturing spatiotemporal correspondences; the decoder predicts static Gaussian properties and their deformable fields, supporting efficient feedforward prediction.
• Depth and optical flow regularizationDepth regularization and optical flow regularization are introduced. Depth regularization enhances the robustness of depth prediction based on calibrated monocular depth priors. Optical flow regularization supervises the motion trajectory of a Gaussian distribution using a pre-computed optical flow field, ensuring temporal consistency.
• Fragment-based reasoningThe segment-based inference strategy divides long videos into multiple overlapping segments for processing. A Gaussian distribution is propagated in the overlapping frames to maintain global spatiotemporal consistency.

NutWorld's project address

• GitHub repository:https://github.com/Nut-World/NutWorld/
• arXiv technical paper:https://arxiv.org/pdf/2502.03465

Application scenarios of NutWorld

• Video content creation and editingUsed for new view compositing to generate new perspectives from monocular video; supports frame interpolation to improve video frame rate; enables precise video editing and stylization, providing powerful tools for video creators.
• Augmented Reality and Virtual RealityReal-time reconstruction of dynamic 3D scenes provides more accurate scene understanding for AR/VR applications; natural integration of virtual objects into real-world scenes enhances user experience.
• Autonomous driving and robot visionIt can efficiently reconstruct dynamic scenes, provide depth and motion information to help autonomous driving achieve environmental perception; and support robots in real-time modeling and human-computer interaction in complex environments.
• Game developmentIt generates high-quality 3D scenes in real time, supporting interactive content and a smooth gaming experience.
• training industryIt is used in training simulations for driving, flying, etc., to provide realistic dynamic scenarios and improve training effectiveness.