ViewExtrapolator - A novel view synthesis method developed by Nanyang Technological University in collaboration with the UCAS team.

What is ViewExtrapolator?

ViewExtrapolator, a novel perspective extrapolation method jointly developed by research teams from Nanyang Technological University and UCAS, synthesizes new perspectives far exceeding the range of the training view based on the generation prior of Stable Video Diffusion (SVD). This method effectively optimizes views prone to artifacts in radiance field or point cloud rendering by redesigning the SVD denoising process, generating clearer and more realistic new perspective images. ViewExtrapolator does not require fine-tuning of SVD, is data and computationally efficient, and can be widely applied to various 3D rendering techniques.

Main functions of ViewExtrapolator

• New Perspective ExtrapolationGenerating new perspective images beyond the training view is crucial for providing an immersive 3D experience and free exploration of the reconstructed radiation field.
• Artifact reductionBased on the generation prior of stable video diffusion (SVD), artifacts generated by radiation field or point cloud rendering are optimized to improve the visual quality of synthesized new perspectives.
• Data and computing efficiencyAs an inference-stage method that does not require fine-tuning of SVD, it is highly efficient in terms of both data and computation, making new perspective extrapolation more practical and accessible.
• Wide applicabilityIt can be used in conjunction with different 3D rendering methods, including point cloud rendering derived from single-view or monocular video, and has good versatility and adaptability.

The technical principles of ViewExtrapolator

• SVD denoising processBased on the Static Video Diffusion (SVD) model, the model generates videos based on progressively denoised Gaussian noise.
• Guided annealing and resampling annealingGuided annealing and resampling annealing techniques are introduced. By applying different levels of guidance at different stages of the denoising process, unseen areas are effectively repaired and visual quality is improved.
• Multi-view consistencyUsing SVD's multi-view consistency, natural video details are generated for parts of the radiation field or point cloud rendering that are not observed, while preserving the original content, including camera motion and scene dynamics.
• No training requiredUsing the generative priors of SVD for new perspective extrapolation is highly efficient in terms of both data and computation.
• The flexibility of 3D renderingViewExtrapolator can handle both radiation fields generated from multi-view images and point clouds generated from single views, effectively improving rendering quality.

ViewExtrapolator project address

• Project official website:kunhao-liu.github.io/ViewExtrapolator
• GitHub repository:https://github.com/Kunhao-Liu/ViewExtrapolator
• arXiv technical paper:https://arxiv.org/pdf/2411.14208

Application scenarios of ViewExtrapolator

• Virtual Reality (VR) and Augmented Reality (AR)In VR and AR applications, it generates new perspectives beyond the pre-rendered view, enhancing the user experience.
• 3D content creationFor 3D artists and game developers, compositing complete 3D scenes from limited views reduces the amount of manual modeling work and accelerates the content creation process.
• Film and video productionIn filmmaking, it is used to generate perspectives that are not actually captured by the camera, or to create complex special effects scenes.
• Digitization of cultural relics and historical sitesFor the digital preservation of cultural relics and historical sites, a complete 3D model can be reconstructed from a limited perspective, making research and display more comprehensive.
• Visualization of architecture and interior designIn the fields of architecture and interior design, it helps designers preview design effects from different angles, even those that are difficult to achieve in actual shooting.