project
UniWorld-View - A world model jointly developed by RabbitShow Intelligence and Peking University, among others, and open-sourced.
UniWorld-View is an open-source world model developed by RabbitShow Intelligence in collaboration with Peking University and Pengcheng Laboratory. It topped the WorldScore world model evaluation list from Fei-Fei Li's team. The model is based on a single image or a video clip and can be accessed by pressing...
What is UniWorld-View?
UniWorld-View is an open-source world model developed by RabbitShow Intelligence in collaboration with Peking University and Pengcheng Laboratory. It topped the WorldScore world model evaluation list by Fei-Fei Li's team. Based on a single image or a video clip, the model can generate new perspective images according to a specified camera trajectory, achieving a unified architecture for single-image 3D generation and video 4D generation. It has been adapted to the domestic Ascend computing power, and both the code and weights are open source.
Main functions of UniWorld-View
-
Single Image New Perspective SynthesisInput a single image and generate multi-view images according to the specified camera path, supporting push, pull, pan, tilt, and 360° surround.
-
New Perspective Synthesis of Dynamic VideoInput monocular video, infer uncaptured space and generate dynamic video from a new perspective, maintaining temporal consistency.
-
Unified Generation FrameworkThe same model and code can simultaneously support new perspective generation tasks for both static images and dynamic videos.
-
Camera trajectory controlIt supports precise camera pose control and enables large baseline viewpoint switching.
The technical principles of UniWorld-View
-
Occlusion-aware point cloud renderingTo address the issues of foreground-background tearing and backlight leakage in point cloud rendering, a dual projection and normal filtering mechanism is proposed.
-
Double-reprojectionThe source image is projected onto the target viewpoint and then projected back along the same path. Pixels that cannot be returned are the occluded areas. These are accumulated frame by frame to form an occlusion mask, which avoids incorrect textures from misleading the generated model.
-
Normal-FilteringUse normal information to remove backside point clouds to prevent frontside pixel leakage when the camera moves behind an object.
-
Fusion of geometry and generationFirst, a 3D point cloud is constructed using a geometric model and the target viewpoint condition map is rendered. Then, the video diffusion model is input to complete the generation, balancing camera control precision and image quality.
Follow us on WeChat and reply with "open source",join inAI open source project discussion group
How to use UniWorld-View
-
Environmental preparationClone the GitHub repository, configure the Python environment, and install dependencies.
-
Model DownloadDownload the open-source weights from Hugging Face or GitHub Release.
-
Data inputPrepare a single image or a video clip as source material and define the target camera trajectory.
-
Running inference: Execute the inference script, and the model will automatically generate new images or videos from the specified perspective.
-
Ascend DeploymentIf you require domestic computing power, follow the official documentation to complete the Ascend NPU adaptation and inference acceleration.
UniWorld-View project address
- GitHub repository：https://github.com/PKU-YuanGroup/UniWorld-View
- HuggingFace model library：https://huggingface.co/Drexubery/UniView
Comparison of UniWorld-View with similar competing products
| Dimension | UniWorld-View | WorldScape-0.2(MoE) |
|---|---|---|
| WorldScore Total Score | Multiple firsts (Static 85.53 / Camera control 97.72 / 3D consistency 91.63) | The dynamic score of 76.23 is slightly high, while it lags behind in other dimensions. |
| technical route | Occlusion-aware point cloud rendering + video diffusion model | MoE Architecture Generation Path |
| Open source situation | The code and weights are all open source. | Not explicitly stated as fully open source |
| Domestic computing power | Already adapted to Ascend | Not mentioned |
| Task uniformity | Single model covering single-image 3D and video 4D | Focus on world generation |
Application scenarios of UniWorld-View
-
Film and advertising productionQuickly generate multi-camera previews of scenes, reducing the cost of live-action shooting and 3D reconstruction.
-
Virtual Reality and GamesGenerate interactive 360° surround scenes based on a single concept image or video.
-
Robotics and Autonomous Driving Simulation: Construct new perspective data of the real world for training visual perception and path planning models.
-
Digitalization of cultural heritageAfter capturing a single image or video of a cultural relic or ancient building, a digital display with a surround view is generated.
-
E-commerce and real estate: Generate multi-angle display content using a single product image or room video to enhance user immersion.