OmniCam - A multimodal video generation framework jointly developed by Zhejiang University, Shanghai Jiao Tong University, and other universities.

What is OmniCam?

OmniCam is an advanced multimodal video generation framework that achieves high-quality video generation through camera control. It supports multiple input modalities, allowing users to provide text descriptions, video trajectories, or images as references to precisely control the camera's motion. OmniCam combines a large-scale language model (LLM) and a video diffusion model to generate spatiotemporally consistent video content. Through a three-stage training strategy—including large-scale model training, video diffusion model training, and reinforcement learning fine-tuning—it ensures the accuracy and consistency of the generated videos.

OmniCam's main functions

• Multimodal input supportUsers can provide text or video as trajectory references, and images or videos as content references, to achieve precise control over camera movement.
• High-quality video generationBased on a large-scale language model and video diffusion model, it generates high-quality videos with spatiotemporal consistency.
• Flexible camera control:
  • It supports frame-level control, allowing you to set the start and end frames for the operation.
  • It supports compound motion in any direction, camera zoom in and out, and movement and rotation to any angle.
  • It supports speed control, providing a foundation for fast editing.
  • It supports seamless connection of various operations, supports long sequence operations, and allows multiple instructions to be executed continuously.
  • It supports common effects such as camera rotation.
• Dataset supportThe OmniTr dataset was introduced, which is the first large dataset for multimodal camera control, providing a solid foundation for model training.

OmniCam's technical principles

• Trajectory PlanningAfter the user inputs text or video, OmniCam first converts this input into discrete motion representations, breaking down complex instructions into simple actions. Through a precise trajectory planning algorithm, it calculates the camera's specific position and pose in each frame, preparing for subsequent generation. Specifically, the algorithm models the camera's motion around the object's center as spherical motion, calculates the spatial position of each point on the trajectory, and converts it into a sequence of camera extrinsic parameters.
• Content renderingCombining user-provided content references (images or videos) with a planned camera trajectory, OmniCam uses advanced 3D reconstruction technology to render video frames from the initial perspective. During the rendering process, it uses information such as point clouds, camera intrinsics and extrinsic parameters, and optimizes camera intrinsics through specific algorithms to complete the rendering of video frames.
• Perfecting the detailsDuring the rendering process, OmniCam's diffusion model supplements the video frames with details based on its prior knowledge, filling in the blank areas and ultimately generating a complete and beautiful video.
• Large-scale model training: Fine-tuning was performed using Llama3.1 as the backbone network to train a large-scale model.
• Video diffusion model training: Train the video diffusion model.
• Reinforcement learning fine-tuningThe downstream video generation model is frozen and used as the reward model. The PPO algorithm is then used to fine-tune the large trajectory model to optimize its performance.

OmniCam's project address

• arXiv technical paper:https://arxiv.org/pdf/2504.02312

Application scenarios of OmniCam

• Film and television productionOmniCam can quickly generate complex camera movements, helping directors and producers save a lot of time and effort in designing and shooting shots, improving production efficiency, and realizing more creative ideas.
• AdvertisingAdvertisers can use OmniCam to quickly adjust the camera angle and motion trajectory according to different advertising needs, creating more attractive advertising videos to attract consumers' attention.
• Education and TrainingOmniCam generates vivid and engaging instructional videos, simplifying complex concepts. By watching these videos, students can better understand and master the knowledge, improving their learning outcomes.
• Smart securityOmniCam can be used in scenarios such as urban security monitoring, traffic management, and emergency command to achieve the integration and linkage of video resources from multiple departments.