CineMaster - A text-to-video generation framework launched by Kuaishou, featuring 3D perception capabilities.

What is CineMaster?

CineMaster is a text-to-video generation framework with 3D perception capabilities launched by Kuaishou. Similar to a video version of ControlNet, it allows users to precisely manipulate the position of objects and camera movement in videos using various control signals. It can generate videos using text prompts and can be finely adjusted by combining signals such as depth maps, camera trajectories, and object labels. Kuaishou provides a workflow for extracting 3D bounding boxes and camera trajectories from large-scale videos, providing strong data support for the training and application of CineMaster.

CineMaster's main functions

• 3D Object and Camera ControlUsers can freely adjust the position, size, and motion trajectory of objects in 3D space, while defining the camera's motion (such as translation and rotation) to achieve precise scene layout and lens design.
• Interactive Design and Live PreviewThrough the interactive interface, users can preview the 3D layout effect in real time and gradually optimize the design until the ideal visual effect is achieved, similar to the shot design in film shooting.
• 3D-aware video generationBased on depth maps, object labels, and camera trajectories as conditional signals, CineMaster can generate video content that accurately matches the user's design intent and supports complex object and camera movements.
• Automated data annotationIt provides an automated process to extract 3D bounding boxes and camera trajectories from ordinary videos, solving the problem of scarcity of large-scale 3D labeled data and improving the training effect of the model.
• High-quality video outputIt supports the generation of high-quality and diverse video content, suitable for various creative scenarios, and meets both professional and entertainment needs.

CineMaster's technical principles

• Two-stage workflow
  • Phase 1Users define the layout of objects and cameras through a 3D interface, generating depth maps and motion trajectories as control signals.
  • Phase TwoThe control signal is input into the diffusion model to generate a video that matches the design intent.
• Semantic layout control networkBy combining 3D spatial layout and semantic information, and fusing depth maps and object labels through MLP, video generation can be accurately guided.
• Camera adapterInjects camera pose into the generation process, distinguishes between object and camera motion, and supports complex shot design.
• Automated data annotation pipelineUsing instance segmentation, depth estimation, and 3D point cloud computing, 3D bounding boxes and camera trajectories are extracted from ordinary videos to provide data support for model training.
  • Instance splittingUse Grounding DINO and SAM 2 to obtain foreground instance segmentation results.
  • Depth estimationGenerate a metric depth map using DepthAnything V2.
  • 3D point cloud and bounding box calculation: Calculate the 3D point cloud of each entity by back projection, and calculate the 3D bounding box with the smallest volume.
  • Entity tracking and 3D bounding box adjustmentUse SpatialTracker for point tracking, calculate 3D bounding boxes for each frame, and render a depth map.
• Diffusion Model and Training StrategyBased on a pre-trained diffusion model, the generation quality and controllability are improved through multi-stage training (including depth map training, semantic layout training, and joint training).

CineMaster's project address

• Project official website:https://cinemaster-dev.github.io/
• arXiv technical paper:https://arxiv.org/pdf/2502.08639

Application scenarios of CineMaster

• Film and television productionUsed for storyboard design, special effects preview, and animation production, it helps directors and creators quickly realize their ideas and improve production efficiency.
• Advertising and MarketingCreate creative videos, product demonstrations, and virtual scenarios to enhance advertising appeal and user experience.
• Game developmentGenerate cutscenes, scene designs, and character animations to enhance the game's narrative and visual effects.
• Education and trainingTo improve teaching and training effectiveness, we will create instructional videos, virtual experiments, and safety training content.
• Entertainment and social mediaCreate creative videos, virtual tours, and interactive entertainment content to enhance user engagement and immersion.