LeviTor - A 3D target trajectory control video synthesis technology open-sourced by Nanjing University, Ant Financial, and other institutions.
LeviTor is an image-to-video synthesis technology developed by institutions such as Nanjing University, Ant Group, and Zhejiang University. It combines depth information and K-means clustering points to control the trajectory of 3D objects in a video, eliminating the need for explicit 3D trajectory tracking. LeviTor...
What is LeviTor?
LeviTor is an image-to-video synthesis technology developed by institutions such as Nanjing University, Ant Group, and Zhejiang University. It combines depth information and K-means clustering points to control the trajectory of 3D objects in videos, eliminating the need for explicit 3D trajectory tracking. Trained on high-quality video object segmentation datasets, LeviTor effectively captures object motion and interactions in complex scenes. Its user-friendly inference process simplifies 3D trajectory input, making video generation technology more advanced and easier to use. The introduction of LeviTor paves the way for 3D object trajectory control, broadens the scope of creative applications, and is suitable for a wider range of users.
LeviTor's main functions
- Precise control of object movementPrecisely control the motion trajectory of objects when generating video from still images.
- Enhance creative applicationsBased on 3D trajectory control, the creative application scope of video synthesis is broadened.
- Simplify user inputUsers can input 3D trajectories by drawing and adjusting depth on simple 2D images, lowering the technical barrier.
- Automatically extract depth information and object maskThe system automatically extracts depth information and object masks from images, reducing user operations.
- Interactive trajectory drawingUsers interactively draw object trajectories, which the system interprets as 3D paths.
LeviTor's technical principles
- K-means clusteringK-means clustering is performed on the pixels of the video object mask to obtain a representative set of control points.
- Deep information fusionThe DepthAnythingV2 depth estimation network predicts the relative depth map and samples the depth at each control point to add depth information to the control points.
- Control signal constructionBy combining 2D coordinates and estimated depth values, a control trajectory is constructed, which serves as the control signal for the video diffusion model.
- Video diffusion modelThe control signal is input into the video diffusion model to generate a video aligned with the 3D trajectory.
- User-friendly reasoning processDesign a user-friendly interactive system where users input 3D trajectories by clicking and adjusting depth values.
LeviTor project address
- Project official website:ppetrichor.github.io/levitor
- GitHub repository:https://github.com/qiuyu96/LeviTor
- HuggingFace model library:https://huggingface.co/hlwang06/LeviTor
- arXiv technical paper:https://arxiv.org/pdf/2412.15214
Application Scenarios of LeviTor
- Movie special effects productionIt generates realistic special effects scenes, reduces on-location shooting costs, and improves production efficiency.
- Game animation generationIn game development, creating dynamic game backgrounds and character animations enhances the game's immersive experience.
- Virtual Reality ExperienceIn VR applications, realistic virtual environments are synthesized to provide users with a more immersive and authentic experience.
- Augmented Reality DisplayIn the field of AR, virtual information is seamlessly integrated with the real world and used in scenarios such as education and navigation.
- Advertising video productionCreate dynamic advertising videos to attract viewers' attention and enhance brand image and product appeal.