I2V3D - An image-to-video generation framework jointly developed by City University of Hong Kong and Microsoft.

What is I2V3D?

I2V3D is an innovative image-to-video generation framework developed by City University of Hong Kong and Microsoft GenAI. It supports the conversion of still images into dynamic videos, achieving precise animation control based on 3D geometry guidance. I2V3D combines the precise control capabilities of traditional computer graphics (CG) pipelines with the visual fidelity of generative AI models, using a two-stage generation process: 3D-guided keyframe generation and video interpolation, to achieve high-quality, controllable video generation. I2V3D supports complex 3D animation and camera movements, allowing users to start animations from any initial point and generate video sequences of any length. I2V3D lowers the barrier to entry for creation, simplifies the video generation process, and provides an efficient and flexible solution for animation production, video editing, and content creation.

Main functions of I2V3D

• Conversion from still image to moving videoConvert a single still image into a video with dynamic effects, supporting complex animations and camera movements.
• Precise 3D controlIt enables precise control of animation based on 3D guidance, including object rotation, translation, scaling, and camera movement (such as rotation, translation, zoom, etc.).
• Flexible animation starting pointSupports freely defining the starting frame of animations and generating videos of arbitrary length.
• Support for editing complex scenesUsers can add, copy, replace, or edit objects in a 3D scene to generate new video content.

I2V3D Technical Principles

• 3D geometric reconstructionThe process involves reconstructing the complete 3D scene geometry from a single image, including foreground objects and the background. Foreground objects are extracted and converted into 3D meshes, while the background is generated using multi-view generation and 3D mesh reconstruction.
• Two-stage video generation process:
  • 3D-guided keyframe generationA customized image diffusion model is used, guided by coarse rendering results, to generate high-quality keyframes. Multi-view enhancement and extended attention mechanisms improve the model's generalization ability and temporal consistency.
  • 3D-guided video interpolationGenerates smooth, high-quality video frames between keyframes. No training is required; bidirectional guidance (forward and backward) ensures the temporal coherence of the video.
• Deep guidance and feature controlDuring video generation, depth maps and rendering features (such as self-attention features and convolutional features) are used as control signals to ensure that the generated video is consistent with the 3D rendering result.
• Extended attention mechanismBased on the extended attention mechanism, the spatiotemporal consistency between frames is enhanced during the keyframe generation stage, avoiding flickering or discontinuity issues in the generated video.

I2V3D project address

• Project official website:https://bestzzhang.github.io/I2V3D/
• arXiv technical paper:https://arxiv.org/pdf/2503.09733

Application scenarios of I2V3D

• Animation ProductionIt can quickly generate dynamic videos from static images, supports complex 3D animations, and is suitable for short animation production such as advertisements and games.
• Video editing and creationAdd, replace, or modify objects in a 3D scene to generate creative video content, suitable for short videos and special effects previews.
• VR/AR content generationGenerate realistic 3D dynamic content for interactive demonstrations in virtual environments, enhancing immersion.
• Education and TrainingConverting static teaching illustrations into dynamic videos helps students understand complex concepts more intuitively.
• Game developmentQuickly generate game cutscenes or virtual character animations, saving development time and costs.