VideoDoodles - Adobe's AI video editing framework

What are VideoDoodles?

VideoDoodles is an AI video editing framework developed by Adobe in collaboration with several universities. It allows users to easily insert hand-drawn animations into videos, achieving seamless integration with the video content. By preprocessing video frames, the system provides a flat canvas on which users can draw animations, and the system automatically handles perspective and occlusion effects. VideoDoodles' tracking algorithm enables animations to follow the movement of objects in the video, simplifying the animation production process.

Main functions of VideoDoodles

• Hand-drawn animation insertionUsers can add hand-drawn style animation elements to their videos.
• Automatic fusionThe animation and video content are seamlessly integrated, requiring no complex post-production processing.
• 3D scene rendering: Place a flat canvas in three-dimensional space to achieve automatic rendering of perspective and occlusion effects.
• Keyframe controlThe system automatically generates intermediate animations by setting the start and end states of the animation through keyframes.
• Dynamic object trackingA custom algorithm enables animations to follow the movement of dynamic objects in a video.

The technical principles of VideoDoodles

• Video preprocessingVideoDoodles analyzes the input video, extracting camera pose, depth map, and optical flow information for each frame, laying the foundation for subsequent animation compositing.
• Plane Canvas MappingUsers place a virtual flat canvas in 3D space, which can be mapped to specific areas of the video. VideoDoodles uses camera pose and depth map information to automatically render a canvas with correct perspective and occlusion effects.
• Tracking AlgorithmVideoDoodles uses a custom tracking algorithm, allowing the canvas to follow static or dynamic objects in a video. Users can control the canvas's position and orientation by setting keyframes, and the system automatically calculates the optimal 3D trajectory.
• 3D position and orientation tracking:
  • Location trackingThe keyframe-based 2D tracking algorithm constructs a directed graph to search for the trajectory of an object, and the relative position of the canvas and the object remains consistent.
  • Direction trackingUsers can specify the rotation matrix of the canvas in the keyframe and use an optimized algorithm to keep the canvas orientation consistent with the orientation of the moving object.
• Automated animation generationOnce the user draws the animation elements on the keyframes, VideoDoodles automatically generates the entire animation process without requiring the user to draw frame by frame.
• User interface designVideoDoodles provides an intuitive user interface that allows users to easily select, draw, and adjust animated elements.

VideoDoodles project address

Application scenarios of VideoDoodles

• Social media content creationUsers can add fun hand-drawn animations to their social media videos to increase their appeal and interactivity.
• Educational video productionTeachers or educators can add animations to instructional videos to explain complex concepts or processes in a more vivid way.
• CommercialsAdvertisers can use VideoDoodles to create engaging animated ads for products or services, enhancing the creativity and memorability of their ads.
• Film and video post-productionVideo editors can use VideoDoodles to add unique visual effects to their work during film or video post-production.