AB
AB
AiBoss

All Categories

project

Motionshop - Alibaba's AI character animation project that can replace video characters with 3D animation.

Motionshop is an AI character animation framework developed by Alibaba Research Institute of Intelligent Computing. It can replace characters in a video with 3D avatars without altering other scenes or characters in the video. This framework utilizes video processing...

What is Motionshop?

Motionshop is an AI character animation framework developed by Alibaba's Intelligent Computing Research Institute (XR Lab and TIDE 3D Rendering Team). It can replace characters in videos with 3D avatars without altering other scenes or characters. Utilizing technologies such as video processing, character detection/segmentation/tracking, pose analysis, model extraction, and animation rendering, the framework allows protagonists in dynamic videos to easily transcend the boundaries between reality and virtuality, transforming into fun 3D character models with a single click.

Official project homepage:https://aigc3d.github.io/motionshop/

Online experience access:https://www.modelscope.cn/studios/Damo_XR_Lab/motionshop/summary

Motionshop Features

  • Video characters replaced with 3D avatarsUsers simply need to upload a video, and AI can intelligently identify the main characters in the video and seamlessly convert them into vivid 3D character models.
  • Synchronization of motion and realismMotionshop not only replaces characters, but also accurately replicates the movement details of characters in the original video, ensuring smooth and natural movements of 3D characters and providing highly realistic visual effects.
  • Seamless integration of reality and virtualityMotionshop allows real-world characters to be seamlessly integrated with 3D virtual characters, creating a brand-new experience that transcends the boundaries between reality and virtuality, adding endless possibilities to video content.

How Motionshop works

The Motionshop framework consists of two parts: a video processing workflow for extracting and repairing the background video sequence, and a pose estimation and rendering workflow for generating the 3D virtual character video sequence. By running these two workflows in parallel and using the high-performance ray tracing renderer TIDE, the entire video replacement process can be completed in minutes.

The specific technologies are as follows:

  1. Character detection: First, Motionshop uses advanced algorithms to accurately identify characters in the video, ensuring the accuracy of subsequent processing.
  2. Video object segmentation and tracking: Based on character detection, the system further performs pixel-level segmentation and tracking of video objects to prepare for subsequent repair and replacement.
  3. Video Restoration: Employing advanced video restoration technology, this tool repairs flaws and imperfections in background video sequences.
  4. Pose estimation: By applying high-precision pose estimation methods such as CVFFS, the system can stably capture the character's motion poses, laying the foundation for animation retargeting.
  5. Animation mapping: After estimating the character's poses, Motionshop remaps these poses onto the selected 3D model, allowing the new model to naturally simulate the original character's movements.
  6. Lighting and shadow estimation: To ensure that the lighting and shadow effects of the new model are consistent with those of the original video, the system will perform fine lighting and shadow estimation to achieve seamless visual fusion.
  7. 3D Rendering: Using the TIDE rendering engine, Motionshop renders 3D models into realistic image sequences, while combining algorithms such as motion blur and temporal anti-aliasing to improve rendering quality.
  8. Video compositing: Finally, the newly generated 3D rendered image is composited with the repaired video background to create a brand new video sequence that seamlessly integrates with the original video.

How to use Motionshop

  1. Access the ModelScope demo experience from Motionshop here:https://www.modelscope.cn/studios/Damo_XR_Lab/motionshop/summary
  2. Prepare a video, ensuring that the main subject is intact and the video length does not exceed 15 seconds.
  3. After uploading the video, the system will automatically select the people in the video.
  4. Select the virtual character model you want to use for replacement.
  5. Finally, click the "Generate Video" button and wait a moment for the result to be generated.
Back