AB
AB
AiBoss

All Categories

project

LivePortrait - An open-source portrait animation generation framework launched by Kuaishou

LivePortrait is an open-source portrait animation generation framework launched by Kuaishou. It focuses on efficiently and controllably transferring the expressions and poses that drive videos to static or dynamic portraits, creating expressive videos. This technology uses implicit...

What is LivePortrait?

LivePortrait is an open-source portrait animation generation framework launched by Kuaishou. It focuses on efficiently and controllably transferring the expressions and poses that drive videos to static or dynamic portraits, creating expressive videos. This technology is implemented through an implicit keypoint framework, utilizing large-scale, high-quality training data and hybrid training strategies to improve the model's generalization ability and motion control accuracy. LivePortrait achieves extremely fast single-frame generation speed on an RTX 4090 GPU, reaching 12.8 milliseconds, and has further optimization potential. The open-source community has responded enthusiastically, and LivePortrait's GitHub page provides detailed usage guides and resources.

LivePortrait's main functions

  • Facial expression and posture transferLivePortrait can transfer the expressions and poses in a driving video to a static or dynamic portrait in real time, generating videos with rich expressions.
  • High efficiencyThe framework achieves a single-frame generation speed of up to 12.8 milliseconds on an RTX 4090 GPU, demonstrating extremely high processing speed.
  • Generalization abilityThrough a hybrid video-image training strategy and large-scale, high-quality training data, LivePortrait exhibits good generalization ability and can adapt to portraits of people with different styles and identities.
  • ControllabilityBy leveraging implicit keypoints and a lightweight MLP network, LivePortrait enhances control over the animation generation process.
  • Multiple styles supportedLivePortrait can handle portraits of various styles, including realistic and stylized (such as anime) portraits.
  • High-resolution animation generationLivePortrait supports generating high-resolution animations, providing a clearer visual experience.
  • Attach and redirect modulesThe system incorporates a fitting module and eye and mouth repositioning modules to adapt to complex scenarios such as cropping and group photos, and to avoid pixel misalignment.

The technical principles of LivePortrait

  • Basic model trainingIn the first stage, the main focus is on optimizing the appearance extractor, motion extractor, twisting module, and decoder. The models are all trained from scratch.
  • Training of splicing and redirection modulesAfter training the base model in the first stage, the appearance extractor, motion extractor, warp module, and decoder are frozen and remain unchanged in the second stage. In the second stage, only the stitching module and the repositioning module are optimized.
  • Video-image fusion trainingLivePortrait treats each image as a video frame and trains the model on both video and images simultaneously, improving the model's generalization ability.
  • Upgraded network architectureLivePortrait unifies the canonical implicit keypoint estimation network, head pose estimation network, and facial deformity estimation network into a single model, and uses ConvNeXt-V2-Tiny as its structure to directly estimate the canonical implicit keypoints, head pose, and facial deformities of the input image.
  • Implicit keypoint optimization guided by keypoint: Introduce 2D keypoints to capture micro-expressions, and use the loss guided by keypoints as a guide for implicit keypoint optimization.
  • Cascaded loss functionWe employ implicit keypoint invariant loss, keypoint prior loss, head pose loss, and deformation prior loss from face vid2vid, and apply perceptual and GAN losses to improve texture quality.

LivePortrait project address

Application scenarios of LivePortrait

  • Social media content creationUsers can transform their photos or videos into dynamic content for use on social media platforms, increasing interactivity and engagement.
  • Virtual anchors and live streamingLivePortrait technology allows users to create virtual avatars for live streaming or video production without needing to appear on camera, making it suitable for scenarios such as game streaming and educational presentations.
  • Film and animation productionIn film and television post-production, LivePortrait can be used for character facial expression capture and animation generation, improving production efficiency and reducing costs.
  • Advertising and MarketingBusinesses can use LivePortrait technology to create engaging advertising videos with dynamic character designs to attract the attention of potential customers.
Back