HiFiVFS - A high-fidelity video face-swapping framework jointly launched by Tencent and VIVO

What is HiFiVFS?

HiFiVFS (High Fidelity Video Face Swapping) is a high-fidelity video face-swapping framework launched by Tencent and VIVO. Based on the Stable Video Diffusion (SVD) framework, HiFiVFS uses multi-frame input and temporal attention mechanisms to ensure the stability of video generation. During the training phase, the framework introduces Fine-Grained Attribute Learning (FAL) and Detailed Identity Learning (DIL) to enhance attribute control and identity similarity. FAL achieves attribute decoupling based on identity desensitization and adversarial learning, while DIL uses identity features more suitable for face swapping to improve similarity. HiFiVFS is trained and tested in the latent space; to demonstrate the results, the entire process is visualized in the original image space.

Main functions of HiFiVFS

• High-fidelity video face swappingThis function replaces facial features in a source image with those in a target video, while preserving the target video's attributes, such as pose, expression, lighting, and background.
• Timing stabilityApplying a temporal attention mechanism to multi-frame video ensures continuity and stability between video frames, avoiding the timing jitter problem that may occur in traditional methods.
• Fine-grained attribute controlBased on fine-grained attribute learning (FAL), it extracts and controls fine-grained attributes in videos, such as lighting and makeup, which are often difficult to maintain in previous face-swapping technologies.
• Enhanced identity similarityBased on Detailed Identity Learning (DIL), we improve the similarity between the face after face swapping and the identity of the source image by using more detailed facial recognition features.

HiFiVFS Technical Principles

• Based on SVD frameworkBuilt on the Stable Video Diffusion (SVD) framework, which is designed for high-resolution text-to-video and image-to-video compositing.
• Multi-frame inputUnlike methods that only process single-frame images, HiFiVFS processes multi-frame video input, which helps maintain the temporal stability of the video.
• Time attention mechanismThis technology enhances the correlation between video frames based on a time attention mechanism, thereby further improving video stability.
• Fine-grained attribute learning (FAL):
  • Attribute decouplingBased on identity desensitization and adversarial learning, FAL can extract fine-grained attribute features that are decoupled from identity.
  • Enhanced attribute controlFAL enhances the control over attributes based on adversarial learning, allowing the face-swapped video to better retain the attributes of the target video.
• Detailed Identity Learning (DIL):
  • Identity feature extractionDIL uses deeper features from facial recognition models to obtain more detailed facial identity information.
  • Identity similarity enhancementDIL improves the similarity between the face-swapping result and the original identity by injecting these detailed identity features into the face-swapping process.

HiFiVFS project address

• Project official website:cxcx1996.github.io/HiFiVFS
• arXiv technical paper:https://arxiv.org/pdf/2411.18293v1

Application scenarios of HiFiVFS

• Film and video productionIn film and video production, this involves replacing or altering an actor's facial expressions and identity to suit specific plot requirements or for special effects production.
• Game developmentIn game development, it is used to create realistic character facial animations, providing a richer and more authentic interactive experience.
• Virtual Reality (VR) and Augmented Reality (AR)In VR and AR applications, it is used to generate or modify the facial features of a user's virtual avatar, providing a personalized and immersive experience.
• social mediaOn social media platforms, users can perform entertaining face swaps, such as face-swapping filters or facial expression changes, to increase interactive fun.
• Advertising and MarketingAdvertisers use the facial features of celebrities or models in their advertisements to enhance their appeal and personalization.