PersonaTalk - A high-fidelity and personalized visual dubbing framework launched by ByteDance.

What is PersonaTalk?

PersonaTalk, developed by ByteDance, is a two-stage attention-based framework used to achieve high-fidelity and personalized visual voice-overs. PersonaTalk can synthesize videos with precise lip-sync to the target audio while preserving the speaker's unique speaking style and facial details. The first stage involves style-aware audio encoding and lip-sync geometry generation, while the second stage uses a dual-attention face renderer to render the texture of the target geometry. PersonaTalk demonstrates superior performance compared to existing technologies (including Wav2Lip, VideoReTalking, DINet, and IP_LAP) in terms of visual quality, lip-sync accuracy, and personality preservation. As a general-purpose framework, it achieves results comparable to person-specific methods.

PersonaTalk's main functions

• Lip synchronizationEnsure that the lip movements of the person in the video accurately match the input audio.
• Preservation of individualityDuring the video synthesis process, the speaker's unique style and facial features are preserved.
• Style perceptionBased on the analysis of the speaker's 3D facial geometry, the speaker's speaking style is learned and incorporated into the audio features.
• Dual-attention facial renderingWe use two parallel attention mechanisms, Lip-Attention and Face-Attention, to process the texture rendering of the lips and other facial regions respectively, generating facial images with rich details.

PersonaTalk's technical principles

• Geometry Construction:
  • Style-aware audio codingWe use pre-trained models such as HuberT to convert audio signals into rich contextual speech representations, and inject speaking style into audio features based on cross-attention layers.
  • Lip-shaped synchronous geometry generation: Drive the speaker's template geometry with stylized audio features, and generate lip geometry synchronized with the audio based on multiple cross-attention and self-attention layers.
• Facial rendering:
  • Geometry and texture codingThe geometry and texture of the reference video are encoded into the latent space to facilitate subsequent processing.
  • Dual attention texture samplingBased on two parallel cross-attention layers (Lip-Attention and Face-Attention), the textures of the lips and face are sampled from different reference frames.
  • Reference frame selection strategySelect different reference frames for lip and facial textures to enhance the diversity and global consistency of texture sampling.
  • Texture DecodingThe sampled texture is decoded from latent space back to pixel space, preserving the facial geometry and generating the final facial image.

PersonaTalk project address

• Project official website:grisoon.github.io/PersonaTalk
• arXiv technical paper:https://arxiv.org/pdf/2409.05379

Application scenarios of PersonaTalk

• Film and video productionIn film post-production, PersonaTalk provides voice acting for characters, especially when the original recordings are unsatisfactory or language changes are needed, generating voice-over videos synchronized with the character's lip movements.
• video gamesIn game development, it is used to generate realistic dialogues for non-player characters (NPCs), providing a more immersive gaming experience.
• Virtual assistants and digital humansIt provides more natural and realistic voice and facial expression synchronization for virtual assistants or digital humans, enhancing the user interaction experience.
• Language learning applicationsIn language learning software, P generates lip-synced videos of teachers or virtual characters to help learners better learn and imitate pronunciation.
• News and media broadcastingUsed to translate news anchors' speeches into different languages while preserving their original facial expressions and lip movements, thereby improving the naturalness and accuracy of multilingual broadcasts.