project
INFP - An audio-driven AI framework for generating realistic facial expressions and head poses.
INFP is an audio-driven head generation framework designed specifically for two-person dialogue interaction. It automatically switches roles guided by the dialogue audio, eliminating the need for manual role assignment and switching. INFP consists of two phases: action-based head generation...
What is INFP?
INFP is an audio-driven head generation framework designed specifically for two-person dialogue interaction. It automatically switches roles guided by dialogue audio, eliminating the need for manual role assignment and switching. INFP comprises two phases: a motion-based head imitation phase and an audio-guided motion generation phase. Experiments and visualizations validate the superior performance and effectiveness of the INFP method. INFP also proposes the large-scale two-person dialogue dataset DyConv to support advancements in this research area.
Main functions of INFP
- Automatic role switchingIn two-person conversations, INFP can automatically switch roles without the need for manual role assignment and switching, enhancing the naturalness and smoothness of the interaction.
- Lightweight and efficientWhile maintaining powerful functionality, INFP is also lightweight. Achieving inference speeds exceeding 40 fps on the Nvidia Tesla A10 means INFP can support real-time intelligent agent interaction, whether it's communication between agents or interaction between humans and agents.
- Interactive header generationINFP comprises two key stages: motion-based head mimicry and audio-guided motion generation. The first stage encodes facial communication behaviors from real-world dialogue videos into a low-dimensional motion latent space, while the second stage maps the input audio to these motion latent codes, enabling audio-driven head generation.
- DyConv, a large-scale two-person dialogue datasetTo support advancements in this research area, INFP proposed the large-scale two-person dialogue dataset DyConv, which collects a wealth of binary dialogues from the Internet.
INFP Technical Principles
- Head imitation stage based on movementAt this stage, frame learning projects facial communication behaviors from real-life conversation videos into a low-dimensional motion latent space. This process involves extracting facial communication behaviors from a large number of real-world conversation videos and encoding them into motion latent codes that can drive animations of static images.
- Audio-guided motion generation stageIn the second stage, the framework learns a mapping from the input dual-channel audio to the latent motion code. This stage is achieved through a denoising process, enabling audio-driven head generation in interactive scenarios.
- Real-time interaction and style controlINFP supports real-time interaction, allowing users to interrupt or respond to the virtual avatar at any time during the conversation. By extracting style vectors from arbitrary portrait videos, INFP can also globally control the emotions or attitudes in the generated results.
INFP's project address
- Project official website:https://grisoon.github.io/INFP/
- arXiv technical paper:https://www.arxiv.org/pdf/2412.04037
Application scenarios of INFP
- Video conferencing and virtual assistantsThe INFP framework enables realism, interactivity, and real-time performance, making it suitable for real-time scenarios such as video conferencing and virtual assistants, providing a more natural and smooth interactive experience.
- Social media and interactive entertainmentINFP can be used to generate interactive avatars with natural facial expressions and head movements on social media platforms or interactive entertainment applications, enhancing the user's interactive experience.
- Education and TrainingINFP can be used to create virtual teachers or trainers, providing a more vivid and interactive teaching experience.
- Customer ServiceIn the field of customer service, INFP can be used to generate virtual customer service representatives to provide more personalized services.
- Advertising and MarketingINFP can be used to generate more engaging virtual spokespeople for advertising and marketing campaigns, providing a more realistic and interactive advertising experience.
- Games and SimulationsIn gaming and simulation environments, INFP can be used to create more realistic and interactive characters, enhancing the immersion and interactivity of games.