AB
AB
AiBoss

All Categories

project

MultiTalk - An audio-driven framework for generating multi-person dialogue videos

MultiTalk is a novel audio-driven multi-person dialogue video generation framework jointly developed by Sun Yat-sen University Shenzhen Campus, Meituan, and Hong Kong University of Science and Technology. The framework generates videos containing human dialogue based on multi-channel audio input, reference images, and text prompts...

What is MultiTalk?

MultiTalk is a novel audio-driven multi-person dialogue video generation framework jointly developed by Sun Yat-sen University Shenzhen Campus, Meituan, and Hong Kong University of Science and Technology. The framework generates videos featuring interactive characters whose lip movements match the audio, based on multi-channel audio input, reference images, and text prompts. It introduces the Label Rotary Position Embedding (L-RoPE) method to effectively address the issue of binding multi-channel audio to characters, and retains the basic model's instruction-following ability through partial parameter training and multi-task training strategies. MultiTalk performs exceptionally well on multiple datasets, demonstrating powerful video generation capabilities and is suitable for generating various scenarios such as cartoons, singing, and instruction-following videos.

MultiTalk's main functions

  • Audio-driven multi-person dialogue video generationGenerates a video featuring multi-person interaction and lip-syncing based on multi-channel audio input, reference images, and text prompts.
  • Solving the problem of audio and character bindingThe jiyu Label Rotary Position Embedding (L-RoPE) method effectively solves the problem of binding multi-channel audio to characters, avoiding the incorrect binding of audio to the wrong character.
  • Powerful command following capabilityBased on partial parameter training and multi-task training strategies, it retains the instruction-following ability of the basic model and generates video content that meets the requirements according to text prompts.

MultiTalk's technical principles

  • Audio-driven video generation frameworkMultiTalk uses a video diffusion model based on Diffusion-in-Transformer (DiT) as its infrastructure. This architecture combines a 3D Variational Autoencoder (VAE) to compress the spatiotemporal dimensions of the video, efficiently generating video content. Based on the inverse process of the diffusion model, video content is gradually reconstructed from noise. The 3D VAE is used for video encoding and decoding, compressing video frames into a low-dimensional space for easier model processing.
  • Audio feature extractionTo combine audio features with video content, MultiTalk introduces an audio cross-attention mechanism. Audio features are extracted using Wav2Vec, compressed temporally, and matched to the video's frame rate. An audio cross-attention layer is added to each DiT block, allowing the video generation process to dynamically adjust the video content based on the audio features.
  • Label Rotary Position Embedding (L-RoPE)Assign different tag ranges to each person and background in the video. Based on rotational position embedding technology, the tag information is integrated into the features of the audio and video to ensure the correct binding of audio to people.
  • Adaptive character positioningTo dynamically track the position of each person in the video, an adaptive person localization method is used. Based on the reference image and the self-attention map of the generated video, the model automatically identifies the position of each person, achieving accurate audio binding.
  • Training strategyThe first stage focuses on single-person animation, while the second stage handles multi-person animation. During partial parameter training, only the parameters of the audio cross-attention layer and the audio adapter are updated, while other network parameters are frozen to preserve the basic model's instruction-following ability.
  • Multi-task trainingBy combining audio-to-image to-video (AI2V) and image-to-video (I2V) tasks and training with different datasets, the overall performance of the model can be improved.

MultiTalk's project address

MultiTalk Application Scenarios

  • Film and EntertainmentIt can be used in animated films, special effects production, game cutscenes, etc., to generate multi-person dialogue and interactive scenes, improve production efficiency and visual effects, and enhance the audience's immersion.
  • Education and TrainingIn fields such as online education, virtual classrooms, and language learning, interactive teaching videos are created to simulate real dialogue and communication scenarios, thereby improving learning effectiveness and engagement.
  • Advertising and MarketingGenerate product demonstration videos, virtual customer service interactive videos, etc., to enhance advertising appeal, improve customer service efficiency and quality, and promote product promotion.
  • Social media and content creationCreate creative multi-person dialogue videos, virtual live streams, etc., to attract user attention and sharing, enhance the fun and interactivity of the content, and increase user stickiness.
  • Intelligent servicesIt can be applied in fields such as intelligent customer service and virtual assistants to generate natural and smooth interactive videos, provide a more humanized service experience, and improve user satisfaction.
Back