AB
AB
AiBoss

All Categories

project

TANGO - The University of Tokyo and CyberAgent AI Lab launch a voice-driven full-body gesture video generation framework.

TANGO is an open-source framework jointly developed by the University of Tokyo and CyberAgent AI Lab, focusing on generating full-body gesture videos synchronized with the target's speech. Based on hierarchical audio motion embedding and diffusion interpolation networks, it synchronizes the target speech...

What is TANGO?

TANGO is an open-source framework jointly developed by the University of Tokyo and CyberAgent AI Lab, focusing on generating full-body gesture videos synchronized with target speech. Based on hierarchical audio motion embedding and diffusion interpolation networks, it perfectly matches target speech audio with actions in a reference video library, ensuring high-fidelity, synchronized video production. TANGO's technological breakthroughs significantly reduce the cost of video content production, including news broadcasting, virtual human narration, and virtual YouTube content creation, providing users with an efficient and cost-effective solution.

TANGO's main functions

  • Voice-driven full-body gesture generationGenerate a full-body gesture video synchronized with the target's audio.
  • High-fidelity video production: Ensure that the generated video has high fidelity, with natural movements and accurate matching with the audio content.
  • Cross-modal alignmentBased on layered audio motion embedding technology, it achieves precise alignment between audio signals and video motion.
  • Transition Frame GenerationHigh-quality transition frames are generated using a diffusion interpolation network to ensure the continuity of video motion.
  • Maintaining appearance consistencyMaintain the same appearance of people and backgrounds as the reference video in the generated video to avoid visual inconsistencies.

TANGO's technical principles

  • Layered Audio Motion Embedding (AuMoCLIP)An implicit hierarchical audio-action joint embedding space is used to encode paired audio and action data. Based on contrastive learning, speech audio and action data are mapped to a common latent space, making the matched audio and action closer in space, thus achieving accurate action retrieval.
  • Diffusion interpolation network (ACInterp)The network is based on an existing video generation diffusion model and is used to generate high-quality transition frames. It includes a reference motion module and a homography background stream to maintain visual consistency between the generated video and the reference video. It effectively eliminates blurring and ghosting artifacts common in traditional stream-based interpolation methods.
  • Action graph retrieval methodTANGO uses a learning-based approach, rather than simple audio start features and keyword matching, to retrieve the action path that best matches the target speech audio. This better handles situations where the actions of different speakers are out of sync with the audio start, and addresses the issue of missing specific keywords in the reference video.
  • Graph structureTANGO uses a directed graph structure to represent video frames (nodes) and the valid transitions (edges) between them. Given a target audio, the system extracts temporal features and uses these features to retrieve a subset of the video playback path. When no transition edges exist in the original reference video, ACInterp is used to generate smooth transition frames.

TANGO project address

Application scenarios of TANGO

  • News broadcastGenerate full-body gesture videos synchronized with news releases to enhance the naturalness of news broadcasts and the viewing experience for audiences.
  • Virtual YouTuberCreate full-body motion videos synchronized with voice for virtual YouTubers to enhance fan interaction and engagement.
  • Online EducationWhen creating educational content, TANGO generates full-body gesture videos of teachers, making remote teaching more vivid and effective.
  • Corporate TrainingAdding synchronized gestures to corporate training videos can enhance the appeal of learning materials and improve the efficiency of information delivery.
  • videoconferenceIn video conferencing, gesture-generated videos using TANGO can enhance the communication experience for participants, especially when collaborating remotely.
Back