DanceFusion - Tsinghua University launches an open-source framework for audio-driven dance motion reconstruction and generation.

What is DanceFusion?

DanceFusion is an open-source framework developed by Tsinghua University, focusing on audio-driven dance motion reconstruction and generation. Combining a hierarchical spatiotemporal Transformer-VAE and a diffusion model, DanceFusion can handle incomplete and noisy skeletal data from social media, generating realistic dance movements synchronized with music. Based on advanced masking strategies and iterative diffusion processes, DanceFusion optimizes motion sequences, ensuring high fidelity in motion generation and music synchronization, and is widely used in content creation, virtual reality, and interactive entertainment.

DanceFusion's main functions

• Audio-driven dance motion reconstruction and generationDanceFusion generates dance moves synchronized with music, suitable for incomplete and noisy skeletal data extracted from social media platforms such as TikTok.
• Processing incomplete and noisy dataThe framework can effectively handle joint loss, occlusion and noise issues, and accurately captures the spatial and temporal information of skeletal sequences based on hierarchical spatiotemporal VAE.
• Audio and motion synchronizationBased on the diffusion model, DanceFusion ensures that dance moves perfectly match the rhythm, melody, and emotion of the music.
• Advanced masking technology: Develop a masking strategy to handle incomplete skeletal data and ensure that the model only considers reliable joint data during reconstruction.
• Generate high-quality dance movesThe framework can generate high-quality, realistic dance movement sequences with a high degree of diversity and style.

The technical principles of DanceFusion

• Layered Spatiotemporal VAE Encoding:
  • Spatial codingTreat each skeletal joint as a token to capture the spatial relationships between joints in the same frame.
  • Time codingLearn the temporal dependencies between frames to ensure the temporal continuity and smoothness of action sequences.
• diffusion modelStarting with a noisy skeletal sequence, we iteratively optimized it step by step to improve the realism of the movements and their synchronization with the audio.
• Masking mechanismApply a masking mechanism during the encoding phase to mark the presence or absence of joints, preventing the model from considering missing joints.
• Audio feature fusion: Incorporate audio features during the iteration process to achieve precise synchronization between the generated actions and the music.
• Experimental evaluationThe quality of the generated dance sequences was evaluated using FID and diversity scores to ensure the diversity and non-repetitiveness of the output.

DanceFusion's project address

• Project official website:th-mlab.github.io/DanceFusion
• arXiv technical paper:https://arxiv.org/pdf/2411.04646

Application scenarios of DanceFusion

• Content creationGenerate dance videos synchronized with music for use on social media and in video production.
• Virtual Reality (VR) and Augmented Reality (AR)Provides natural dance movements for virtual characters, enhancing immersion.
• Interactive entertainment and gamesGenerate realistic dance moves in the game to enhance the gaming experience.
• Dance education and training: As a teaching demonstration, it helps learners learn and master dance skills.
• Animation and film productionGenerate complex dance moves for virtual characters, reducing the need for motion capture.