GCDance - A 3D dance generation framework developed by the University of Surrey and Jiangnan University

What is GCDance?

GCDance (Genre-Controlled 3D Full Body Dance Generation Driven by Music) is a 3D dance generation framework developed by the University of Surrey and Jiangnan University in the UK. It can generate full-body dance sequences that conform to specific styles based on music and text cues. GCDance combines high-level music features extracted from pre-trained music foundation models (such as Wav2CLIP) with hand-designed music features (such as STFT) to achieve multi-granularity music feature fusion. The CLIP model embeds text cues into each time step of dance generation, achieving style-controlled dance generation. GCDance supports generating dances of multiple styles within the same music clip, ensuring a high degree of consistency between dance movements and the rhythm and melody of the music.

GCDance's main functions

• Style-controlled dance generationBased on text prompts specifying a dance style (such as street dance, jazz dance, etc.), generate dance moves that match that style.
• Precisely aligned with the rhythm of the musicThe dance movements are highly synchronized with the rhythm, beat, and melody of the music, ensuring a natural match between the dance and the music.
• Diverse dance generationThe same music clip can be used to generate dances in multiple styles, enriching the expressiveness and diversity of dance.
• Supports partial editingUsers can customize specific parts of the dance (such as hand movements or specific time periods).
• Generate high-quality full-body motionIt covers 52 joints (including finger joints), and the generated dance movements are natural, realistic, and have a high degree of physical plausibility.

GCDance's technical principles

• Diffusion Model FrameworkBased on a classifier-free diffusion model, a dance sequence is generated from noise using a stepwise denoising method.
• Music Feature ExtractionThis approach combines pre-trained music models (such as Wav2CLIP) to extract high-level semantic features with handcrafted music features (such as Short-Time Fourier Transform (STFT)) to capture low-level musical details. This multi-granularity feature fusion method better captures the complex relationship between music and dance.
• Text feature embeddingBased on the CLIP model, textual cues (such as dance style descriptions) are embedded into each time step of dance generation, and a feature adapter is used to align with its music features to achieve style-controlled dance generation.
• Feature Modulation (FiLM)The Feature-wise Linear Modulation (FiLM) layer dynamically adjusts the dance generation process based on text prompts to ensure that the generated dance conforms to the specified style.
• Dual-stream architectureThe model separates body movements and hand movements, and processes them independently using two Transformer networks to generate more detailed and expressive full-body dance movements.
• Editing functionEditing mechanisms based on diffusion models (such as diffusion repair) allow users to impose constraints on specific parts of the dance during the generation process, enabling local editing and customized generation.

GCDance project address

• arXiv technical paper:https://arxiv.org/pdf/2502.18309

GCDance Application Scenarios

• Virtual Reality (VR) and Augmented Reality (AR)Generate virtual character dances to enhance immersion.
• Game development: Dynamically generate dance moves for game characters to enhance interactivity.
• Dance teachingIt assists in choreography and instruction, providing dance examples of different styles.
• Music video productionIt automatically generates dances based on music and provides creative materials.
• Smart FitnessCombine music to create fitness dances, increasing the fun of exercising.