X-Dancer - ByteDance and other organizations launch music-driven human-shaped dance video generation framework

What is X-Dancer?

X-Dancer is a music-driven human dance video generation framework jointly developed by ByteDance, researchers from UC San Diego, and USC. It supports the generation of diverse and realistic full-body dance videos from single still images. X-Dancer combines an autoregressive transformer (Transformer) and a diffusion model to model 2D human poses, capturing the complex alignment between dance movements and music rhythms based on widely available monocular video data. X-Dancer generates dance pose sequences synchronized with music using multi-part 2D pose representations and confidence-aware quantization methods, transforming these sequences into coherent video frames based on the diffusion model. X-Dancer outperforms existing methods in motion diversity, music alignment, and video quality, adapting to reference images of different body types and styles, and supporting zero-shot generation and fine-tuning for specific choreography.

X-Dancer's main functions

• Generating dance videos from a single still imageUse a human image and a piece of music to generate a full-body dance video synchronized with the music's rhythm.
• Diverse and personalized dance movementsIt supports generating various styles and complex movements, including detailed movements of the head and hands, and supports character animations of different body types and styles.
• Music rhythm alignmentThe dance moves are closely synchronized with the rhythm of the music, capturing the beat and style of the music.
• Zero-sample generation and customizationSupports zero-sample generation and fine-tuning for specific choreography styles to adapt to different dance needs.
• High-quality video compositingThe generated dance videos have high resolution and realistic visuals, while maintaining consistency with the reference images.

X-Dancer's technical principles

• Pose modelingThis method extracts dance movements from monocular videos based on 2D human pose estimation, avoiding the complexity and data limitations of 3D pose estimation.
• Multipart attitude quantizationThe human body is divided into multiple parts (such as the upper body, lower body, head, and hands), and each part is encoded and quantized into a pose token. These tokens are then combined into a complete pose based on a shared decoder.
• Autoregressive TransformerUsing an autoregressive model of the GPT type, future posture marker sequences are predicted based on music features and historical posture information to achieve synchronization with music.
• Diffusion model synthesisThe generated pose markers are converted into spatial guidance signals using a trainable motion decoder, and combined with reference image features, a coherent dance video is generated based on a diffusion model.
• AdaIN and the Spacetime ModuleAdaptive Instance Normalization (AdaIN) and spatiotemporal modules are used to ensure the spatiotemporal coherence and identity consistency of the generated videos.

X-Dancer's project address

• arXiv technical paper:https://arxiv.org/pdf/2502.17414

Application scenarios of X-Dancer

• Social media sharingUsers can transform photos and music into personalized dance videos, increasing the fun and engagement of content on social media platforms.
• Virtual character animationGenerate synchronized dance movements for characters in virtual reality and the metaverse to enhance their expressiveness.
• Music game interactionReal-time generation of dance moves enhances the visual effects and interactivity of music rhythm games.
• AdvertisingCombine brand music to generate dance videos for advertising and promotion, attracting user attention.
• Dance EducationGenerate dance videos in different styles to assist in dance teaching or showcase cultural characteristics.