SongGen - A song generation model developed by Shanghai AI Lab, Beihang University, and the Chinese University of Hong Kong.

What is SongGen?

SongGen is a single-stage autoregressive Transformer model developed by the Shanghai AI Lab, Beijing University of Aeronautics and Astronautics, and the Chinese University of Hong Kong for generating songs from text. SongGen takes lyrics and descriptive text (such as instrumentation, style, and emotion) as input and supports two output modes: a mixed mode for directly generating a mixed audio of vocals and accompaniment, and a dual-track mode for separately synthesizing vocals and accompaniment for easier post-editing. Based on innovative audio tokenization strategies and training methods, SongGen significantly improves the naturalness and vocal clarity of generated songs, solving the problem of cumbersome training and inference processes in traditional multi-stage methods. SongGen's open-source nature and high-quality dataset provide a new benchmark for future music generation research.

SongGen's main functions

• Fine-grained controlUsers can control the generated songs based on lyrics and descriptive text (such as instruments, style, emotion, etc.).
• Sound cloningIt supports sound cloning based on a three-second reference audio, enabling the generated song to have the timbre of a specific singer.
• Two generation modesIt offers "Mix Mode" (directly generates a mixed audio of vocals and accompaniment) and "Dual Track Mode" (synthesizes vocals and accompaniment separately for easy post-editing).
• High-quality audio outputBased on optimized audio tokenization and training strategies, songs with high naturalness and clear vocals are generated.

SongGen's technical principles

• Autoregressive Generative FrameworkBased on the autoregressive Transformer decoder, lyrics and descriptive text are encoded as conditional inputs, and the generation of audio tags is guided by a cross-attention mechanism.
• Audio tokenizationX-Codec encodes audio signals into discrete audio tokens, processes multiple code sequences based on codebook delay mode, and supports efficient generation.
• Hybrid mode and dual-track mode:
  • Hybrid modeIt directly generates mixed audio tags and introduces an auxiliary human voice audio tag prediction target (Mixed Pro) to enhance the clarity of human voices.
  • Dual-track modeVocal and accompaniment markers are generated based on parallel or interlaced modes to ensure alignment at the frame level and improve generation quality.
• Conditional input encoding:
  • Lyrics encodingThe VoiceBPE word segmenter converts lyrics into phoneme-level tokens and extracts key pronunciation information based on a small Transformer encoder.
  • Sound encodingThe MERT model extracts the timbre features of reference audio and supports sound cloning.
  • Text description encodingThe FLAN-T5 model encodes descriptive text into feature vectors, providing control over musical style, emotion, and other aspects.
• Training strategy:
  • Multi-stage trainingThis includes modal alignment, referenceless sound support, and high-quality fine-tuning, gradually improving model performance.
  • Course LearningGradually adjust the code loss weights to optimize the model's learning of audio details.
  • Data preprocessingDevelop an automated data preprocessing pipeline to collect audio from multiple data sources, separate vocals and accompaniment, and generate high-quality lyrics and descriptive text datasets.

SongGen's project address

• GitHub repository:https://github.com/LiuZH-19/SongGen
• arXiv technical paper:https://arxiv.org/pdf/2502.13128

SongGen's application scenarios

• Music compositionQuickly generate song prototypes, explore different styles, generate accompaniment for lyrics, and accelerate the creation process.
• Video background musicGenerate background music for short videos, advertisements, and movies, adjust the style according to the content, and enhance the visual effect.
• Educational SupportIt helps students understand music composition, learn language pronunciation through song generation, and stimulates creativity.
• Personalized experienceGenerate customized songs based on user input and use voice cloning to create a "personalized singer" to enhance entertainment.
• Business applicationsGenerate custom music for brands, replacing copyrighted music materials, for use in advertising and promotion.