SongGeneration - A large-scale music generation model open-sourced by Tencent AI Lab

What is SongGeneration?

SongGeneration is a large-scale AI music generation model launched by Tencent AI Lab. The model addresses key issues in the field of AI music creation (AIGC), such as sound quality, musicality, and generation speed. Based on an LLM-DiT fusion architecture, SongGeneration significantly improves sound quality and generation speed, outperforming most open-source models in multiple dimensions, with some metrics comparable to commercial closed-source models. SongGeneration supports text control, multi-track synthesis, style following, and other features, satisfying creators' playability needs while possessing stability and scalability for commercial applications. SongGeneration is widely used in short video soundtracks, game sound effects, virtual human performances, commercial advertising, and personal music creation, propelling AI music creation from an "auxiliary tool" to a new stage of "intelligent co-creation."

The main functions of SongGeneration

• Text controlInput keyword text, such as "happy pop", and AI will quickly generate a complete musical work with the corresponding style and mood.
• Style FollowUpload reference audio of 10 seconds or more to generate new full-length songs with the same style, covering multiple genres.
• Multitrack generationAutomatically generates separate vocal and accompaniment tracks, ensuring a high degree of matching between melody, structure, rhythm, and instrumentation.
• Tone FollowBased on the timbre follow-up of the reference audio, it achieves a "timbre clone" level of human voice performance, which is natural and emotional.

SongGeneration's technical principles

• LeLM (Language Model)Mixed tokens represent the combined audio of vocals and accompaniment, used to capture the overall structure and rhythm of a song, ensuring harmony between vocals and accompaniment. Dual-track tokens encode vocals and accompaniment separately, used to generate high-quality audio details. LeLM can predict mixed tokens and dual-track tokens in parallel, avoiding interference between different token types and improving generation quality and efficiency.
• Music codecThe encoder extracts the music audio into a mix token and a dual-track token. The decoder reconstructs the dual-track token into high-fidelity music audio, ensuring the generated song has high-quality audio performance.
• Multi-Preference AlignmentDirect Preference Optimization (DPO) is based on semi-automatic data construction and post-DPO training, handling diverse human preferences and improving the model's performance in musicality, instruction compliance, and harmony between vocals and accompaniment. Multi-dimensional preferences support preference alignment across multiple dimensions, including lyrics alignment, cue consistency, and musicality, enhancing the overall quality of generated songs.
• Three-stage training paradigm:
  • Pre-trainingPre-training on large-scale music data to align different input modalities and mixed labels.
  • Modular Extension TrainingFurther training of the AR decoder, modeling dual-track markers, and improving sound quality and musicality.
  • Multi-Preference Alignment: Based on DPO post-training, optimize the model's performance on multi-dimensional preferences.

SongGeneration's project address

• GitHub repositoryhttps://github.com/tencent-ailab/SongGeneration
• HuggingFace model libraryhttps://huggingface.co/tencent/SongGeneration
• arXiv technical paper: https://arxiv.org/pdf/2506.07520
• Experience the demo onlinehttps://huggingface.co/spaces/tencent/SongGeneration

SongGeneration's core advantages

• Low bitrate music encoding and decodingAchieve high-quality music reconstruction at extremely low bit rates of 25Hz and 0.35kbps, reducing the burden of language modeling.
• Parallel prediction of multi-class tokensBased on the strategy of "hybridization first, dual-track second", it avoids mutual interference of tokens and improves sound quality and musicality.
• Multidimensional human preference alignmentAligning musicality, lyrics alignment, and cue consistency preferences improves model performance and robustness.
• Three-stage training paradigmPre-training, modular extended training, and multi-preference alignment training optimize music generation results.

SongGeneration's performance

• Overall subjective and objective assessmentIn a comprehensive subjective and objective evaluation comparison with three commercial models (Suno v4.5, Sponge Music, Mureka O1) and four open-source models (YuE, DiffRhythm, ACE-Step, SongGen), SongGeneration ranked first among open-source models and among the top commercial models, demonstrating a significant competitive advantage.
• Objective evaluation (third-party open-source model evaluation)In objective evaluation and horizontal comparison, SongGeneration ranked first in three key dimensions: Content Enjoyment (CE), Content Usability (CU), and Production Quality (PQ), and was in a leading position in Production Complexity (PC).
• Subjective reviews (reviews by ordinary users + professional musicians)In subjective evaluation, SongGeneration outperformed many large models, including Suno, in lyric accuracy, demonstrating the model's significant advantage in speech-text alignment and its maturity in handling content generation details.

Application scenarios of SongGeneration

• Music compositionSongGeneration provides musicians and producers with high-quality song drafts, saving them creation time, helping them focus on core creative work, and inspiring creativity.
• Entertainment industryIn the entertainment industry, including film, games, and advertising, SongGeneration quickly generates background music that meets specific needs, enhancing the immersiveness and appeal of works and enriching musical content.
• EducationAs a music education tool, SongGeneration helps students understand basic music knowledge, stimulates creativity, provides example songs to complement online courses, and enhances teaching effectiveness.
• Advertising and MarketingSongGeneration generates theme-appropriate music for advertisements and brands, enhancing ad appeal and brand recognition, and boosting brand marketing.
• Personal entertainmentOrdinary users can use SongGeneration to create personalized songs, express their emotions, share them on social media platforms, and add fun and entertainment.