SongCreator - an AI music model that can understand, generate, and edit songs.

What is SongCreator?

SongCreator is an AI music generation model developed by Tsinghua University Shenzhen International Graduate School, the Chinese University of Hong Kong, and other institutions. It can generate complete songs including vocals and accompaniment from lyrics. Based on a dual-sequence language model (DSLM) and an attention masking strategy, it understands and generates various related song generation tasks, including editing and generation. SongCreator performs exceptionally well across multiple tasks, particularly in lyrics-to-song and lyrics-to-vocal tasks, and can independently control the acoustic conditions of the vocals and accompaniment in the generated songs.

SongCreator's main functions

• Lyrics to SongGenerate a complete song including vocals and accompaniment based on the provided lyrics.
• Lyrics to VocalsThe vocal parts are generated based on the lyrics, but do not include accompaniment.
• Accompaniment to SongGiven an accompaniment, generate vocal parts that coordinate with the accompaniment to form a complete song.
• Vocals-to-SongGiven the vocal parts, generate the accompaniment to form a complete song.
• Song EditingModify specific sections of an existing song to match target lyrics while maintaining consistency with the original song.
• Vocal EditingEdit the vocal parts of a song without changing the accompaniment.
• Music ContinuationGenerate continuous parts of music based on an accompaniment or vocal piece.
• Unconditional music generationGenerate music or vocals without lyrics.

SongCreator's technical principles

• Bisequence Language Model (DSLM)The model uses DSLM to capture information from vocals and accompaniment. It includes two decoders, one for vocals and one for accompaniment, and captures the interaction between them through a dynamic bidirectional cross-attention module.
• Attention masking strategyWe designed a series of attention masking strategies. These strategies enable the model to work in a consistent manner across different song generation tasks, such as editing, understanding, and generation.
• Multi-task trainingBased on multi-task training, it enhances its ability in composition, arrangement and understanding, enabling it to handle various complex musical scenarios.
• condition signalIt accepts a variety of optional inputs, including lyrics, vocal prompts, and accompaniment prompts, giving it a high degree of flexibility and control when generating songs.
• Semantic TokensThe BEST-RQ model was trained on an unlabeled dataset to extract semantic tokens from songs based on vector quantization. These tokens contain the semantic and acoustic details needed to reconstruct the songs.
• Latent Diffusion Model (LDM)The LDM is used to decode semantic tokens into high-quality song audio. The model consists of a variational autoencoder (VAE) and a diffusion model, which is used to generate high-fidelity and musical music.

SongCreator project address

• Project official website:songcreator.github.io
• arXiv technical paper:https://arxiv.org/pdf/2409.06029

Application scenarios of SongCreator

• Music ProductionMusic producers and composers generate demos of songs, quickly prototype them, or find inspiration during the creative process.
• Education and LearningIn music education, it serves as a teaching tool to help students understand song structure and learn composition and arrangement.
• Entertainment and GamesIn video games and interactive media, background music can be generated in real time based on the game context to enhance player immersion.
• Content creationVideo content creators and podcasters can create original music for their projects without hiring professional musicians.
• Advertising and MarketingThe advertising industry can quickly generate background music that aligns with brand image and advertising concepts.