Freestyler - A rap music generation model developed by Northwestern Polytechnical University in collaboration with Microsoft and the University of Hong Kong.

What is a Freestyler?

Freestyler is a rap music generation model jointly developed by the Audio, Speech and Language Processing Group (ASLP@NPU) of the School of Computer Science at Northwestern Polytechnical University, Microsoft, and the Big Data Institute of the Chinese University of Hong Kong, Shenzhen Research Institute. It can directly create rap music based on lyrics and accompaniment. Freestyler generates semantic tags based on a language model, then produces a spectrogram based on a conditional flow matching model, and finally converts it into audio using a neural vocoder. Freestyler offers the RapBank dataset to support training and model development, enabling zero-shot timbre control and allowing users to generate rap vocals with specific timbres.

Freestyler's main functions

• Rap vocal generationIt can generate rap vocals directly from lyrics and accompaniment input, without requiring users to have musical knowledge.
• Zero-sample tone controlBased on a 3-second reference audio, Freestyler can adapt to the timbre of any speaker, achieving zero-sample timbre control.
• Dataset creationTo address the scarcity of rap data, the team created the RapBank dataset and provided a data processing workflow.
• Style and rhythm alignmentThe generated rap vocals are highly aligned with the style and rhythm of the accompaniment, encompassing both style and rhythm.
• High-quality audio outputBased on advanced vocoder technology, it generates natural and high-quality rap audio.

The technical principles of Freestyler

• Lyrics-to-Semantic:
  • Use language models (such as LLaMA) to predict discrete semantic tags based on lyrics and accompaniment features.
  • Features are extracted using self-supervised learning (SSL) models such as Wav2Vec XLS-R, and semantic tags are obtained using K-means clustering.
• Semantic-to-Spectrogram:
  • Conditional flow matching (CFM) is used to convert discrete semantic tags into continuous mel spectrograms.
  • Reference audio was incorporated into the CFM model to supplement the missing timbre information in the semantic tags.
• Spectrogram to Audio:
  • The audio is reconstructed from the spectrogram using a pre-trained vocoder (such as BigVGAN-V2).
  • Vocoders can process various types of audio data, including speech, singing, and ambient sounds in different languages.
• Datasets and Processing Flow:
  • The RapBank dataset contains a large number of rap songs collected from the Internet, processed through steps such as automatic crawling, source separation, segmentation, lyric recognition, and quality filtering.
  • Dataset processing includes music source separation, speech activity detection (VAD) segmentation, automatic speech recognition (ASR) lyrics recognition, and calculation of quality-related metrics.
• Zero-sample controlThe reference encoder extracts the global speaker embedding from the reference audio, which is used to control the timbre of the generated sound.

Freestyler's project address

• GitHub repository:https://github.com/NZqian/RapBank
• arXiv technical paper:https://arxiv.org/pdf/2408.15474

Application scenarios of Freestyler

• Music compositionIt provides inspiration for music producers and composers, enabling them to quickly generate rap lyrics and vocals and create new musical works.
• Live performanceDuring concerts or DJ performances, rap vocals are generated in real time to accompany the music, providing a dynamic musical experience for the audience.
• Game sound effectsIn video games, generating rap sound effects for characters enhances the game's immersion and interactivity.
• Education and TrainingIn music education, we help students learn rapping skills and music composition by generating demo audio to improve learning outcomes.
• Social media content creationContent creators generate unique rap audio to enrich their video or audio content on social media platforms and attract more viewers.