AB
AB
AiBoss

All Categories

project

YuE - An open-source AI music generation model jointly launched by HKUST and M-A-P

YuE is an open-source AI music generation model jointly developed by the Hong Kong University of Science and Technology and the Multimodal Art Projection team. It can convert lyrics into complete songs and supports various music styles, including pop, metal, jazz, etc.

What is YuE?

YuE is an open-source AI music generation model jointly developed by the Hong Kong University of Science and Technology and the Multimodal Art Projection team. It can convert lyrics into complete songs, supports multiple music styles including pop, metal, jazz, and hip-hop, and supports multiple languages such as English, Chinese, Japanese, and Korean. YuE addresses challenges in music generation, such as long contexts, complex musical signals, and language distortion, through a semantically enhanced audio segmenter, dual-segmentation technology, chain-like lyric generation, and a three-stage training scheme. The generated songs have a coherent musical structure and appealing vocal melodies. The model is completely open-source, allowing users to freely use and modify the code. It provides flexible generation options, allowing users to adjust details such as the style and vocal type of the generated songs through simple command-line parameters.

YuE's main functions

  • Lyrics to songIt can convert input lyrics into a complete song, including vocals and accompaniment.
  • Supports multiple music stylesSupports various music styles including pop, metal, jazz, and hip-hop.
  • High-quality generationThrough various technical optimizations, we ensure that the generated songs are coherent and of high quality.
  • Multilingual supportSupports multiple languages including English, Chinese, Japanese, and Korean.
  • Long-term music creationIt can generate a complete song that is up to 5 minutes long.
  • Open source and customizabilityThe code and model are completely open source, and users are free to use and modify them.
  • Emotion and style matchingIt can generate a matching music style based on the emotional tone of the lyrics.
  • Cross-modal applicationsIt can be combined with other multimodal models for multimedia art creation.

YuE's technical principles

  • Semantic Enhancement Audio SegmenterYuE uses a semantically enhanced audio segmenter to reduce training costs and accelerate convergence. It can better understand the semantic information of lyrics and combine it with musical signals to generate music that better matches the lyrics.
  • Double word segmentation technologyYuE proposed a dual-segmentation technique for achieving synchronized vocal-instrument modeling without modifying the LLaMa decoder-only architecture. The model can simultaneously generate suitable accompaniment while generating the lead vocals, ensuring rhythmic and melodic harmony between the two.
  • Lyrics chain thinking generationYuE introduces a lyric chain-based thought generation technique, enabling the model to progressively generate the entire song while adhering to lyric conditions. This allows the model to better handle long contextual information, ensuring that the generated song maintains overall structural coherence.
  • Three-stage training programYuE employs a three-stage training program to ensure better scalability, musicality, and lyrical controllability. Specifically:
    • Phase 1: Training the basic model to learn the fundamental patterns of music generation.
    • Phase TwoStyle and emotion alignment, through a large number of samples, allows the model to generate music with specific styles and emotions.
    • Phase ThreePreference correction, through techniques such as reinforcement learning, further optimizes the generated results to ensure that the generated music is more in line with human aesthetic standards.

YuE's project address

Application scenarios of YuE

  • Music compositionMusic creators can use YuE to quickly generate melodies and accompaniments, sparking creative inspiration. Based on lyrics provided by the user, it generates personalized songs that match their emotional and stylistic needs.
  • Film and Video ProductionGenerate background music for movies, TV series, and short videos, matching the emotions and atmosphere of different scenes. Create catchy ringtones for advertisements to effectively convey brand messages.
  • Game musicGenerate background music that matches the game scene to enhance the game's immersion.
  • Social media contentGenerate unique background music for short videos on platforms such as TikTok and Instagram.
Back