AB
AB
AiBoss

All Categories

project

NotaGen - A music generation model jointly developed by the Central Conservatory of Music, Beihang University, Tsinghua University, and others.

NotaGen is a music generation model developed by institutions such as the Central Conservatory of Music, Beijing University of Aeronautics and Astronautics, and Tsinghua University. It generates high-quality classical music scores based on a training paradigm that mimics Large Language Models (LLM). NotaGen is based on...

What is NotaGen?

NotaGen is a music generation model developed by institutions such as the Central Conservatory of Music, Beijing University of Aeronautics and Astronautics, and Tsinghua University. It generates high-quality classical music scores based on a training paradigm that mimics Large Language Models (LLMs). NotaGen employs a combination of pre-training, fine-tuning, and reinforcement learning. The pre-training phase uses over 1.6 million pieces of music, while the fine-tuning phase uses approximately 9,000 high-quality classical works, generating scores conditionally based on "period-composer-instrument" cues. In the reinforcement learning phase, NotaGen introduces the CLaMP-DPO method, providing feedback to the generated music in contrast to the CLaMP² model, optimizing musicality and controllability without requiring manual annotation or predefined rewards. NotaGen outperforms baseline models in subjective A/B tests, approaching the musical beauty of human works and significantly improving the artistry of symbolic music generation.

NotaGen's main functions

  • Generate high-quality classical music scoresGenerates scores that conform to a specific style based on the period (such as Baroque, Classical, Romantic), composer (such as Bach, Mozart, Chopin), and instrument (such as keyboard, string quartet, orchestra).
  • Enhance musicalityIt generates highly musical works (such as beautiful melodies, smooth harmonies, and coherent structures) by combining pre-training, fine-tuning, and reinforcement learning.
  • Controllable generationUsers can control the style and characteristics of the generated music by using conditions such as "period - composer - instrument".
  • Supports multiple model sizesIt provides models with different parameter sizes (such as NotaGen-small, NotaGen-medium, and NotaGen-large) to adapt to different hardware and application scenarios.

NotaGen's technical principles

  • Pre-trainingIt is pre-trained on over 1.6 million musical pieces, covering a wide range of styles and instruments. It learns the basic structure and patterns of music based on massive amounts of data, similar to the "language understanding" stage in language models. It is trained using symbolic musical representations (such as ABC symbols) for encoding and predicting the next symbol.
  • Fine-tuningFine-tuning was performed on approximately 9,000 high-quality classical works, optimizing for specific styles (such as Baroque, Classical, and Romantic) and composers to generate music that fits those styles. Conditional generation was enhanced by using "period-composer-instrument" prompts to improve the model's understanding and generation capabilities for specific styles.
  • reinforcement learningBased on the CLaMP-DPO (Contrastive Language-Music Pretraining with Direct Preference Optimization) method, the contrastive learning model CLaMP² provides feedback to optimize the quality and consistency of generated music, without the need for manual annotation or predefined rewards. Generated music samples are categorized into "selection" and "rejection" based on CLaMP² scores. The DPO algorithm is then used to optimize the model, improving the musicality and controllability of the generated music.
  • Symbolic music representationMusical scores are represented using ABC symbols, simplifying the encoding and generation process and supporting the generation of multi-part music. Data is optimized by removing whole rests and adding measure indexes, improving generation efficiency and quality.
  • Model ArchitectureA hierarchical Transformer decoder, including a patch-level decoder and a character-level decoder, handles music fragments and symbol generation respectively. Combined with multimodal feedback provided by CLaMP², it optimizes the global features of the generated music (such as style, harmony, and structure).

NotaGen's project address

NotaGen application scenarios

  • Music creation assistanceIt provides inspiration for composers, quickly generating musical score drafts that fit a specific style, saving creation time.
  • Music EducationIt helps students understand the stylistic characteristics of different musical periods and generates musical scores for teaching.
  • Music ProductionIt can quickly generate background music or scores for movies, games, etc., improving production efficiency.
  • Music performanceGenerate high-quality sheet music for orchestras, choirs, and other groups to use in rehearsals or performances.
  • Music EntertainmentBased on interactive demonstrations, users can generate music in real time and experience the joy of creation.
Back