AB
AB
AiBoss

All Categories

project

ACE-Step - A music generation foundation model jointly developed by ACE Studio and Step Starry Sky.

ACE-Step is an open-source music generation model jointly launched by ACE Studio and StepFun. Based on an innovative architecture, it enables efficient, coherent, and controllable music creation. ACE-Step combines diffusion models, deep compression...

What is ACE-Step?

ACE-Step is an open-source music generation model jointly developed by ACE Studio and StepFun. Based on an innovative architecture, it enables efficient, coherent, and controllable music creation. Combining a diffusion model, a deep compression autoencoder (DCAE), and a lightweight linear transformer, ACE-Step can generate high-quality music in a short time, up to 15 times faster than traditional LLM models. ACE-Step supports various music styles, languages, and controllability features, providing powerful tools for music creation. Suitable for rapid music generation, ACE-Step serves as a foundational model supporting various music creation subtasks, helping musicians, producers, and content creators achieve efficient creation.

Main functions of ACE-Step

  • Rapid synthesisIt can generate high-quality music in a short time; for example, it only takes 20 seconds to generate 4 minutes of music on an A100 GPU.
  • Diverse stylesIt supports the generation of lyrics for various mainstream music styles (such as pop, rock, electronic, jazz, etc.) and multiple languages.
  • Variant generationAdjusting the noise ratio generates different variations, providing a diverse range of music choices.
  • Redraw function: Regenerate specific parts, modify the style, lyrics or vocals, and retain other elements.
  • Lyrics editingIt supports partial lyrics modification of generated music while keeping the melody and accompaniment unchanged.
  • Multilingual supportIt supports 19 languages, with English, Chinese, Russian, Spanish, Japanese, and 10 other languages performing particularly well.
  • Lyric2VocalGenerate vocal audio directly from lyrics using LoRA fine-tuning.
  • Text2SamplesGenerates music samples and loops, helping producers quickly create instrument loops, sound effects, and more.

ACE-Step Technical Principles

  • Diffusion ModelBased on progressive noise removal during data generation, the model can quickly synthesize music, while traditional diffusion models suffer from limitations in maintaining the coherence of long structural sequences. ACE-Step addresses this issue through its innovative architectural design.
  • Deep compression autoencoderDCAE is used for efficient data compression and decompression, preserving fine-grained audio details of music while reducing the consumption of computing resources.
  • Lightweight linear converterUsed to process sequence information in music, ensuring that the generated music is coherent in melody, harmony and rhythm.
  • Semantic alignmentACE-Step uses MERT (Music Embedding Representation) and m-hubert techniques to align semantic representations (REPA) during training, achieving fast convergence and high-quality generation results.
  • Training optimizationBased on semantic alignment and optimization techniques during training, ACE-Step can generate high-quality music in a short time, maintaining a balance between generation speed and coherence.

ACE-Step project address

Application scenarios of ACE-Step

  • Music compositionIt can quickly generate melodies and lyrics, providing creative inspiration.
  • Human voice generationGenerates vocal audio directly from lyrics, suitable for creating vocal presentations.
  • Music ProductionGenerates instrument loops and sound effects, providing creative materials.
  • Multilingual supportSuitable for cross-language music composition.
  • Music EducationAs a teaching tool, it helps learn music composition.
Back