ACE-Step 1.5 - A music generation model developed by ACE Studio in collaboration with StepFun.

What is ACE-Step 1.5?

ACE-Step 1.5 is an open-source music generation model jointly developed by ACE Studio and StepFun, enabling commercial-grade music generation on consumer-grade hardware. The model employs a hybrid architecture: a language model acts as a planner, transforming user prompts into song blueprints, while a Diffusion Transformer handles acoustic rendering. Through 4-8 steps of distillation inference, it generates a 4-minute song in just 2 seconds on an A100 and approximately 10 seconds on an RTX 3090, with a VRAM requirement of less than 4GB. ACE-Step 1.5 supports over 50 languages, precise style control, and editing functions such as covers, redrawing, and vocal-to-instrument conversion. Users can train LoRA with a small number of songs to achieve personalized style transfer.

Main functions of ACE-Step 1.5

• Music generationIt supports generating complete songs from text prompts, with lyrics in more than 50 languages, and can be flexibly expanded to any length of music, from 10-second short loops to 10-minute long works.
• Editing functionIt offers six major editing capabilities: audio re-rendering, cover song generation, vocal-to-accompaniment conversion, track separation, layered arrangement, and continuation completion, enabling refined control and re-creation of existing audio.
• Style controlIt can accurately parse and execute complex cue words containing professional musical terminology, achieving zero-sample timbre cloning and strict style adherence.
• PersonalizationUsers only need to provide a few reference songs to quickly train a customized model that captures their unique personal style through LoRA with minimal fine-tuning.
• Efficiency characteristicsThe model can run locally on consumer-grade GPUs with less than 4GB of video memory, achieving sub-second high-speed generation and supporting batch parallel sampling to explore diverse creative candidates.

Technical Principles of ACE-Step 1.5

• Hybrid Reasoning-Diffusion ArchitectureACE-Step 1.5 employs a dual-component collaborative architecture, decoupling music generation into two stages: planning and rendering. The language model (based on Qwen3-0.6B) acts as the "composer agent," using thought chain reasoning to transform user prompts into a YAML-formatted blueprint containing BPM, tonality, duration, lyrics, and acoustic descriptions. The Diffusion Transformer (approximately 2 billion parameters) acts as the acoustic renderer, receiving standardized conditions and focusing on generating high-fidelity audio. This division of labor relieves DiT of the burden of semantic understanding, while the multi-task training of the LM ensures robust alignment across more than 50 languages.
• Efficient Reasoning OptimizationTo achieve real-time generation on consumer-grade hardware, the team introduced an adversarial dynamic offset distillation technique. Based on Decoupled DMD2, it introduces a GAN target and latent space discriminator. By randomly sampling offset parameters from {1,2,3}, the model is exposed to diverse denoising states, avoiding overfitting caused by a fixed step size. This approach compresses the inference steps from 50 to 4-8, generating a 240-second audio track on an A100 in approximately 1 second, achieving a 200x speedup. Furthermore, the adversarial feedback helps the student model surpass the teacher model in audio quality performance.
• Intrinsic reinforcement learning alignmentThe system establishes a unified internal reinforcement learning framework to avoid external bias. For DiT, an Attention Alignment Score (AAS) is proposed as an intrinsic reward. Dynamic time warping is used to measure lyric token coverage, attention monotonicity, and path confidence. After optimization, the correlation between lyrics-audio synchronization and human judgment exceeds 95%. For LM, the GRPO algorithm is adopted, and a reward model is constructed using point mutual information. LM is regarded as a dual role of "composer" and "listener". PMI is used to penalize general descriptions and reward specific annotations. The final reward is dynamically weighted by style and atmosphere (50%), lyric content (30%), and metadata constraints (20%).
• Unified mask generation frameworkFinite Scalar Quantization (FSQ) discretizes continuous audio latent variables into a 5Hz codebook representation, constructing a flexible mask generation paradigm. Manipulating source latent variables and mask configurations, a single model can support six modalities: text-to-music, cover songs, re-rendering, track extraction, layering, and completion. FSQ compresses the 25Hz latent space into structured source latent variables through attention pooling. After concatenation with noise targets and masks, the structured representation is processed by a patchify layer, simplifying multi-task training. Quantizing latent variables ensures high-fidelity preservation of melodic and rhythmic elements during the conversion process.

Project address for ACE-Step 1.5

• Project official websitehttps://ace-step.github.io/ace-step-v1.5.github.io/
• GitHub repository: https://github.com/ace-step/ACE-Step-1.5
• arXiv technical paper: https://arxiv.org/pdf/2602.00744
• Experience the demo onlinehttps://huggingface.co/spaces/ACE-Step/Ace-Step-v1.5

Application scenarios of ACE-Step 1.5

• Music composition and productionMusicians and producers can use ACE-Step 1.5 as an inspiration generation tool to quickly transform text descriptions into complete song drafts and break through creative bottlenecks.
• Personalized content creationContent creators can fine-tune their personal style models using LoRA to generate customized background music in batches for projects such as videos, podcasts, and games, maintaining timbre consistency across different works.
• Multilingual music productionThe model supports accurate lyric generation and singing in more than 50 languages, and is suitable for global music distribution, cross-cultural cooperation projects, and content production in the music market of less commonly spoken languages.
• Education and LearningMusic learners can observe the model's generated results by inputting professional terms (such as specific modes and chord progressions), thus gaining an intuitive understanding of music theory concepts.