AB
AB
AiBoss

All Categories

project

Step-Audio - Step-Star's open-source voice interaction model

Step-Audio is the first product-level open-source voice interaction model launched by the Step-Leap Star team. It can generate expressions with emotions, dialects, languages, singing voices, and personalized styles according to different scenario needs, enabling natural and high-quality interaction with users...

What is Step-Audio?

Step-Audio is the first product-grade open-source voice interaction model launched by the Step-Leap Star team. It can generate expressions with emotions, dialects, languages, singing voices, and personalized styles according to different scenario needs, enabling natural, high-quality dialogue with users. Based on a unified model with 130 parameters, Step-Audio combines speech understanding and generation, supporting functions such as speech recognition, dialogue, and speech synthesis. Step-Audio's core advantages include: a highly efficient speech data generation engine, fine-grained voice control capabilities supporting multiple emotions and dialects, enhanced tool invocation and role-playing functions, and effective handling of complex tasks. In terms of performance, Step-Audio performs excellently in multiple benchmark tests, demonstrating a significant leading advantage in command compliance and complex voice interaction scenarios.

Step-Audio's main functions

  • Unified Speech Understanding and GenerationIt simultaneously processes speech recognition (ASR), semantic understanding, dialogue generation, and text-to-speech (TTS) to achieve end-to-end voice interaction.
  • Multilingual and dialect supportIt supports multiple languages and dialects (such as Cantonese, Sichuanese, etc.) to meet the needs of users in different regions.
  • Emotional and style controlSupports the generation of speech with specific emotions (such as anger, joy, sadness) and styles (such as rap, singing).
  • Tool usage and role-playingIt supports real-time tool calls (such as checking the weather and obtaining information) and role-playing, improving the flexibility and intelligence of interaction.
  • High-quality speech synthesisBased on the open-source Step-Audio-TTS-3B model, it provides natural and fluent speech output and supports voice cloning and personalized speech generation.

Step-Audio's technical principles

  • Dual-codebook speech segmenterSpeech is segmented using a language codebook (16.7Hz, 1024 codebooks) and a semantic codebook (25Hz, 4096 codebooks). Speech features are integrated using a 2:3 time-interleaved approach to improve the semantic and acoustic representation capabilities of the speech.
  • 130B parameter multimodal large modelBased on the Step-1 pre-trained text model, this system enhances the model's ability to understand and generate speech and text through continuous pre-training and post-training with audio context. It supports bidirectional interaction between speech and text, achieving unification of speech recognition, dialogue management, and speech synthesis.
  • Mixed speech synthesizerIt combines stream matching and neural vocoder technology to optimize real-time waveform generation. It supports high-quality speech output while preserving the emotional and stylistic features of the speech.
  • Real-time inference and low-latency interactionEmploying a speculative response generation mechanism, it generates possible responses in advance when the user pauses, reducing interaction latency. Based on Voice Activity Detection (VAD) and a streaming audio segmenter, it processes input speech in real time, improving the smoothness of interaction.
  • Reinforcement learning and instruction followingWe use reinforcement learning with human feedback (RLHF) to optimize the model's dialogue capabilities, ensuring that the generated speech better matches human commands and semantic logic. Based on command labels and multi-turn dialogue training, we improve the model's performance in complex scenarios.

Step-Audio project address

Application scenarios of Step-Audio

  • Intelligent voice assistantUsed in smart home, office and other scenarios, it supports voice interaction to complete tasks.
  • Intelligent Customer ServiceIt provides multilingual and dialect support and responds quickly to user issues.
  • EducationIt assists in language learning and supports emotional voice output.
  • Entertainment and GamesGenerate personalized voices to enhance immersion.
  • Accessibility technology: To help visually or speech-impaired people with speech impairments to interact via voice.
Back