AB
AB
AiBoss

All Categories

project

Step-Audio-AQAA - StepFun's end-to-end large audio language model

Step-Audio-AQAA is an end-to-end large-scale audio language model developed by the StepFun team, specifically designed for Audio Query-Audio Response (AQAA) tasks. It can directly process audio input to generate natural and accurate speech responses, without...

What is Step-Audio-AQAA?

Step-Audio-AQAA is an end-to-end large-scale audio language model from the StepFun team, specifically designed for Audio Query-Audio Response (AQAA) tasks. It directly processes audio input to generate natural and accurate speech responses, without relying on traditional Automatic Speech Recognition (ASR) and Text-to-Speech (TTS) modules, simplifying the system architecture and eliminating cascading errors. The training process for Step-Audio-AQAA includes multimodal pre-training, supervised fine-tuning (SFT), Direct Preference Optimization (DPO), and model merging. Through these methods, the model performs exceptionally well in complex tasks such as voice emotion control, role-playing, and logical reasoning. In the StepEval-Audio-360 benchmark, Step-Audio-AQAA outperforms existing LALM models across multiple key dimensions, demonstrating its powerful potential in end-to-end voice interaction.

Main functions of Step-Audio-AQAA

  • Direct audio input processingIt can generate voice responses directly from raw audio input without relying on traditional automatic speech recognition (ASR) and text-to-speech (TTS) modules.
  • Seamless voice interactionIt supports voice-to-voice interaction, allowing users to ask questions by voice and the model to answer directly by voice, thus improving the naturalness and fluency of the interaction.
  • Emotional tone adjustmentIt supports adjusting the emotional tone of the voice at the sentence level, such as expressing emotions like happiness, sadness, or seriousness.
  • Speech rate controlUsers can adjust the speed of their voice responses to better suit the specific needs of the scenario.
  • Timbre and pitch controlIt can adjust the timbre and tone of the voice according to user instructions to adapt to different roles or scenarios.
  • Multilingual interactionIt supports multiple languages, including Chinese, English, and Japanese, to meet the language needs of different users.
  • Dialect supportIt covers Chinese dialects such as Sichuanese and Cantonese, improving the applicability of the model in specific regions.
  • Voice emotion controlIt can generate voice responses with specific emotions based on context and user instructions.
  • role playIt supports playing specific roles in conversations, such as customer service, teacher, friend, etc., and generates voice responses that match the characteristics of the role.
  • Logical reasoning and knowledge-based question answeringIt can handle complex logical reasoning tasks and knowledge-based question answering, and generate accurate voice responses.
  • High-quality voice outputIt generates high-fidelity, natural, and fluent speech waveforms through a neural vocoder, enhancing the user experience.
  • Speech coherenceMaintaining the coherence and consistency of speech during the generation of long sentences or paragraphs, avoiding interruptions or abrupt changes in speech.
  • Text and speech interleaved outputIt supports interleaved text and voice output, allowing users to choose between voice or text responses as needed.
  • Multimodal input understandingIt can understand mixed inputs containing both speech and text and generate corresponding speech responses.

The technical principles of Step-Audio-AQAA

  • Dual-codebook audio word segmenterThis function converts the input audio signal into a structured token sequence. It includes two segmenters: a language segmenter extracts phonemes and linguistic attributes from the speech, sampling at 16.7 Hz with a codebook size of 1024; and a semantic segmenter captures acoustic features of the speech, such as emotion and intonation, sampling at 25 Hz with a codebook size of 4096. This allows for better capture of complex information in speech.
  • Backbone LLMWe use a pre-trained 130 billion parameter multimodal LLM (Step-Omni), with pre-trained data covering text, speech, and image modalities. We embed dual-codebook audio tags into a unified vector space and perform deep semantic understanding and feature extraction through multiple Transformer blocks.
  • Neural vocoderThis method synthesizes generated audio tags into natural, high-quality speech waveforms. Employing the U-Net architecture, combined with ResNet-1D layers and Transformer blocks, it efficiently converts discrete audio tags into continuous speech waveforms.

Step-Audio-AQAA project address

  • HuggingFace model libraryhttps://huggingface.co/stepfun-ai/Step-Audio-AQAA
  • arXiv technical paper: https://arxiv.org/pdf/2506.08967

Application scenarios of Step-Audio-AQAA

  • Emotional companion robotIt automatically adjusts its response tone based on the user's emotions, providing emotional support and companionship.
  • Multilingual Intelligent Customer ServiceIt can directly handle dialect voice queries and supports multiple languages (such as Chinese, English, Japanese, etc.).
  • Game NPC InteractionIt generates voice feedback with emotional changes in real time. It enables dynamic switching of emotion and speech rate in single-turn conversations.
  • Intelligent voice assistantIt supports voice query and voice answer, and provides services such as real-time information query and schedule reminder.
  • Education and EntertainmentUsed in educational settings for voice-based teaching, story creation, and poetry recitation. It can flexibly switch between voice and text responses based on user needs.
Back