Step-Audio-TTS-3B - A high-performance TTS model capable of generating speech with specific emotions and rap styles.

What is Step-Audio-TTS-3B?

Step-Audio-TTS-3B is a high-performance text-to-speech (TTS) model developed by the Stepfun-AI team, boasting powerful speech synthesis capabilities. Trained on massive amounts of synthetic data, it features 3 billion parameters and can generate natural, fluent, and expressive speech. The model supports multiple languages and dialects, including Mandarin, English, Japanese, Cantonese, and Sichuanese. It can generate speech expressing different emotions, such as joy, sadness, or anger, through emotion control. Step-Audio-TTS-3B also supports speech synthesis with special prosodic styles, such as rap, to meet diverse scenario needs.

Main functions of Step-Audio-TTS-3B

• Multilingual and dialect supportIt supports multiple languages (such as Chinese, English, and Japanese) and dialects (such as Cantonese and Sichuanese) to meet the needs of users in different regions.
• Emotional and style controlIt can generate speech with specific emotions (such as anger, joy, sadness) and styles (such as rap, humming), and supports fine-grained speech control.
• High-quality speech synthesisIt provides natural and fluent voice output, supports voice cloning and personalized voice generation, and enhances the realism of voice interaction.
• Enhanced instruction tracing capabilitiesThrough a command-driven control system, controllable speech synthesis can be achieved, accurately following the user's instructions.
• High-efficiency data generationBreaking away from the reliance of traditional TTS on manually collected data, this approach enhances the model's generalization ability and generation efficiency through large-scale synthetic data training.

The technical principle of Step-Audio-TTS-3B

• Dual codebook encoder architectureThe model employs a dual-codebook encoder scheme using a linguistic tokenizer and a semantic tokenizer. The linguistic tokenizer has a bitrate of 16.7 Hz and a codebook size of 1024, used to capture language structure information; the semantic tokenizer has a bitrate of 25 Hz and a codebook size of 4096, used to capture finer acoustic details.
• High-efficiency synthetic data linkBreaking away from the traditional reliance on manually collected data in TTS, it generates high-quality synthetic audio data through a cyclical iterative framework of large-scale synthetic data generation and model training.
• Hybrid speech decoderCombining flow matching and a mel-to-wave vocoder, discrete labeled information is converted into continuous speech signals, optimizing the clarity and naturalness of synthesized speech.
• Command-driven precision control systemIt supports precise control of various emotions (such as anger, happiness, sadness), dialects (such as Cantonese, Sichuanese) and vocal styles (such as rap, humming) to meet diverse speech generation needs.
• Pre-training and fine-tuningThe audio is continuously pre-trained based on the Step-1 multimodal language model with 130 billion parameters, and the speech generation capability of the model is enhanced through task-oriented fine-tuning.
• Real-time inference pipelineBy using a streaming audio segmenter and a speculative response generation mechanism, interaction latency is reduced, and the real-time performance and response speed of the system are improved.

Step-Audio-TTS-3B project address

• HuggingFace model library:https://huggingface.co/stepfun-ai/Step-Audio-TTS-3B

Application scenarios of Step-Audio-TTS-3B

• Intelligent voice assistantThe Step-Audio-TTS-3B can be integrated into smart home devices, office equipment, or mobile devices to enable functions such as voice control, information retrieval, and schedule management.
• Intelligent Customer ServiceIn customer service systems, models can provide real-time voice interaction, quickly respond to user questions, support multiple languages and dialects, and significantly improve service quality and efficiency.
• EducationIt can be used in language learning software to provide real-time voice dialogue practice, supports multiple languages and dialects, and helps learners improve their oral skills.
• Entertainment and GamesIn role-playing games (RPGs) or interactive stories, Step-Audio-TTS-3B can generate voices with emotion, dialects, and style, enhancing the player's immersion.
• Intelligent vehicle systemThe model can be used in in-vehicle voice systems to provide voice navigation, information query and entertainment control functions, and supports natural voice interaction and multiple dialects.