FireRedTTS-2 - A streaming text-to-speech system launched by Xiaohongshu (Little Red Book)

What is FireRedTTS-2?

FireRedTTS-2 is an advanced long-format streaming text-to-speech (TTS) system focused on multi-speaker dialogue generation. Employing a 12.5Hz streaming speech segmenter and a dual Transformer architecture, it achieves low-latency, high-fidelity, and multilingual speech synthesis. It supports multiple languages including English, Chinese, Japanese, Korean, French, German, and Russian, and features zero-sample cross-language and code-switching speech cloning capabilities. Currently, it supports 3-minute dialogue generation with 4 speakers, and the dialogue length and number of speakers can be further extended by expanding the training corpus. It excels in podcast generation and chatbot integration, providing stable and natural speech output and generating emotionally resonant speech based on context.

Main functions of FireRedTTS-2

• Long dialogue speech generationSupports 3-minute dialogue generation with 4 speakers, and the training corpus can be expanded to increase dialogue duration and number of speakers.
• Multilingual supportIt covers languages such as English, Chinese, Japanese, Korean, French, German, and Russian, and has zero-sample cross-language and code-switching speech cloning capabilities.
• Low latency and high fidelityIn an L20 GPU environment, the initial data packet latency is as low as 140 milliseconds, which is suitable for real-time interactive scenarios while ensuring high-quality audio output.
• Stable voice outputIn monologue and dialogue tests, the generated speech showed high similarity to the target speaker, low speech recognition error rate, and maintained stable sound quality and rhythm.
• Random timbre generationIt can generate speech with random features, which is suitable for building training data for speech recognition models or providing diverse test materials for voice interaction systems.
• Emotional rhythm generationIn chatbot integration, it can generate emotionally rich voice based on context, enhancing the interactive experience.
• Real-time streaming generationIt adopts a 12.5Hz streaming speech segmenter, supports high-fidelity streaming decoding, and is suitable for real-time applications.

Technical Principles of FireRedTTS-2

• 12.5Hz Streaming Speech SegmenterIt runs at a low frame rate, can encode richer semantic information, shorten speech sequences, stabilize text-to-word segmentation modeling, and supports high-fidelity streaming decoding, making it suitable for real-time applications.
• Dual Transformer architectureThe method employs a text-to-speech interleaving format, concatenating speaker-tagged text with aligned speech segments in chronological order, and modeling using dual Transformers. A large decoder uses only the Transformer to predict the segments of the first layer, while a smaller Transformer completes the subsequent layers.
• Multilingual modelingThrough multilingual pre-training, it supports speech generation in multiple languages, has zero-shot cross-language and code-switching speech cloning capabilities, and can adapt to dialogue scenarios in different languages.
• Low latency designThe model architecture and inference process have been optimized to ensure that the initial data packet latency can be as low as 140 milliseconds in an L20 GPU environment, meeting the needs of real-time interaction.
• Long conversation supportThrough efficient word segmentation and modeling mechanisms, it supports the generation of 3-minute dialogues with 4 speakers, and the dialogue duration and number of speakers can be further extended by expanding the training corpus.
• Context-aware prosodyWhen generating speech, it can adjust the rhythm and emotion according to the context information, making the speech output more natural and expressive.

FireRedTTS-2 Project Address

• Project official website: https://fireredteam.github.io/demos/firered_tts_2/
• Github repository: https://github.com/FireRedTeam/FireRedTTS2
• arXiv technical paper: https://arxiv.org/pdf/2509.02020v1

Application scenarios of FireRedTTS-2

• Podcast generationIt can generate podcast content with multiple speakers, supports multiple languages, and provides stable and natural voice output, making it suitable for producing multilingual podcast programs.
• ChatbotIt can be integrated into chat frameworks to generate emotionally rich voices based on context, enhancing the interactive experience and making it suitable for various chatbot applications.
• Voice cloningIt supports zero-sample cross-language and code-switching speech cloning, and can generate speech with high similarity to the target speaker, making it suitable for speech cloning related applications.
• Voice interaction systemIt can be used to build voice interaction systems, provide diverse test materials, support random tone generation, and meet the needs of different scenarios.
• Speech recognition model trainingIt can generate speech with random features, which is suitable for building speech recognition model training data and reducing the reliance on real recording data.
• Multilingual speech synthesisIt supports speech synthesis in multiple languages and is suitable for voice applications that require multilingual support, such as international conferences and multilingual customer service.