FireRedASR2S - Xiaohongshu's open-source speech recognition model

What is FireRedASR2S?

FireRedASR2S is an open-source, industrial-grade end-to-end speech recognition model developed by Xiaohongshu's Super Intelligence-AudioLab. It integrates four state-of-the-art (SOTA) modules: ASR, VAD, language recognition, and punctuation prediction. The model supports Mandarin Chinese and over 20 dialects, English, code switching, and lyrics recognition. It boasts a low error rate of 2.89% for Mandarin Chinese and an average of 11.55% for dialects, significantly outperforming competitors like Doubao-ASR and Qwen3-ASR. The system supports one-click local deployment without external APIs and has already been deployed on a large scale in high-frequency scenarios such as Xiaohongshu's voice comments and voice search.

Main functions of FireRedASR2S

• Speech recognition (FireRedASR2)Supports Mandarin Chinese, 20+ dialects/accents, English, mixed Chinese and English, and lyrics recognition. Offers both LLM and AED architecture versions. The AED version supports word-level timestamps and confidence output.
• Voice Activity Detection (FireRedVAD)The model can detect speech/singing/music, supports 100+ languages, and provides both streaming and non-streaming modes, achieving an F1 score of 97.57%.
• Language recognition (FireRedLID)It supports the recognition of 100+ languages and 20+ Chinese dialects with an accuracy rate of 97.18%, which is significantly better than open source solutions such as Whisper.
• Punctuation Prediction (FireRedPunc)The model automatically adds Chinese and English punctuation, with an average F1 score of 78.90%, significantly improving the readability of the transcribed text.

Technical Principles of FireRedASR2S

• Speech recognition (FireRedASR2)The model employs two architectures: Encoder-Adapter-LLM and Attention-based Encoder-Decoder. The LLM version utilizes the capabilities of a large language model to achieve end-to-end speech understanding, while the AED version optimizes computational efficiency within the encoder-decoder framework, fusing speech and text representations through an adapter layer and supporting timestamp and confidence output.
• Voice Activity Detection (FireRedVAD)This paper models audio temporal features based on a deep feedforward sequence memory network (DFSMN). It uses a smoothing window and thresholding to determine the start and end points of speech, distinguishing audio events such as speech, singing, and music, and supports streaming processing to meet real-time requirements.
• Language recognition (FireRedLID)The FireRedASR2 encoder is reused to extract speech representations, and a classifier is trained to predict language labels. Large-scale multilingual data is used for pre-training to establish a cross-language shared representation space, achieving high-precision recognition of 100+ languages and dialects.
• Punctuation Prediction (FireRedPunc)Based on the BERT architecture, this algorithm takes text without punctuation as input and predicts the punctuation type at each position. Through fine-tuning with multi-domain Chinese and English data, it learns the semantic and syntactic structure of the text and automatically inserts appropriate punctuation.

FireRedASR2S Project Address

• GitHub repository: https://github.com/FireRedTeam/FireRedASR2S
• HuggingFace model libraryhttps://huggingface.co/collections/FireRedTeam/fireredasr2s

Application scenarios of FireRed ASR2S

• Content community interactionIt supports features such as voice comments and voice search on Xiaohongshu, allowing users to participate in community interactions using diverse voices such as dialects and songs, enhancing the platform's liveliness and fun.
• Social and CommunicationIt enables scenarios such as private voice messaging and New Year's greetings, achieving natural and fluent voice input and real-time transcription, lowering the communication threshold and improving the efficiency of emotional transmission.
• Content creation and productionIt supports creation tools such as voice note publishing, live stream subtitle generation, and automatic video subtitles, helping creators to efficiently produce multimedia content.
• Enterprise-level servicesIt is suitable for B2B scenarios such as meeting transcription, intelligent customer service, and telephone analysis. Its private deployment capability meets the data security and compliance requirements of industries such as finance and healthcare.