FireRedASR - Xiaohongshu's open-source automatic speech recognition model

What is FireRedASR?

FireRedASR is an open-source, industrial-grade automatic speech recognition (ASR) model family from Xiaohongshu (Little Red Book). It supports Mandarin, Chinese dialects, and English, achieving a new state-of-the-art (SOTA) level in Mandarin ASR benchmark tests and performing exceptionally well in lyrics recognition. The model family includes two main versions:

FireRedASR-LLM: Employing the Encoder-Adapter-LLM framework and leveraging the capabilities of large language models (LLMs), it achieves state-of-the-art (SOTA) performance and supports seamless end-to-end voice interaction. In Mandarin benchmark tests, its average character error rate (CER) is 3.05%, a reduction of 8.4% compared to the previous SOTA model (3.33%).

FireRedASR-AED: Employing an attention-based encoder-decoder (AED) architecture, it balances high performance and computational efficiency, serving as an effective speech representation module in LLM-based speech models. In Mandarin benchmark tests, it achieves an average CER of 3.18%, outperforming Seed-ASR, which has over 12 parameters.

FireRedASR's main functions

• High-precision speech recognitionFireRedASR includes two versions: FireRedASR-LLM and FireRedASR-AED. Among them, FireRedASR-LLM uses the Encoder-Adapter-LLM framework, focusing on ultimate speech recognition accuracy.
• Efficient ReasoningFireRedASR-AED is based on the classic Attention-based Encoder-Decoder architecture with 1.1B parameters, balancing high accuracy and inference efficiency.
• Multi-scenario adaptationFireRedASR performs exceptionally well across a variety of everyday scenarios, including short videos, live streaming, voice input, and smart assistants. Compared to leading ASR service providers and Paraformer-Large, FireRedASR-LLM achieves a relative CER reduction of 23.7% to 40.0%.
• Lyrics recognition capabilityIn lyrics recognition scenarios, FireRedASR-LLM achieved a relative reduction in CER of 50.2% to 66.7%, demonstrating extremely strong adaptability.
• Multilingual supportFireRedASR supports Mandarin and performs exceptionally well in Chinese dialects and English speech recognition, further expanding its application scope.
• Open source and community supportFireRedASR's model and inference code are both open source, driving community-driven improvements and academic research in speech recognition technology.

FireRedASR Technical Principles

• FireRedASR-LLMFireRedASR-LLM employs the Encoder-Adapter-LLM framework, combining the capabilities of large language models (LLMs) to achieve exceptional speech recognition accuracy. It comprises three core components:
  • Conformer Basic Encoder: Responsible for extracting speech features and generating continuous speech representations.
  • Lightweight adapter: Converts the encoder's output into a representation that matches the LLM semantic space.
  • Pre-trained text LLM: Based on Qwen2-7B-Instruct initialization, used to generate the final text output.
  • Training strategyDuring training, the encoder and adapter are trainable, and most parameters of the LLM remain fixed, only fine-tuned through Low-Rank Adaptation (LoRA). This ensures that the encoder and adapter can effectively map speech features to the semantic space of the LLM while preserving the pre-training capabilities of the LLM.
  • Input and ReasoningDuring inference, the input includes prompts and speech. The LLM performs next-token-prediction to generate the recognized text.
• FireRedASR-AEDFireRedASR-AED is based on the classic attention-based encoder-decoder (AED) architecture, balancing high performance and computational efficiency. It consists of the following components:
  • Conformer encoderIt processes speech features based on the Conformer model, and can capture both local and global dependencies simultaneously.
  • Transformer decoderIt adopts the Transformer architecture for sequence transformation, including a multi-head self-attention module and a feedforward module.
  • Input featuresThe input features are 80-dimensional log Mel filter banks, which are then processed by global mean and variance normalization.
  • Training dataThe training data includes approximately 70,000 hours of high-quality Mandarin audio data and approximately 11,000 hours of English audio data.

FireRedASR project address

• Github repository:https://github.com/FireRedTeam/FireRedASR
• HuggingFace model library:https://huggingface.co/FireRedTeam/FireRedASR-AED-L
• arXiv technical paper:https://arxiv.org/pdf/2501.14350

Application scenarios of FireRedASR

• Intelligent voice assistantFireRedASR can be used to develop intelligent voice assistants, such as smart home control and intelligent customer service. Its high-precision voice recognition capabilities accurately understand user voice commands, providing a smooth interactive experience.
• Videos and live streamsIn the field of short videos and live streaming, FireRedASR can generate subtitles in real time to help viewers better understand the content.
• Lyrics recognitionFireRedASR performs particularly well in lyrics recognition scenarios and can be widely used in music platforms and karaoke settings.
• Voice inputFireRedASR can be used in voice input scenarios, such as voice typing and voice note-taking. Its efficient reasoning capabilities and high-precision recognition significantly improve user input efficiency.