Dolphin - A large-scale speech recognition model jointly developed by Tsinghua University and Haitian Ruisheng

What is Dolphin?

Dolphin is a large-scale speech model for Eastern languages, jointly developed by the Speech and Audio Technology Laboratory of the Department of Electronic Engineering at Tsinghua University and Haitian Ruisheng. It supports speech recognition for 40 Eastern languages, including 22 dialects of Chinese (including Mandarin), and can accurately identify the language characteristics of different regions. The model has been trained for a total of 212,000 hours, including 138,000 hours of high-quality proprietary data and 74,000 hours of open-source data. In terms of performance, Dolphin's word error rate (WER) is significantly lower than that of Whisper models of similar size; for example, the base version has an average WER reduction of 63.1%, and the small version a reduction of 68.2%. It adopts a CTC-Attention architecture, combining an E-Branchformer encoder and a Transformer decoder, and accelerates computation through a 4x downsampling layer while preserving key speech information.

Dolphin's main functions

• Multilingual and dialect recognitionDolphin supports speech recognition for 40 Eastern languages, covering a wide range and meeting the speech recognition needs in different language environments.
• High-precision speech-to-textDolphin can efficiently and accurately convert speech signals into text. It maintains high accuracy in speech recognition tasks involving multiple languages and dialects, effectively reducing errors and misunderstandings in the speech-to-text process.
• Customize language and region settingsUsers can flexibly specify the language and region for speech recognition according to their actual needs. Dolphin employs a two-level language labeling system: the first label specifies the language (e.g., <zh> represents Chinese), and the second label specifies the region (e.g., <CN> represents China). This enables the model to accurately capture the differences between different dialects and accents within the same language, as well as the similarities between different languages within the same region, improving the model's recognition performance in specific language and regional environments and enhancing its generalization ability.
• Open source support and secondary developmentDolphin's base and small versions of the model and inference code are fully open source, providing great convenience for developers. This open-source nature allows developers to further develop and optimize the model according to their own needs, such as fine-tuning the model for specific application scenarios, expanding model functionality, etc., to meet personalized requirements and promote the application and innovation of speech recognition technology in more fields.
• Convenient and quick way to useDolphin offers multiple usage methods, including command-line calls and Python API calls. Users can quickly start speech recognition tasks with simple command-line commands, or flexibly call models for speech processing and analysis within the Python environment, facilitating integration with other applications or systems and improving development efficiency and application flexibility.

Dolphin's technical principles

• CTC-Attention ArchitectureDolphin employs a CTC-Attention architecture, combining the sequence modeling capabilities of CTC (Connectionist Temporal Classification) with the contextual capture capabilities of the attention mechanism. This effectively captures subtle changes in speech information while maintaining high processing speed, thus improving the model's recognition accuracy and efficiency when handling complex audio input.
• E-Branchformer encoderThe E-Branchformer encoder employs a parallel branching structure, which more effectively captures the local and global dependencies of the input speech signal. This provides the model with richer feature representations, enabling it to better understand complex patterns in the speech signal and improve recognition accuracy.
• Transformer decoderDolphin's decoder uses the Transformer architecture. Transformers excel in sequence-to-sequence tasks, generating high-quality text output. Through self-attention, they effectively capture long-range dependencies in the text, ensuring the quality and coherence of the final output.
• 4x downsampling layerTo further improve training efficiency and performance, Dolphin introduces a 4x downsampling layer. This reduces the sequence length of input features, accelerates the computation process, preserves key speech information, and ensures that the model's recognition performance is not affected.
• Two-level language tagging systemDolphin introduces an innovative two-level language tagging system. The first tag specifies the language (e.g., <zh> for Chinese), and the second tag specifies the region (e.g., <CN> for China). This enables the model to capture the differences between different dialects and accents within the same language, as well as the similarities between different languages within the same region, improving the model's ability to distinguish closely related dialects and enhancing its generalization ability.

Dolphin's project address

• Github repository:https://github.com/DataoceanAI/Dolphin
• HuggingFace model library:https://huggingface.co/DataoceanAI
• arXiv technical paper:https://arxiv.org/pdf/2503.20212

Application scenarios of Dolphin

• Meeting minutesAutomatically converts audio content from meetings into text records, improving work efficiency.
• Voice input methodOn smart devices, users can input text via voice, reducing the hassle of manual input and improving input speed and convenience.
• Intelligent voice assistantIt supports multiple languages and dialects, can better understand users' voice commands, provide more accurate feedback, and reduce misunderstandings caused by dialects or accents.
• Smart HomeUsers can control smart home devices, such as lights and air conditioners, via voice commands, enhancing the intelligence and convenience of their homes.
• News mediaQuickly convert audio content such as news interviews and podcasts into text for easy editing and publishing.
• Language learningIt supports the recognition of multiple languages and dialects, which can help students practice pronunciation and language expression.