Fun-ASR-Realtime - A streaming real-time speech recognition model launched by Alibaba Qianwen - AiBoss

What is Fun-ASR-Realtime?

Fun-ASR-Realtime is a large-scale streaming real-time speech recognition model launched by Alibaba's Qianwen platform. The model uses the WebSocket streaming protocol to output words as the user speaks, with a latency of only a few hundred milliseconds for the first word and extremely low latency for the last word. The model supports real-time recognition of Mandarin, 16 dialects, and 30 languages, and features capabilities such as contextual understanding, customized hot words, sentiment analysis, and timestamp output.

The main functions of Fun-ASR-Realtime

Real-time streaming recognitionIt achieves low-latency audio-to-text conversion through WebSocket full-duplex communication, supporting simultaneous speaking and text output.
Multi-dialect and multilingual supportIt covers 16 dialects (including Cantonese, Wu, Min, Hakka, etc.) and 30 languages, and is suitable for scenarios such as overseas customer service and international conferences.
Context correctionIt can self-correct based on historical dialogue context and real-time hot words, such as correcting "叶鹿" to "夜鹭" based on the context.
Emotion recognitionThe Qwen-ASR model supports 7 types of fine-grained emotion recognition; Paraformer supports 3 types of polar emotion recognition.
Timestamp outputFun-ASR and Paraformer output sentence-level and character-level timestamps by default, which facilitates subtitle alignment and keyword highlighting.
Hot word customizationIt supports improving the recognition accuracy of brand names, personal names, and proper nouns through hot keywords.
VAD Intelligent Sentence BreakdownThe server automatically detects the start and end points of voice messages and supports customizable silence thresholds and sensitivity.
Non-human voice filteringAutomatically filters out non-human voice content to improve recognition accuracy.

The technical principle of Fun-ASR-Realtime

Large Model Streaming ArchitectureThe model uses a large model based on Transformer or Conformer as the backbone network, and realizes audio input and output on the same time through streaming inference mechanism, avoiding waiting for the entire audio segment to finish before processing, and compressing the first word delay to the level of hundreds of milliseconds.
Causal attention and chunkingThe model employs causal attention masks during inference to ensure that the prediction at each time step depends only on the current and historical audio frames, and not on future information; at the same time, it balances latency and accuracy by using fixed-length audio blocks as input.
General Context Reinforcement TrainingDuring the training phase, historical dialogue context and real-time hotwords are introduced as auxiliary inputs. Through the context-aware rescoring mechanism, the model can dynamically correct the recognition results according to the context, such as correcting the leaf deer to a night heron based on the birdwatching context.
Multi-task joint modelingThe model is trained on the main task of speech recognition, and jointly trained on the tasks of emotion recognition and speech activity detection (VAD). It extracts general acoustic representations through a shared encoder and outputs transcribed text, emotion tags and sentence break boundaries in a single forward propagation.
Timestamp alignment mechanismBased on connection-time classification (CTC) or attention alignment path, the start and end timestamps of each character/word in the audio are predicted simultaneously during the decoding process, and the output is at both sentence-level and character-level granularity to meet the requirements of subtitle alignment and keyword localization.

How to use Fun-ASR-Realtime

Obtain credentialsObtain the API Key from the Alibaba Cloud Refinement console (https://help.aliyun.com/zh/model-studio/real-time-speech-recognition-user-guide) and configure it in your environment variables.
Install SDKInstall the latest version of the DashScope SDK via pip or Maven.
Configuration parametersCreate a Recognition instance and set the model name. fun-asr-realtimeAudio format (such as pcm/wav) and sampling rate.
Establish connectionEstablish a full-duplex connection with the Alibaba Cloud inference server via WebSocket.
Send audio: Read microphone or audio file data in a loop with a fixed frame length and send it.
Received resultsHandle real-time returns in the callback. on_event Events, retrieve intermediate results and sentence-level final results.
End of missionCalled after audio transmission is complete stop() Or send finish-task The command closes the identification session.

Fun-ASR-Realtime's core advantages

Low latency and high accuracyThe first character has a delay of hundreds of milliseconds, and the accuracy is close to that of the offline model, achieving both speed and accuracy.
Leading in dialect recognitionIn 12 out of 16 dialect tests, it outperformed similar products from Volcano and Tencent, with the Wu dialect accuracy rate being among the best in the industry.
Context self-correctionIt possesses the ability to understand context and can automatically correct previously misidentified content based on subsequent information.
Robust in industrial scenariosIn complex backgrounds, noisy far-field scenarios, and scenes with accents, its accuracy in recognizing both Chinese and English is superior to that of its competitors.
Convenient Ecosystem AccessProvided uniformly through Alibaba Cloud's Bailian platform, it supports multi-language SDKs and direct WebSocket connections, offering a complete high-concurrency best practice solution.

Comparison of Fun-ASR-Realtime with similar competing products

Dimension	Fun-ASR-Realtime	GPT-4o-Transcribe/Realtime API
Product Positioning	Large model designed specifically for streaming real-time speech recognition	The GPT-4o-based transcription model supports streaming via Realtime API or batch transcription via REST.
First character delay	Millisecond-level, native streaming architecture	In streaming mode, the latency is approximately 300–500ms, depending on the network and VAD configuration; batch mode has no real-time capability.
Streaming architecture	Native WebSocket full-duplex streaming, end-to-end optimization	Streaming input is supported via the Realtime API WebSocket, but transcription mode only returns the complete result at the end of the turn, resulting in limited delta continuity.
Dialect support	Accuracy rate of 92.41% for 16 dialects (Wu, Min, Hakka, etc.) and Shanghainese.	99+ languages have zero sample coverage, but the accuracy of dialects/low-resource languages drops significantly, and there is no specific optimization for Chinese dialects.
Context correction	Supported; can automatically correct previous misidentifications based on context (e.g., "leaf deer" → "night heron").	Based on GPT-4o language understanding capabilities, it has advantages in context parsing ambiguous words (such as their/there/they’re), but lacks a real-time cross-sentence error correction mechanism.

Application scenarios of Fun-ASR-Realtime

Live Streaming SubtitlesIt provides millisecond-level latency subtitle output for long-duration live broadcasts and multi-person interactive scenarios, supports frequent speaker switching and complex competition environments, and has been verified in the 100-hour live broadcast of "Return to the Desert Island" in the film and television storm, with a total of 1.32 million words of real-time subtitles output.
Real-time meeting transcriptionIn scenarios such as remote meetings and offline forums, it can output structured text with sentence-level and character-level timestamps while speaking, making it easy to quickly review key speeches and locate time nodes after the meeting.
Customer service call quality inspectionIt can identify the two-way dialogue between customer service and users in real time, detect emotional state simultaneously, and support hot word customization to accurately capture brand name, product name and complaint keywords, so as to realize real-time risk warning during the call.
Voice input methodIt provides low-latency dialect input capabilities for mobile devices, in-vehicle terminals, and IoT devices, supporting real-time transcription of 16 dialects including Cantonese, Wu, and Min, improving the input efficiency and experience for non-Mandarin speakers.