Mega-ASR - an open-source speech recognition model from NTU, NUS, and Shanghai AI Lab.

What is Mega-ASR?

Mega-ASR is an open-source, robust speech recognition platform model for all scenarios, jointly developed by Nanyang Technological University (NTU), National University of Singapore (NUS), and Shanghai Artificial Intelligence Laboratory. Based on the Qwen3-ASR 1.7B underlying architecture, the model addresses the challenges of speech recognition in complex acoustic environments in the real world (such as noise, far field, echo, occlusion, and transmission packet loss), proposing a scalable framework for constructing composite data and progressive acoustic-to-semantic optimization.

Main functions of Mega-ASR

• Robust speech recognition across all scenariosIt covers 7 atomic acoustic effects (noise, far field, occlusion, echo reverberation, recording coloration, electronic distortion, and transmission packet loss) and 54 physically reasonable composite acoustic scenarios, enabling a single model to cope with a variety of real-world environments.
• Progressive Acoustics to Semantic Optimization (A2S-SFT)The three-stage progressive training first trains the encoder and aligner to adapt to moderately degraded speech (WER<<30%→50%→70%), then activates the semantic recovery capability of the large language model, and finally performs end-to-end joint fine-tuning to solve the two major coupling bottlenecks of "acoustic perception" and "semantic reconstruction".
• Dual-granularity dynamic reward optimization (DG-WGPO)To address the issue that the model error pattern abruptly changes from "word-level confusion" to "sentence-level illusion/sentence loss" when WER>30%, we introduce token-level refinement rewards and sentence-level reconstruction rewards, and enhance semantic preservation capabilities under extreme conditions through dynamic fusion with WER gating.
• Environment-aware plug-and-play routingIt features a built-in lightweight audio quality classifier (single-layer Transformer) that automatically determines whether the input audio is "clean" or "degraded". Clean audio uses the original Qwen3-ASR backbone, while degraded audio uses the Mega-ASR robust LoRA branch, achieving zero-intrusive enhancement.
• Open source dataset Voices-in-the-Wild-2MReleased a large-scale robust ASR dataset containing 2.4 million synthesized audio tracks and 5,000 evaluation audio tracks (including 1,500 real recordings), covering 7 meta-scenes and 54 mixed scenes, with the difficulty distribution calibrated through controlled experiments.

The technical principle of Mega-ASR

• Voices-in-the-Wild-2M Data ConstructionThe system employs spectrum-level code simulation, first independently simulating 7 atomic acoustic effects as a basis, and then verifying the physical rationality through an intelligent agent (such as "church = far field + echo"), combining to generate 54 composite scenes; the difficulty distribution is controlled by a unified severity parameter k∈[0,1], and unstable samples with WER>70% are removed.
• A2S-SFT Three-Stage TrainingThe first stage trains only the audio encoder and aligner, gradually expanding the training according to WER << 30%, << 50%, and << 70%. The second stage freezes the acoustic modules and fine-tunes the large language model individually to activate semantic priors. The third stage jointly trains all modules to achieve end-to-end alignment between acoustic evidence and semantic reasoning.
• DG-WGPO Dynamic Reward MechanismThe static reward consists of WER reward and anti-duplication penalty; the dynamic reward core includes token-level refinement reward (distinguishing between soft/hard replacement errors) and sentence-level reconstruction reward (based on the longest common subsequence and length penalty), which are mirrored and fused through WER gating (threshold τ=0.3). When WER is low, the focus is on local refinement, and when WER is high, the focus is on overall semantic preservation.
• Environment-aware routing modelBased on 80-dimensional log-Mel features, using a lightweight convolutional front-end + single-layer Transformer encoder + attention pooling structure, the binary classification accuracy exceeds 99.5%, with extremely low inference overhead, achieving automatic splitting of clean/degraded speech.

How to use Mega-ASR

• Environment preparation and dependency installationThe runtime environment is built based on the Qwen3-ASR open-source ecosystem. Relevant audio processing libraries and model inference dependencies (such as transformers, torchaudio, etc.) are installed to ensure support for 80-dimensional log-Mel spectrum feature extraction and model weight loading.
• Loading dual-model weightsSimultaneously load two components: the original Qwen3-ASR 1.7B backbone as the base model, and the robust LoRA weights of Mega-ASR; in addition, a lightweight environment-aware routing model (single-layer Transformer, 128 hidden dimensions, fine-tuned based on LoRA) needs to be loaded.
• Audio preprocessing is then fed into the routing model.The system extracts 80-dimensional log-Mel spectral features from the input audio, encodes them using a lightweight convolutional front-end, and then feeds them into a single-layer Transformer in the routing model for binary classification. This automatically identifies whether the current audio belongs to "clean speech" or "degraded speech" (accuracy exceeding 99.5%).
• Automatic flow reasoningWhen the routing model determines that the speech is clean, it directly calls the original Qwen3-ASR backbone for standard recognition, ensuring that the performance of clean domains such as LibriSpeech is not compromised; when it determines that the speech is degraded, it automatically switches to the Mega-ASR robust LoRA branch, activating strong noise resistance and semantic recovery capabilities in complex acoustic scenarios.
• Obtain recognition resultsThe model outputs text transcription results that can still avoid blank output or semantic illusions and maintain high integrity speech recognition results even in extreme degradation scenarios (such as far-field peak -5.2dB, NOIZEUS 0dB).

Mega-ASR's core advantages

• Leading performance in complex scenariosIn the Voices-in-the-Wild-Bench hybrid degradation scenario, the WER reached 2.73/4.57, a reduction of 65.8%/69.1% compared to Whisper-Large-v3 and a reduction of 65.8% compared to Gemini-3-Flash; under the extreme condition of NOIZEUS 0dB, the WER dropped to 19.80, a reduction of 17.4% compared to Qwen3-ASR and a reduction of 64.5% compared to Gemini-3-Flash.
• Excellent semantic recovery capabilitiesIn the extreme case of a far-field peak of -5.2dB, the Qwen3-ASR outputs blank (WER 100%), the Gemini-3-Pro produces smooth but irrelevant illusory content (WER 86.1%), while the Mega-ASR accurately recovers the reference text (WER 0.0%).
• Clean domain performance is losslessThrough environment-aware routing, the WER of clean benchmarks such as LibriSpeech has been further optimized from 1.78/3.57 to 1.63/3.37, while the original capabilities such as hot word recognition and streaming inference remain unaffected.
• High training stabilityDuring the dataset construction process, samples with a WER > 70% that are not learnable are filtered out. A2S-SFT uses a course learning strategy to gradually increase the difficulty, avoiding model crashes during training under extreme degradation conditions.
• Fully open source and reproducibleThe model weights, training code, dataset construction process, and evaluation benchmarks are all open source, based on the mature Qwen3-ASR ecosystem, making community access easy.

Mega-ASR project address

• Project Homepage: https://xzf-thu.github.io/Mega-ASR/
• GitHub repository: https://github.com/xzf-thu/Mega-ASR
• Hugging Face Model Libraryhttps://huggingface.co/zhifeixie/Mega-ASR
• arXiv technical paper: https://arxiv.org/pdf/2605.19833

Mega-ASR Comparison with Similar Products

Application scenarios of Mega-ASR

• In-vehicle voice interactionTo address complex acoustic environments such as engine noise, wind noise, and multiple conversations, and improve the accuracy of navigation and command recognition.
• Far-field meetings/classroom transcriptionIt solves problems such as echo in meeting rooms, distance attenuation, and coloration of equipment recordings, and achieves high-quality automatic minutes.
• Outdoor interviews and live broadcast captionsMaintain the integrity and accuracy of subtitle generation even under adverse conditions such as street noise, wind, and obstructions.
• Smart home voice controlImprove the robustness of wake word and command recognition in long-distance, multi-room reverberation environments.
• Telephone customer service and VoIP call analysisTo address channel degradation issues such as packet loss, bitrate compression, and electronic distortion, and ensure the quality of call content recognition.