Nemotron Speech ASR - NVIDIA's open-source speech recognition model

What is Nemotron Speech ASR?

Nemotron Speech ASR is an open-source model from NVIDIA focused on low-latency, real-time streaming speech recognition. Through a cache-aware architecture, it caches processed speech features and only computes them for new audio frames, achieving single-sentence transcription locking in just 24 milliseconds. This effectively solves the cumulative latency problem of traditional streaming models in long speech recognition. The model supports multiple latency modes (80ms, 160ms, 560ms, 1.12s), which can be flexibly adjusted according to application scenarios without retraining. It is suitable for various scenarios such as game voice, real-time translation, and meeting recording. It boasts higher throughput and lower operating costs, with end-to-end latency controlled within 500 milliseconds, and natively supports punctuation and case sensitivity.

Main functions of Nemotron Speech ASR

• Low-latency real-time recognitionDesigned specifically for low-latency, real-time streaming scenarios, single-sentence transcription lock takes only 24 milliseconds, almost equivalent to the human neural response speed, making it suitable for voice interaction scenarios with extremely high real-time requirements.
• Cache-aware architectureIt adopts a cache-aware design, directly caching processed speech features and only calculating the incremental part of new audio frames, avoiding repeated calculations and effectively solving the problem of cumulative delay in long speech recognition.
• Multiple delay modesIt supports multiple latency modes such as 80ms, 160ms, 560ms, and 1.12s, which can be flexibly adjusted according to different application scenarios without retraining the model, meeting diverse needs from extreme speed to high accuracy.
• High throughput and low operating costsCompared to traditional streaming models, it offers higher throughput, can handle more parallel streams with the same GPU memory constraints, and significantly reduces operating costs in production environments.
• End-to-end low latencyThe end-to-end latency is controlled within 500 milliseconds, ensuring the smoothness and immediacy of voice interaction.
• Native support for punctuation and capitalizationThe model natively supports punctuation and capitalization, improving the readability and usability of the recognition results.
• Integrated Voice Intelligent Agent SolutionThe Nemotron Speech ASR is not an isolated model, but is integrated into a complete speech agent solution. It works in conjunction with the Nemotron 3 Nano 30B (LLM) and Magpie (TTS) to provide comprehensive support for building true speech agents.

The technical principle of Nemotron Speech ASR

• Cache-aware designBy maintaining an encoder state cache, processed audio features are stored. When a new audio frame arrives, the cache is directly called to calculate only the current increment, avoiding redundant calculations and thus achieving real-time processing with extremely low latency.
• Incremental calculation mechanismUnlike traditional streaming models, Nemotron Speech ASR does not re-encode historical data. Instead, it performs incremental calculations based on cached activation values, effectively solving the problem of cumulative latency in long speech recognition.
• Dynamic delay adjustmentIt supports multiple latency modes (such as 80ms, 160ms, 560ms, 1.12s), allowing users to flexibly adjust the latency during the inference phase without retraining the model, thus adapting to the latency requirements of different scenarios.
• High-efficiency parallel processingWith an optimized architecture, it can process more parallel streams under the same GPU memory constraints, significantly improving throughput and reducing operating costs in production environments.
• End-to-end optimizationThe entire process from audio input to text output has been optimized to ensure that the end-to-end latency is controlled within 500 milliseconds, meeting the needs of real-time voice interaction.
• Context-aware decoding: Through configurable context size parameters (e.g.att_context_sizeThe model dynamically adjusts its use of contextual information to further optimize the balance between recognition accuracy and latency.

Nemotron Speech ASR project address

• Github repositoryhttps://github.com/NVIDIA-NeMo/NeMo
• HuggingFace model libraryhttps://huggingface.co/nvidia/nemotron-speech-streaming-en-0.6b

Application scenarios of Nemotron Speech ASR

• Real-time voice assistantIt provides low-latency voice recognition capabilities for intelligent voice assistants, enabling instant responses to user voice commands and enhancing the interactive experience.
• Game voice interactionIn game scenarios, it supports low-latency voice chat and command recognition, enhancing real-time communication and interaction between players.
• Real-time translationIt is used for real-time speech translation in multilingual environments, quickly converting speech content in one language into text or speech in another language, thus promoting cross-language communication.
• Meeting minutesIn meeting scenarios, it provides high-precision speech-to-text functionality, generating meeting minutes in real time and improving meeting efficiency.
• Live InteractionDuring the live stream, real-time subtitles are provided to enhance the audience's sense of participation and interactive experience.
• Customer service systemIn customer service, quickly identify user voice problems and provide immediate responses to improve customer service efficiency and user satisfaction.