NVIDIA Nemotron-Speech-Streaming-En-0.6b: A Powerful Model for Real-Time Speech-to-Text

The Nemotron-Speech-Streaming-En-0.6b is NVIDIA’s 600M-parameter English automatic speech recognition (ASR) model, designed for high-quality transcription in both low-latency streaming and high-throughput batch scenarios. It features a native cache-aware streaming architecture, supports punctuation and capitalization out of the box, and allows runtime flexibility with chunk sizes from 80ms to 1120ms, achieving average Word Error Rates (WER) between 7.16% and 8.53%.

If you’re building applications like voice assistants, live captioning, or conversational AI, you’ve probably faced a common challenge: how to achieve fast, responsive speech-to-text without sacrificing accuracy. Many traditional ASR models force a trade-off—either high latency for better results or quick responses with more errors. That’s where NVIDIA’s Nemotron-Speech-Streaming-En-0.6b stands out. This unified model handles both real-time streaming and batch processing exceptionally well, making it a go-to choice for production voice agents.

Why Choose Nemotron-Speech-Streaming-En-0.6b?

Developers often ask: What makes this model different from others? Here are the key advantages:

• Native Streaming Support — Its cache-aware design is optimized for continuous audio streams, ideal for low-latency applications like interactive voice agents.
• Better Efficiency — It delivers higher throughput than traditional buffered streaming methods, allowing more concurrent streams on the same GPU and lowering operational costs.
• Runtime Flexibility — Switch between latency and accuracy on the fly—no retraining needed. Perfect for adapting to different use cases.
• Built-in Punctuation and Capitalization — Outputs polished text directly, saving you from extra post-processing steps.

This flexibility lets you navigate the latency-accuracy trade-off seamlessly, outperforming many production models across various chunk sizes.

How Does the Architecture Work?

At its core, the model uses a Cache-Aware FastConformer encoder with an RNN-T (Recurrent Neural Network Transducer) decoder.

• Encoder → 24 layers of Cache-Aware FastConformer.
• Decoder → RNN-T.
• Total Parameters → 600 million.

Traditional streaming ASR relies on buffered approaches, overlapping audio chunks to maintain context—this leads to redundant computations and higher latency. In contrast, the cache-aware mechanism maintains caches for all self-attention and convolution layers in the encoder. When processing a new chunk, it reuses hidden states from previous frames, computing only the new audio without overlaps.

Think of it like listening to a conversation: you don’t replay old parts to understand the new ones; you just build on what you already know. This eliminates wasted computation and enables truly efficient real-time transcription.

Adjusting Latency: Chunk Sizes and Context Settings Explained

One of the most practical features is how easily you can control latency. It all comes down to the att_context_size parameter, which sets left and right context in 80ms frames:

• [70, 0] → Chunk size 1 (80ms) – Ultra-low latency.
• [70, 1] → Chunk size 2 (160ms).
• [70, 6] → Chunk size 7 (560ms).
• [70, 13] → Chunk size 14 (1120ms) – Highest accuracy.

Chunk size = current frame + right context frames. Processing is strictly non-overlapping, maximizing efficiency.

No model retraining required—just tweak this at inference time to match your needs, whether it’s snappy responses for a voice assistant or precise transcripts for meetings.

Performance Breakdown: Word Error Rates Across Chunk Sizes

Accuracy is measured by Word Error Rate (WER)—lower is better. Here are results from standard Hugging Face OpenASR leaderboard datasets (without additional punctuation processing):

1120ms Chunk Size (Highest Accuracy)

Dataset	WER (%)
Average	7.16
AMI	11.58
Earnings22	12.48
Gigaspeech	11.45
LibriSpeech test-clean	2.31
LibriSpeech test-other	4.75
SPGI	2.62
TEDLIUM	4.50
VoxPopuli	7.57

560ms Chunk Size

Dataset	WER (%)
Average	7.22
AMI	11.69
Earnings22	12.61
Gigaspeech	11.43
LibriSpeech test-clean	2.40
LibriSpeech test-other	4.97
SPGI	2.64
TEDLIUM	4.46
VoxPopuli	7.59

160ms Chunk Size

Dataset	WER (%)
Average	7.84
AMI	13.88
Earnings22	13.61
Gigaspeech	12.12
LibriSpeech test-clean	2.43
LibriSpeech test-other	5.33
SPGI	2.82
TEDLIUM	4.80
VoxPopuli	7.72

80ms Chunk Size (Lowest Latency)

Dataset	WER (%)
Average	8.53
AMI	16.05
Earnings22	14.60
Gigaspeech	12.92
LibriSpeech test-clean	2.55
LibriSpeech test-other	5.79
SPGI	3.01
TEDLIUM	5.07
VoxPopuli	8.23

Even at the fastest 80ms setting, the average WER stays competitive, especially on clean speech datasets like LibriSpeech test-clean (just 2.55%).

Getting Started: How to Use the Model

The easiest way to work with this model is through the NVIDIA NeMo framework, which handles training, fine-tuning, and inference.

Installing NeMo

Start by installing dependencies:

apt-get update && apt-get install -y libsndfile1 ffmpeg
pip install Cython packaging
pip install git+https://github.com/NVIDIA/NeMo.git@main#egg=nemo_toolkit[asr]

Loading the Model

In Python:

import nemo.collections.asr as nemo_asr
asr_model = nemo_asr.models.ASRModel.from_pretrained(model_name="nvidia/nemotron-speech-streaming-en-0.6b")

Running Streaming Inference

Use NeMo’s dedicated script for cache-aware streaming:

python examples/asr/asr_cache_aware_streaming/speech_to_text_cache_aware_streaming_infer.py \
    model_path=<your_model_path> \
    dataset_manifest=<dataset_manifest> \
    batch_size=<batch_size> \
    att_context_size="[70,13]" \  # Adjust right context: 0, 1, 6, or 13
    output_path=<output_folder>

For a full pipeline with punctuation and capitalization:

from nemo.collections.asr.inference.factory.pipeline_builder import PipelineBuilder
from omegaconf import OmegaConf

cfg = OmegaConf.load('cache_aware_rnnt.yaml')  # Load the config file
audios = ['/path/to/audio1.wav', '/path/to/audio2.wav']

pipeline = PipelineBuilder.build_pipeline(cfg)
output = pipeline.run(audios)

for entry in output:
    print(entry['text'])

Input Requirements: Mono-channel 16kHz audio, minimum 80ms duration.
Output: English text with punctuation and capitalization (may be empty if no speech detected).

Training and Evaluation Datasets

The model was trained on approximately 285,000 hours of English audio, primarily from the Granary dataset:

• YouTube-Commons (109.5k hours)
• YODAS2 (102k hours)
• Mosel (14k hours)
• LibriLight (49.5k hours)

Plus classics like LibriSpeech (960 hours), Fisher, Switchboard, and others including Mozilla Common Voice and VoxPopuli.

Evaluations used diverse sets like AMI, Earnings22, Gigaspeech, LibriSpeech, TEDLIUM, and more for reliable benchmarking.

FAQ: Common Questions Answered

Does this model support languages other than English?
No, it’s focused on English only.

What hardware does it run on?
Compatible with NVIDIA Ampere, Blackwell, Hopper, and Volta architectures. Tested on V100, A100, and others.

Why does it include punctuation and capitalization natively?
It’s baked into the training, so no extra modules needed.

How much better is the throughput?
It supports more parallel streams in the same memory footprint, improving efficiency significantly.

What if my audio is very short?
It needs at least 80ms to process.

Is it suitable for commercial use?
Yes, fully open for commercial and non-commercial applications.

The Nemotron-Speech-Streaming-En-0.6b strikes an impressive balance between speed and accuracy, making it an excellent foundation for modern voice applications. If you’re working on real-time speech-to-text, give it a try—you’ll see how it simplifies deployment while delivering robust performance.