On June 4, 2026, NVIDIA quietly dropped the weights of Nemotron 3.5 ASR on Hugging Face — a 600-million-parameter speech recognition model capable of transcribing 40 language-locale combinations from a single checkpoint. Within two weeks, a post by Paul Young on X ignited a viral firestorm, claiming the model runs on pure CPU, 2.5× faster than the official NeMo runtime, no GPU needed.
The truth is both more nuanced and more interesting.
Nemotron 3.5 ASR is a Cache-Aware FastConformer-RNNT architecture: 24 encoder layers paired with an RNNT (Recurrent Neural Network Transducer) decoder. Unlike traditional "buffered" streaming ASR that recomputes overlapping audio windows, the cache-aware design processes each audio frame exactly once and reuses cached encoder states across chunks. This eliminates redundant computation entirely.
| Dimension | Detail |
|---|---|
| Parameters | 600 million |
| Architecture | Cache-Aware FastConformer-RNNT |
| Languages | 40 locales across 3 tiers |
| Configurable Latency | 80ms, 160ms, 320ms, 560ms, 1120ms |
| License | OpenMDW-1.1 (commercial use allowed) |
| Release Date | June 4, 2026 |
| Punctuation & Capitalization | Native, all locales |
| Language Detection | Optional auto-detection mode |
Not all 40 languages are equal. NVIDIA splits them into:
| Language | WER |
|---|---|
| English (en-US) | 7.91% |
| Spanish (es-419) | 4.11% |
| Italian | 4.25% |
| Portuguese (pt-BR) | 5.48% |
| Hindi | 6.81% |
| Korean | 7.12% |
| German | 8.31% |
| French | 9.03% |
| 19-locale average | 8.84% |
Here's where the viral narrative went wrong. The 40-language multilingual Nemotron 3.5 ASR requires a GPU. The official model card lists GPU as a requirement, with support from Volta through Blackwell architectures, plus Jetson edge modules.
CPU execution comes from a completely separate effort.
In April 2026, a Microsoft CoreAI research team led by Nenad Banfic published a paper on arXiv (2604.14493) titled "Pushing the Limits of On-Device Streaming ASR." They took the English-only sibling model — nemotron-speech-streaming-en-0.6b — and reimplemented the entire streaming pipeline inside ONNX Runtime.
The results:
| Variant | Size | Avg WER (8 benchmarks) | RTFx on CPU |
|---|---|---|---|
| FP32 PyTorch baseline | 2.47 GB | 8.03% | — |
| ONNX INT4 k-quant | 0.67 GB | 8.20% | > 6× real-time |
That's right — the 4-bit quantized version loses just 0.17 percentage points of WER while shrinking to 27% of the original size and running 6× faster than real-time on a server CPU (AMD EPYC 7V12, 32 cores).
The viral "2.5× faster than NeMo runtime" figure does not appear in the paper. The real metric is RTFx > 6 (i.e., transcribing one hour of audio in under 10 minutes on CPU).
| Scenario | Model | Hardware | Reality |
|---|---|---|---|
| Multilingual, 40 languages | nemotron-3.5-asr-streaming-0.6b |
GPU required | Official NVIDIA |
| English, CPU-only | nemotron-speech-streaming-en-0.6b + ONNX INT4 |
CPU (any x86_64) | Microsoft paper |
| Multilingual, ONNX INT4 | onnx-community/nemotron-3.5-asr-streaming-0.6b-onnx-int4 |
CPU (via ONNX Runtime) | Community port |
Important: The ONNX Community has also quantized the multilingual model to INT4, but this is a separate effort from the Microsoft paper. The Microsoft paper's rigorous benchmarking (RTFx > 6, 8.20% WER) applies only to English.
| Model | Avg WER | Size | Native Streaming? |
|---|---|---|---|
| Qwen3-ASR-1.7B | 5.90% | 4.70 GB | No (chunked degrades) |
| Parakeet TDT-0.6B-v3 | 6.32% | 2.51 GB | No (chunked degrades) |
| Nemotron-0.6B | 7.07% | 2.47 GB | Yes (0.21% degradation) |
| Canary-1B-v2 | 7.15% | 6.36 GB | No |
| Whisper-v3-Turbo | 7.83% | 1.62 GB | No |
| Whisper Small.en | 8.59% | 0.97 GB | Chunked (degrades) |
The killer stat: When moving from batch to streaming mode, Nemotron loses only 0.21% absolute WER (7.07% → 7.28%). By comparison, Qwen3-ASR-1.7B degrades from 5.90% to 10.45% when chunked. Whisper models aren't even designed for streaming — the chunked workaround causes a 3.5% absolute WER regression at 10-second chunk sizes.
Tested on NVIDIA L4 (23 GB), LibriSpeech test-clean + test-other:
| Model | Best Throughput | Best WER | Notes |
|---|---|---|---|
| Nemotron 0.6B | 258× real-time | 10.30%* | WER unchanged across all chunk sizes |
| Parakeet TDT 0.6B | 238× real-time | 15.72%* | Beam search hurts accuracy |
| Whisper v3-turbo | 40.4× real-time | 8.93% | SDPA 1.8× faster than eager |
*NeMo models output punctuated text vs. LibriSpeech plain-text references — inflates NeMo WER by ~2-3% absolute. Cross-model WER comparisons should account for this.
| Model | Latency | vs Whisper | Hardware |
|---|---|---|---|
| Nemotron 600M | 43ms | 21× faster | L40S GPU |
| Deepgram Nova-2 | 272ms | 3.4× faster | Cloud API |
| Whisper medium | 916ms | baseline | M-series CPU |
Steve's AMD Ryzen AI 9 HX 370 mini PC (Strix Point) is an interesting target. Here's the current state of play:
whisper.cpp via AMD Lemonade: AMD's Lemonade local AI server (v10.3, May 2026) bundles whisper.cpp for speech-to-text. On Ryzen AI 300-series hardware with XDNA 2 NPU (50 TOPS), Whisper can offload encoder inference to the NPU for a significant speedup versus CPU-only.
Key Lemonade specs for the HX 370:
No Nemotron integration in Lemonade. As of June 2026, Lemonade's STT backend is whisper.cpp only. There is no Nemotron support path — neither the GPU NeMo path (which requires CUDA) nor the ONNX CPU path.
The most viable way to run Nemotron on the AMD mini PC would be:
ONNX Runtime CPU path: Use the onnx-community/nemotron-3.5-asr-streaming-0.6b-onnx-int4 model with ONNX Runtime on CPU. The Ryzen AI 9 HX 370's Zen 5 cores (12C/24T) should deliver solid RTFx, though likely lower than the EPYC 7V12 server chip used in Microsoft's paper.
ROCm GPU path: The Radeon 890M iGPU (RDNA 3.5, 16 CUs) could theoretically run the NeMo model via ROCm, but NeMo's primary target is CUDA. This path would require significant porting effort.
NPU acceleration: No path exists yet. XDNA 2 NPU support for transformer-based ASR models would require a dedicated runtime similar to FastFlowLM.
On a single H100 GPU, Nemotron 3.5 ASR sustains:
| Chunk Size | Nemotron 3.5 (0.6B) | Parakeet RNNT (1.1B) | Improvement |
|---|---|---|---|
| 80ms | 240 streams | 14 streams | 17× |
| 160ms | 480 streams | ~80 streams | ~6× |
| 320ms | 800 streams | ~180 streams | ~4.4× |
| 1120ms | 2,400 streams | 400 streams | 6× |
For a production voice agent service, this means one H100 can handle thousands of simultaneous conversations — a dramatic reduction in per-user infrastructure cost.
Nemotron 3.5 ASR lands at a pivotal moment for three reasons:
1. Voice Agents Are Exploding. Every AI platform is racing toward real-time voice interaction. A model that returns final transcription in under 100ms — while the user is still speaking — eliminates the awkward pause that makes voice agents feel robotic.
2. Privacy-Preserving ASR Is Now Feasible. The ONNX INT4 variant at 0.67 GB means you can run state-of-the-art English transcription entirely on-device. No audio leaves the machine. For healthcare, legal, and enterprise use cases, this is a game-changer.
3. Open Weights + Commercial License. OpenMDW-1.1 permits commercial use. You can fine-tune on domain data, deploy behind your firewall, and build products on top — no API keys, no per-request pricing, no vendor lock-in.
The viral narrative — "40 languages, pure CPU, 2.5× faster" — conflates three separate achievements:
No single configuration achieves all three simultaneously — at least not yet. But the pieces are all in place, and the trajectory is clear: streaming ASR is moving from the cloud to the edge, and Nemotron's cache-aware architecture is leading the charge.
Pushing the Limits of On-Device Streaming ASR — Microsoft CoreAI (arXiv:2604.14493)
Nemotron 3.5 ASR: NVIDIA Transcribes 40 Languages in Real Time — Pasquale Pillitteri
Benchmarking Open ASR Models on NVIDIA L4: Parakeet vs Whisper vs Nemotron — E2E Networks
How NVIDIA Nemotron 3.5 Compares to Whisper for Streaming — Geeky Gadgets
QbitLoop RealtimeVoice: ASR Benchmark — Nemotron 21× Faster than Whisper
Nemotron vs Whisper Large V3: 5 Audio Transcription Tests — Wiro AI
AMD Lemonade Local LLM Server: GPU + NPU Inference Guide (2026)
Whisper.cpp NPU Acceleration on Ryzen AI — AMD Documentation
NVIDIA Nemotron Speech ASR: Scaling Real-Time Voice Agents — HuggingFace Blog
