Published: Aug 19, 2026 | Reading Time: ~9 minutes | Channel: techminute
Here's a number that should make every AI engineer sit up: ~670 tokens per second. That's the median output speed Artificial Analysis measured on a pre-release endpoint serving NVIDIA's new Nemotron 3.5 Lightning — roughly 6–7× faster than the open-weight peers it competes with, at a fraction of the token cost.
But the speed isn't actually the story. The story is what Jensen Huang's crew is selling now: not just the model, but the switchyard to route your agent traffic past the expensive models entirely.
NVIDIA announced Nemotron 3.5 Lightning on August 11, 2026 — a 30-billion-parameter mixture-of-experts model with only 3 billion active parameters per token — alongside NeMo Switchyard, an open-source model-routing library. Two pieces of one puzzle: a cheap, fast "worker bee" model designed for the high-volume grunt work inside always-on AI agents (tool calls, result validation, subagent delegation, formatting), and a router that decides when to send work to that cheap model versus a frontier reasoning model like Nemotron 3 Ultra or GPT-5.6.
This is NVIDIA's most interesting move since the Rubin GPU launch. Not because of any single benchmark — but because it's the first time the chipmaker has publicly bet on open-weight software as a growth driver for its hardware business. Jensen said it himself in July: "Free AI should be great for hardware. Free AI should be great for chips."
Let's dig into what this actually changes.
NVIDIA isn't new to software — CUDA is arguably the most successful developer platform ever shipped. But for years, the company's model strategy was "CUDA + cuDNN + TensorRT": a software moat that makes NVIDIA hardware the only sane place to run AI. Models themselves? Those were someone else's problem.
That posture cracked in 2026. Three forces:
The open-weights wave. Alibaba's Qwen3.8-Max (2.4T total parameters) went open; Moonshot's Kimi K3 rattled Washington; Meta's Muse Glimmer brought 30B-class models to laptops. Every open model that runs well on commodity hardware could reduce demand for $40,000 data-center accelerators — unless the open model happens to run especially well on NVIDIA's stack.
China. The Kimi K3 distillation panic triggered real D.C. hand-wringing about open Chinese models. Huang, who normally avoids policy, posted his first-ever X post in late July arguing for open weights: "Open models strengthen safety and cybersecurity, accelerate innovation and diffusion, and enable sovereignty."
The agent economy arrived. Agent-native apps burn tokens in quantities that would have seemed absurd two years ago — and they don't care which model is "best" on a leaderboard; they care about how much a completed task costs. That's a cost-per-task problem. And cost-per-task is a routing problem. Routing is a software problem.
Nemotron 3.5 Lightning + Switchyard is NVIDIA's answer to all three at once.
"30B total, only ~3B active per token" — what does that actually mean?
MoE = mixture-of-experts. The model holds 30 billion parameters, but a learned router activates only a small fraction (on the order of 3B) per token. Think of it like a consultancy with 30,000 staff where, for any given question, only a few hundred specialists do the work — everyone else stays idle. You get the depth of a 30B model at a fraction of the compute per token.
That's the trick NVIDIA has been refining across the Nemotron 3 family. Lightning is the smallest member, but it inherits the family's best training techniques:
The positioning is deliberate: Lightning is the execution layer of an agent system. Frontier models like Nemotron 3 Ultra plan and orchestrate; Lightning does the boring, high-volume work — git pulls, tool output validation, formatting, routine calls — that dominates an agent's token budget.
That's why NVIDIA is also shipping all the surrounding tooling: NeMo for fine-tuning, a new open agentic RL dataset (Nemotron-RL-Agentic-Terminal-Pivot) used for its coding-agent training, and NeMoClaw for running always-on agents securely.
Numbers are the whole story here, so here's the honest scorecard. Independent measurements come from Artificial Analysis; NVIDIA-published figures are flagged as such.
| Metric | Nemotron 3.5 Lightning | Nemotron 3 Nano (prev gen) | Change |
|---|---|---|---|
| AA Intelligence Index (independent) | 24 | 15 | +9 |
| Median output speed (AA, pre-release endpoint) | ~670 tok/s | — | ~6–7× vs 30B-class peers |
| Time per Intelligence Index task (AA) | ~0.5 min | — | vs Qwen3.6 35B ~3.5 min |
| PinchBench accuracy (NVIDIA-reported) | 86% | — | — |
| PinchBench 10k-task completion (NVIDIA-reported) | 30% faster | — | vs Qwen3.6 35B |
| Terminal-Bench v2.1 (AA) | 24% | 7% | >3× |
| GDPval-AA v2 Elo (AA) | 824 | — | surpasses Nemotron 3 Super & gpt-oss-120b |
| SWE-bench Resolved (HF-reported) | 51.56 | — | strong for a 30B |
| MMLU-Pro (HF-reported) | 81.94 | — | strong for a 30B |
Read that table carefully and you'll spot an oddity: Lightning scores behind Qwen3.6 35B (AA Index 32) and Meta's Muse Glimmer (AA Index 35). It is not the smartest small model on the market.
But NVIDIA's pitch — validated by Artificial Analysis — is that it's the fastest, most cost-efficient in its class, and that's what matters in the agent economy. Per Artificial Analysis: the model "is built for a different point on the frontier" — the execution layer. Measure cost per completed task instead of leaderboard score, and Lightning flips the conversation.
And it's cheap. On OpenRouter, serving is free today; the paid tier runs $0.08 per million input tokens, $0.20 per million output — notably below typical 30B-class pricing.
Now the juicier claim. NeMo Switchyard is an Apache-2.0 licensed, pre-alpha Rust proxy + library that sits between your agent and behind-the-scenes models. Two capabilities:
Protocol translation. Your agent speaks native OpenAI Chat / Anthropic Messages / OpenAI Responses — Switchyard translates each request into whatever backend you point it at (vLLM, NIM, Ollama, any OpenAI-compatible endpoint), then translates the response back to the client's native format. Literally: point Claude Code or Codex at an open model and it just works.
Per-request routing. Switchyard decides, per request, which model should handle it — based on your cost/latency/quality priorities. Routing strategies shipped:
The vision: a "system of models," where a frontier brain (Nemotron 3 Ultra, GPT-5.6) handles orchestration and hard reasoning, and worker bees like Lightning handle everything else at a fraction of the cost. Switchyard is the switchboard making that division of labor automatic.
The partner results NVIDIA published are head-turning — treat them as vendor-reported:
Spot the pattern: a ~60% cost reduction on a workflow isn't an optimization anymore; it's a new pricing tier.
The "cheap worker + expensive planner + automatic router" template is the direction everyone is racing toward — OpenClaw (early agent harness), LangChain, Microsoft's Azure AI Foundry, Google's Agent Enterprise. But NVIDIA's entry is unique: it's the only vendor that owns the hardware AND the models AND the router.
For AWS / Azure / Google Cloud: NVIDIA's open-weights strategy is a vector for chip demand. Models that run beautifully on NVIDIA GPUs keep GPU minutes growing even as API prices fall.
For OpenAI / Anthropic: another downward pressure on frontier pricing. If 70–80% of traffic can route to a $0.20/M-output open model, high-margin frontier pricing only survives on the small share of super-hard calls that truly need it.
For developers: cost per agent task just plunged. You no longer need to choose "frontier for everything" or "cheap for everything" — routing makes the choice per-call, with a switchable algorithm.
Most importantly — the platform play. Weights + Switchyard + NeMo + NeMoClaw = a full agent stack inside NVIDIA's own developer ecosystem. If developers adopt Switchyard as their default router, NVIDIA becomes the choke point of agent economics — an "Intel Inside" moment for AI agents, except this time the sticker is on the software.
Switchyard is pre-alpha. The GitHub README is explicit: "Experimental software. Not for production use." APIs and algorithms "are expected to change significantly before we reach v1.0." Adopt it today as a prototype, and expect config churn.
"30B" vs "31.6B" — pick your source. NVIDIA says 30B total/3B active; Artificial Analysis measured 31.6B total/3.6B active. Same model, different counting conventions. Charts that say both aren't contradicting themselves.
Partner cost numbers are NVIDIA-published, not independently audited. Ramp's 58%, Cognition's 28%, LangChain's 74% — no neutral arbitrator signed off. Directional, not gospel.
~670 tok/s was a pre-release endpoint measurement. Benchmark hardware in the wild may vary. It's the best independent snapshot we have, but it's a snapshot.
It's not the smartest 30B. Qwen3.6 35B and Meta's Muse Glimmer still out-rank it on head-to-head intelligence indices. Lightning wins on the time-per-task and cost-per-task frontier, not the intelligence-per-download frontier.
Vendor "systems of models" politics. If you route through Switchyard, you're likely also buying into NVIDIA's model/host ecosystem. It's Apache-2.0, so you can mix competitors — but the default path funnels traffic toward NVIDIA-adjacent targets.
NVIDIA shipped a model that wins on economics, a router that bakes those economics into your workload, and the hardware to run both — all in one week. The agent economy is becoming a cost-per-task game, and NVIDIA just put its thumb on the scale.
If you're building agents in 2026, the end-state design is increasingly clear: small, fast "worker bees" + a big occasional planner + a router that knows when to escalate. NVIDIA hands you the worker, the router, the fine-tuning tools — and the GPUs.
That's a strategy that works whether or not Lightning wins its class. The cheaper your workers are, the better the whole stack performs. Which, given Jensen's hardware-first worldview, is precisely the point.
All claims verified against Gold-tier (NVIDIA official blogs, Hugging Face model card, GitHub repository) and Silver-tier (CNBC) sources, plus independent measurements from Artificial Analysis and community-sourced hands-on reports (Hacker News, Reddit) labeled as such. Vendor benchmark figures are flagged as NVIDIA-reported. Last verified: 2026-08-19.
