Commissioned, Curated and Published by Russ. Researched and written with AI.
What’s New
Nemotron 3 Nano shipped this week alongside training datasets, RL environments, and a full open data stack. Separately, Ai2 released OLMo 3 Hybrid – also a Mamba-Transformer architecture – within days of Nemotron. Two independent teams converging on the same design is a signal worth noting.
Changelog
| Date | Summary |
|---|---|
| 10 Mar 2026 | Initial publication. |
The core tension in multi-agent systems is arithmetic. You want many agents running concurrently, each calling a model, each maintaining context over a multi-step task. Standard transformer inference scales quadratically with context length and linearly with the number of concurrent requests. Run twenty agents at once, each with a 50k-token context window, and you’re paying for a lot of compute that isn’t doing useful work.
Nemotron 3 Nano is NVIDIA’s answer to that arithmetic. The architectural choices – hybrid Mamba-Transformer, sparse MoE, 1M-token context window, NVFP4 4-bit precision throughout – aren’t incidental. They’re a coherent set of decisions aimed at making agent-scale inference viable. Let’s work through what each of them actually means.
MoE Economics: 31.6B Parameters, 3.6B Active
Mixture-of-Experts is not a new idea, but the numbers in Nemotron 3 Nano illustrate why it matters specifically for agent fleets.
The model has 31.6B total parameters distributed across 128 experts. On each forward pass, an MLP router activates 6 of those experts – meaning roughly 3.6B parameters are doing actual work per token. You get the knowledge density of a 30B model while paying inference compute at closer to a 3-4B model rate.
For a single request, that’s a nice efficiency win. For a fleet of a hundred concurrent agents, it’s the difference between affordable and unaffordable infrastructure.
The throughput numbers back this up. Nemotron 3 Nano achieves 4x higher token throughput than Nemotron 2 Nano, and 3.3x higher throughput than comparable-capability models in its size class (tested in an 8K input / 16K output configuration on a single H200). In multi-agent systems, where you’re optimising for total tokens per second per dollar rather than latency for a single request, that multiplier matters a lot.
There’s also a reasoning efficiency gain. Nemotron 3 Nano reduces reasoning-token generation by up to 60% compared to previous models – meaning it reaches the same answer with fewer intermediate thinking tokens. Less waste at inference time, across every agent invocation.
The model supports reasoning ON/OFF modes plus a configurable thinking budget. That’s a practical lever for agent systems: you can cap thinking tokens per call to keep costs predictable, rather than letting the model spend arbitrarily on chain-of-thought.
Super and Ultra (due H1 2026) follow the same MoE logic at larger scales. Super runs ~100B total parameters with ~10B active per token. Ultra runs ~500B total with ~50B active. The active/total ratio holds roughly constant, which means the compute cost scales much more gently than the parameter count suggests.
The Mamba-Transformer Hybrid: Linear Context Scaling
Standard transformer attention is O(n²) with sequence length. At 50k tokens, that’s 2.5 billion attention computations per layer per forward pass. At 200k tokens, it’s 40 billion. This is why long-context transformers are expensive, and why most deployed models with 128k+ context windows are using architectural tricks to get there.
Mamba is a state space model that processes sequences via linear recurrence. Its computational cost scales O(n) with sequence length – linearly rather than quadratically. The tradeoff is that Mamba handles local, sequential patterns well but is weaker on tasks requiring precise, arbitrary attention over distant tokens.
The hybrid Mamba-Transformer approach in Nemotron 3 Nano interleaves Mamba-2 layers (for efficient long-context processing) with grouped-query attention transformer layers (for fine-grained reasoning where attention is needed). The architecture uses both mechanisms where each is strong.
Nemotron 3 Nano supports a 1M-token context window. That’s not a marketing number – it was achieved by a continued pre-training stage at 512k sequence length, with synthetic data designed specifically for long-range retrieval, multi-hop reasoning, and multi-document information aggregation.
For agentic tasks, this matters directly. An agent managing a long-running workflow – reading code, calling tools, tracking state across dozens of steps – needs to keep that context available without it becoming prohibitively expensive. A 1M-token window at near-linear compute cost is a fundamentally different capability profile than a 128k window at quadratic cost.
This is also the same architectural bet Ai2 made with OLMo 3 Hybrid, released within days of Nemotron. Two teams, working independently, arriving at hybrid Mamba-Transformer as the right design for efficient long-context models. That kind of convergence is a meaningful signal about where model architecture is heading. It’s not coincidence.
The Task Routing Pattern
Perplexity’s deployment of Nemotron 3 describes exactly the architecture most production agentic systems should be building toward.
From Aravind Srinivas, Perplexity CEO: “With our agent router, we can direct workloads to the best fine-tuned open models, like Nemotron 3 Ultra, or leverage leading proprietary models when tasks benefit from their unique capabilities – ensuring our AI assistants operate with exceptional speed, efficiency and scale.”
The pattern here is a two-tier model stack. Efficient, specialised, open models (Nemotron) handle well-defined subtasks: summarisation, classification, retrieval, structured extraction, code generation within known patterns. Frontier models (GPT-4 class, Claude, Gemini) handle tasks requiring deep reasoning, novel generalisation, or ambiguous multi-step planning where capability trumps cost.
Most production agentic systems that are taking inference costs seriously are building toward this pattern, even if they haven’t formalised it. The question is always: which calls actually require frontier-model capability, and which don’t? Routing the latter to a cheaper, faster, open model cuts costs dramatically.
Nemotron 3 Nano is specifically designed to be the efficient tier in this pattern. It’s built for coding, content summarisation, information retrieval, and structured agentic workflows. Those are the workloads that make up the majority of tokens in most agent pipelines – not the hard reasoning, but the mechanical application of capability at scale.
The routing decision is the interesting engineering problem. What you need from the efficient tier is: strong accuracy on well-scoped tasks, fast response, predictable cost, and the ability to fine-tune for your specific domain. Nemotron 3 delivers all four, and the open weights mean you can specialise it.
Full Stack Openness
There’s a meaningful distinction between releasing model weights and releasing a training stack.
With weights alone, you can run the model, evaluate it, fine-tune it from a checkpoint. You can’t audit the training data, replicate the process from scratch, or understand why the model behaves the way it does at a deep level.
NVIDIA released: model weights, 3T new pre-training tokens (including datasets on HuggingFace), 13M post-training samples, 10+ RL environments covering 900k+ tasks across math, coding, reasoning, and tool-use, and ~11k agent-safety traces.
That’s a full open stack. You can reproduce the training process. You can audit what the model was trained on. You can modify the RL environments for your specific domain and fine-tune from there, not just from a supervised fine-tuning checkpoint.
This matters for sovereign AI deployments – organisations that need to verify exactly what their models were trained on and how, for regulatory or security reasons. The NVIDIA newsroom specifically names European and South Korean early adopters who are deploying Nemotron in this context. Those deployments aren’t possible with a weights-only release.
For most engineering teams, the practical implications are about customisation. Concurrent multi-environment reinforcement learning at scale produced Nemotron’s reasoning capabilities. With the RL environments released openly, you can apply those same techniques to your domain. You’re not starting from scratch – you’re starting from a production-proven RL setup and adapting it.
This is also a competitive positioning decision for NVIDIA. Making the training stack genuinely open creates an ecosystem around Nemotron in a way that weights-only release does not. Early adopters like Cursor, CrowdStrike, and ServiceNow aren’t just using the base model – they’re building on the open stack.
Self-Hosting Viability
Nemotron 3 Nano has 31.6B total parameters with 3.6B active at inference. The active parameter count is what determines memory bandwidth requirements during generation. But you still need to store all 31.6B parameters in GPU memory.
At BF16 precision, 31.6B parameters occupy approximately 63GB of VRAM. That fits on an A100 80GB with room for KV cache. At 4-bit quantisation (using the NVFP4 format Nemotron was trained with), it drops to roughly 16GB – two RTX 4090s at 24GB each gives you 48GB combined, which is enough.
Single-GPU deployment: A100 80GB or H100 80GB in BF16 is the clean path. Reasonable throughput for agent workloads.
Dual-GPU desktop: Two RTX 4090s with Nemotron quantised to 4-bit. This is actually a viable self-hosting configuration for a well-funded team that doesn’t want cloud inference costs. Quantised to the same NVFP4 format used in training – not a post-hoc accuracy compromise.
NVIDIA NIM microservice is the deployment layer. NIM packages the model with an optimised inference runtime, exposes an OpenAI-compatible API endpoint, and handles batching, KV cache management, and speculative decoding automatically. For teams that don’t want to manage vLLM or TGI directly, NIM is the path from hardware to production endpoint.
The model also runs via vLLM and SGLang directly if you prefer the open-source stack.
Self-hosting a production agentic model at this capability level – 30B parameters, 1M context, state-of-the-art reasoning – is no longer a research project or a task that requires a dedicated ML infrastructure team. It’s a two-GPU machine with NIM or vLLM. That’s a significant change from where we were twelve months ago.
Where This Is Going
Two independent teams shipped hybrid Mamba-Transformer models in the same week. NVIDIA with Nemotron 3 Nano, Ai2 with OLMo 3 Hybrid. Different training approaches, different scales, different organisations – same fundamental architectural choice.
Hybrid Mamba-Transformer is not a marginal optimisation. It’s a different approach to the quadratic attention problem that becomes more acute exactly as context windows grow and agent systems become more ambitious. The convergence suggests this is the direction the field has landed on, not an experiment.
Nemotron 3 Nano – now available, with Super and Ultra following in H1 2026 – is the first fully open implementation of this architecture at production quality and scale. The MoE efficiency, the 1M context window, the full training stack, the NIM deployment path: these are engineering choices that solve specific problems in multi-agent infrastructure.
For anyone building agent systems seriously, the relevant question isn’t whether Nemotron 3 benchmarks well against GPT-4o or Claude 3.5. It’s whether the task routing pattern – efficient open model for well-scoped tasks, frontier model for complex reasoning – fits your architecture, and whether Nemotron is the right efficient tier. Based on the capability profile, the cost structure, and the self-hosting viability, it’s a strong candidate.
I’ve added Nemotron 3 to the AI model landscape under the open/efficient tier for agent workloads. If you’re evaluating model options for your agent pipeline, that’s where I’m tracking comparable models and their practical tradeoffs.
Further context: self-hosting AI models and the agentic turn in AI infrastructure.