Overview

Family and scope

What Nemotron 3 is

  • NVIDIA Nemotron 3 is a family of open large language models designed around a specific systems objective: maximize reasoning and agentic capability per unit of inference cost rather than maximize model density or parameter utilization. The original family consists of Nano, Super, and Ultra, spanning roughly 30B, 120B, and 550B total parameters, while activating only a fraction of those parameters for each token through sparse Mixture-of-Experts (MoE) computation. Across the family, the central architectural pattern is a hybrid Mamba-Transformer MoE that combines sparse expert layers, predominantly Mamba-2 sequence layers, and a comparatively small number of self-attention layers. NVIDIA Nemotron 3: Efficient and Open Intelligence by NVIDIA et al. (2025) introduces the family and its common architecture, long-context, reinforcement-learning, and reasoning-control design.

  • The design can be viewed as combining three complementary mechanisms. MoE provides sparse parameter scaling, allowing total model capacity to grow without activating every parameter for every token. Mamba-2 by Dao and Gu (2024) provides sequence modeling with a fixed-size recurrent state during autoregressive generation, avoiding the sequence-length-dependent KV-cache growth of attention. A smaller number of attention layers retain high-fidelity global token-to-token communication where it is most useful. The Nemotron 3 architecture therefore does not treat Mamba as a complete replacement for attention; it uses attention selectively within a predominantly Mamba-2 and MoE stack.

  • This architecture is particularly aligned with agentic inference. Long-running agents often produce far more output tokens than conventional question-answering systems because they reason, invoke tools, inspect results, revise plans, and continue across many turns. In such workloads, reducing per-token memory traffic and attention cost can materially change end-to-end economics. Nemotron 3 consequently treats model architecture, numerical precision, speculative decoding, post-training, and serving infrastructure as a single optimization problem rather than independent components. NVIDIA Nemotron describes the broader Nemotron stack as an open model, data, training-recipe, and deployment ecosystem.

Why Nemotron is useful as an open research stack

  • Open-weight model families provide one of the few practical ways to study how modern LLMs are actually trained, but the usefulness of an open release depends on how much of the surrounding process is visible. Nemotron is unusually informative because the releases extend beyond final inference checkpoints to technical reports, training recipes, post-training software, RL environments, and substantial portions of the data used to construct the models. The NVIDIA Nemotron developer ecosystem explicitly presents the family as an open stack of model weights, training data, recipes, and deployment tooling rather than as weights alone.

  • This transparency is especially valuable for post-training, where implementation details often determine whether a method works in practice. The important questions are not limited to which optimization algorithm is used. They include how prompts are sourced and filtered, how multiple teacher trajectories are selected, how difficulty is estimated, how reward functions are constructed, how incompatible domains are mixed, how long-running environments are scheduled, and how specialized capabilities are consolidated without regressing others. Nemotron’s reports expose many of these operational details directly.

  • Across the family, the post-training stack can be understood through three recurring mechanisms:

    • supervised fine-tuning establishes behavioral priors, reasoning formats, tool-use conventions, and an initial set of high-quality trajectories;

    • reinforcement learning improves capabilities through task-specific rewards, preferably verifiable rewards when correctness can be checked automatically;

    • Multi-teacher On-Policy Distillation, introduced at Ultra scale, consolidates capabilities from separately optimized specialist policies into a shared student using dense token-level supervision.

  • The relative importance of these mechanisms changes with scale. Nano emphasizes broad SFT plus unified multi-environment RL. Super expands agentic SFT, asynchronous RL, PivotRL, and dedicated software-engineering RL. Ultra increasingly separates specialization from consolidation by training domain-specific teachers and combining them through MOPD. This progression provides a useful lens for understanding how post-training evolves as the number, heterogeneity, and horizon of target capabilities increase.

Nano, Super, and Ultra

  • The three principal Nemotron 3 text models scale total and active capacity while retaining the same broad architectural philosophy:
Model Scale Pre-training Context Main architectural characteristics Intended operating point
Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning by NVIDIA et al. (2025) 31.6B total, 3.2B active parameters 25T text tokens Up to 1M tokens Granular MoE, Mamba-2, GQA attention; 6 of 128 experts activated High-throughput sub-agents, reasoning, coding, and local or economical serving
Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning by NVIDIA et al. (2026) 120B total, 12B active parameters 25T text tokens Up to 1M tokens Hybrid Mamba-Attention, LatentMoE, MTP, NVFP4 pre-training Multi-agent systems and high-volume reasoning or tool-use workloads
Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning by NVIDIA et al. (2026) 550B total, 55B active parameters 20T text tokens Up to 1M tokens Larger LatentMoE hybrid, MTP, NVFP4 pre-training, MOPD Highest-capability planning, coding, research, and long-running autonomous agents

  • Nano contains 31.6B parameters while activating approximately 3.2B per forward pass, or approximately 3.6B when embeddings are included. Its granular MoE router activates 6 of 128 experts, while the sequence backbone combines Mamba-2 and grouped-query attention. It was pre-trained on 25T tokens and then post-trained through SFT, multi-environment RLVR, and RLHF.

  • Super increases capacity to 120B total and 12B active parameters and introduces three technologies that become central to the larger Nemotron 3 models: LatentMoE, native Multi-Token Prediction, and NVFP4 pre-training. Its 25T-token pre-training run is split into a broad 20T-token first phase and a higher-quality 5T-token second phase.

  • Ultra scales the same general design to 550B total and 55B active parameters. Its 20T-token pre-training process uses NVFP4, LatentMoE, and MTP, followed by context extension to 1M tokens and a more elaborate post-training pipeline combining SFT, reinforcement learning, and Multi-teacher On-Policy Distillation.

  • The family therefore scales capacity primarily through sparse parameters rather than proportionally increasing the computation applied to every token. The notation 30B-A3B, 120B-A12B, and 550B-A55B captures this distinction: the first quantity denotes approximate total parameter capacity, while A denotes the approximate parameter count activated for a token.

Design philosophy

Throughput-first design

  • Nemotron 3’s main architectural distinction is not MoE alone. Transformer MoEs still incur the cost of self-attention and an expanding KV cache during generation. Nemotron 3 instead places Mamba-2 layers throughout most of the sequence-processing stack and uses attention sparingly. In the Nano layer pattern, blocks of Mamba-2 and MoE layers dominate, with attention inserted only at selected locations.

  • Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality by Dao and Gu (2024) develops the state-space duality underlying Mamba-2 and shows how its recurrent formulation can provide substantially faster sequence processing than conventional attention implementations. Within Nemotron 3, this property is important during decoding because Mamba-2 maintains a constant-size recurrent state with respect to generated sequence length, while attention must access a KV cache that expands with the sequence.

  • The following figure (source) shows the reported accuracy-throughput positioning of the initial Nemotron 3 Nano model: the hybrid Mamba-Transformer MoE architecture maintains strong reasoning, coding, instruction-following, tool-use, and long-context results while increasing output-token throughput relative to similarly sized Transformer MoEs.

  • In the reported 8K-input/16K-output configuration, Nano reaches 3.3× the output-token throughput of Qwen3-30B-A3B-Thinking-2507 and 2.2× that of GPT-OSS-20B, measured using the better result from vLLM and TensorRT-LLM on a single H200 GPU for each model. The same architectural motivation becomes more important at longer generated sequences, where attention’s KV-cache traffic grows with history length.

  • The larger models continue this systems-oriented scaling. Super reports up to 2.2× the throughput of GPT-OSS-120B and 7.5× that of Qwen3.5-122B in an 8K-input/64K-output configuration on B200-class hardware, while Ultra reports substantially higher throughput than several larger open models in a corresponding 8K/64K regime. These results are configuration-dependent rather than architecture-only measurements, since precision, speculative decoding, serving engine, hardware, batching, and KV-cache representation also differ.

LatentMoE, MTP, and NVFP4

  • Super and Ultra extend the basic hybrid architecture with three mutually reinforcing techniques.

  • LatentMoE: Toward Optimal Accuracy per FLOP and Parameter in Mixture of Experts by Elango et al. (2026)

  • Star Elastic: Many-in-One Reasoning LLMs with Efficient Budget Control by Taghibakhshi et al. (2026); REAP the Experts: Why Pruning Prevails for One-Shot MoE compression by Lasby et al. (2025) changes the expert computation path by projecting token representations from the model hidden dimension into a smaller latent dimension before expert routing and expert computation. The objective is not merely to reduce FLOPs, but also to reduce expert-weight movement and all-to-all communication, two costs that can dominate practical MoE inference. LatentMoE is therefore a hardware-aware modification of conventional sparse expert design and is used in both Super and Ultra.

  • Better & Faster Large Language Models via Multi-token Prediction by Gloeckle et al. (2024) trains auxiliary heads to predict multiple future tokens rather than only the immediate next token. Nemotron 3 uses MTP both as additional training supervision and as a source of draft tokens for native speculative decoding. In the Nemotron 3 ablations, MTP improves a broad collection of downstream tasks and produces high-agreement drafts that can be verified by the main model without requiring an entirely separate draft LLM.

  • NVFP4 pushes low precision upstream from deployment-time quantization into training itself. Super is the first core Nemotron 3 model pre-trained using NVFP4, and Ultra follows the same approach. The intent is to exploit FP4-capable tensor cores not simply by compressing a completed BF16 model, but by training the model under the numerical format and associated scaling scheme from the start.

  • These techniques address different bottlenecks: Mamba reduces sequence-state and attention overhead, LatentMoE reduces sparse-expert memory and communication cost, MTP reduces the number of expensive target-model decoding steps, and NVFP4 lowers arithmetic and memory cost. Their combination is what distinguishes the larger Nemotron 3 stack from a conventional Transformer MoE with post-hoc quantization.

Agentic post-training

  • Nemotron 3 is also explicitly post-trained for agentic behavior rather than treating tool use as an incidental consequence of instruction tuning. Nano uses SFT followed by multi-environment reinforcement learning from verifiable rewards and RLHF. Its SFT mixture includes chat, agentic, reasoning, tool-integrated, reasoning-on/off, and reasoning-budget-control traces. Rather than performing separate RL runs for individual capabilities, multi-environment RL trains across heterogeneous environments within a common optimization run.

  • Super expands the breadth and scale of these agent environments and introduces more resilient asynchronous RL infrastructure for long-horizon tasks. The emphasis shifts toward software engineering, terminal interaction, multi-step tool use, and multi-agent workloads, where trajectories may be substantially longer and more variable than conventional single-response RL.

  • Ultra adds another stage through Multi-teacher On-Policy Distillation. More than ten specialized teacher models are trained for different capability domains, including dedicated agentic teachers. Ultra then generates its own on-policy trajectories and receives dense token-level supervision from those teachers. This combines RL’s ability to optimize interactive behavior with distillation’s denser learning signal.

  • A particularly practical feature shared across Nemotron 3 is reasoning-effort control. The model can be given a maximum reasoning-token budget; when the budget is reached, inference can terminate the private reasoning portion and continue to the final response based on the reasoning accumulated so far. This converts reasoning length into an explicit latency and cost control instead of treating chain-of-thought length as an uncontrolled model behavior.

Long-context agent design

  • The text Nemotron 3 models support context windows up to 1M tokens. The intended use cases include large codebases or code slices, long conversational state, lengthy tool trajectories, and large collections of documents for retrieval-augmented or research-oriented agents.

  • The importance of this feature is tied to the architecture. Merely increasing a Transformer’s configured context limit does not remove the cost of repeatedly reading and storing attention state across long generated trajectories. Nemotron 3 pairs the long window with Mamba-heavy sequence processing so that long-context capability and long-output throughput are optimized together. The resulting design is particularly relevant to agents whose execution state may contain instructions, retrieved documents, observations from previous tools, intermediate code, execution traces, and accumulated conversation history.

Extensions and ecosystem

Nano Omni

  • Nemotron 3 Nano Omni: Efficient and Open Multimodal Intelligence by NVIDIA et al. (2026) extends the 30B-A3B Nano backbone into a unified text, image, video, and audio model. It adds a C-RADIOv4-H vision encoder and a Parakeet-TDT-0.6B-v2 audio encoder, with modality-specific projectors feeding representations into the shared Nemotron 3 language backbone.

  • The multimodal design is also optimized around token efficiency. Images use dynamic resolution rather than fixed tiling; video uses Conv3D-based temporal compression to reduce the temporal token count by 2×; and audio passes through substantial temporal subsampling before entering the LLM. The multimodal model uses a 256K-token context rather than the text backbone’s 1M-token maximum.

  • For audio, inputs are resampled to 16 kHz mono, converted into log-mel features, and passed through three stride-2 convolutional subsampling stages, yielding roughly 12.5 audio tokens per second. Audio, visual, and text tokens can then be interleaved temporally before entering the language model, enabling joint reasoning over, for example, video frames and synchronized speech.

  • This turns Nano into a perception-oriented sub-agent that can inspect interfaces, documents, images, audio, and video while sharing the same general reasoning backbone as the text model.

Nemotron 3.5 Super, Lightning, and Bolt

  • Nemotron 3.5 Super and Lightning: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Models for Agentic AI by NVIDIA et al. (2026) extends the family into two complementary deployment points. Nemotron 3.5 Super is a 120B-total, 12B-active multimodal model initialized from Nemotron 3 Super, while Nemotron 3.5 Lightning is a 30B-total, 3B-active text model initialized from Nemotron 3 Nano Base. Both retain the hybrid Mamba-Transformer MoE design, receive dedicated MTP adaptation, large-scale teacher distillation, and long-context continuation, but they diverge substantially in capability specialization and post-training depth.

  • The 3.5 generation makes distillation a first-class stage rather than relying only on conventional continued pre-training. Both Super and Lightning first warm up their MTP drafter for 100B tokens with the backbone frozen and then undergo 1T tokens of model distillation from the next-larger Nemotron form factor: Ultra 3 Base teaches Super 3.5, while Super 3 Base teaches Lightning 3.5. Super subsequently adds Indic continued pre-training, 1M-context training, agentic software-engineering mid-training, and a long-context recovery phase; Lightning follows a shorter path to a 1M-token text model.

  • Super 3.5 also changes the role of multimodality in the family. Its base checkpoint is text-only, but two-stage SFT introduces large-scale image and video supervision together with reasoning, long-context, safety, and agentic data. Specialized reasoning, agentic, alignment, video, vision-reasoning, and vision-tool-use teachers are later consolidated through MOPD. Lightning intentionally omits this heavier multimodal and multi-teacher path and uses single-stage text SFT followed by RLVR.

  • Nemotron 3.5 Bolt 74B-A9B 24GB: A Hybrid Vision-Language Model for Local AI by NVIDIA et al. (2026) extends the same design philosophy toward local AI. Bolt begins from the Super 3.5 VL 120B-A12B parent and compresses it to 74B total and approximately 9B active parameters, then applies multimodal recovery, agentic post-training, aggressive low-bit quantization, and a LiLiCorr speculative drafter. The design target is not merely a smaller parameter count, but a system that maximizes capability per GB under a local-device memory and latency budget. The NVIDIA Nemotron 3.5 Bolt 74B-A9B 24GB release page describes this local-agent operating point and the associated Hugging Face and GGUF artifacts.

  • Bolt’s compression stage uses Star Elastic: Many-in-One Reasoning LLMs with Efficient Budget Control by Taghibakhshi et al. (2026), which learns nested or reduced architectures through differentiable budget-conditioned routing and knowledge distillation. For Bolt, the router chooses routed-expert counts and MoE intermediate widths layer by layer while preserving the parent hidden width and hybrid layer structure, so compression becomes a deployment-constrained architecture-search problem rather than uniform pruning.

  • The resulting 3.5 branch therefore exposes three distinct ways to move the capability-efficiency frontier. Super preserves the 120B-A12B scale while investing heavily in multimodal and agentic specialization; Lightning preserves the small 30B-A3B form factor while using distillation and long-context training to improve capability; Bolt begins with the larger Super representation and compresses it aggressively toward a local-memory target. Across all three, MTP or a standalone parallel drafter is treated as part of the model-system design rather than as a serving afterthought. NVIDIA Nemotron 3.5 Super and Lightning and NVIDIA Nemotron 3.5 Bolt 74B-A9B 24GB provide the corresponding model, data, recipe, and deployment releases.

Nemotron-Labs-Diffusion

  • A related research branch, Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decoding by Fu et al. (2026), explores a single language model trained jointly for autoregressive and diffusion objectives. The same parameters can then operate in conventional autoregressive mode, pure diffusion mode, or self-speculation mode in which diffusion drafts multiple tokens and the autoregressive model verifies them.

  • This work is not one of the original Nano, Super, or Ultra checkpoints, but it extends the same efficiency-first research agenda. MTP-based speculative decoding predicts future tokens from auxiliary prediction heads; Nemotron-Labs-Diffusion instead uses a diffusion process as the internal drafter. Its central premise is that the optimal decoding algorithm can depend on serving concurrency: autoregressive decoding can maximize aggregate server throughput at high batch sizes, self-speculation can improve per-user latency at low concurrency, and diffusion provides greater theoretical parallel-decoding headroom.

Full-stack openness

  • Nemotron’s notion of openness extends beyond releasing inference weights. The family is accompanied by base and post-trained checkpoints, quantized variants, portions of the pre-training and post-training corpora, RL environments, training recipes, and software integrations. The NVIDIA Nemotron developer ecosystem reports more than 10T openly available pre-training tokens and tens of millions of post-training examples spanning reasoning, coding, RL, safety, RAG, and agentic tasks.

  • Deployment is similarly treated as part of the model design. Nemotron checkpoints are supported across inference stacks including vLLM, SGLang, TensorRT-LLM, Hugging Face Transformers, Ollama, and llama.cpp, with both hosted and self-managed deployment paths. This matters because many of Nemotron 3’s efficiency mechanisms, including sparse expert execution, Mamba state handling, low-precision kernels, MTP, and speculative decoding, depend heavily on inference-engine support to produce their intended end-to-end gains.

  • Taken together, Nemotron 3 is best understood not as a single model architecture but as a coordinated model-systems stack: hybrid Mamba-attention sequence modeling reduces long-sequence cost; MoE and LatentMoE scale capacity sparsely; NVFP4 lowers numerical and memory cost; MTP and newer draft mechanisms accelerate decoding; multi-environment RL teaches long-horizon interaction; explicit reasoning controls expose an inference-time accuracy-cost knob; and open training and deployment artifacts make those design decisions reproducible rather than confined to an API.

Architecture

Core hybrid architecture

  • The defining architectural choice in Nemotron 3 is to combine three forms of computation within the same decoder stack: Mamba-2 for efficient sequential state propagation, sparse Mixture-of-Experts layers for high parameter capacity at relatively low active compute, and a small number of self-attention layers for unrestricted global information exchange. Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning by NVIDIA et al. (2025) establishes this architecture in Nano, while Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning by NVIDIA et al. (2026) and Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning by NVIDIA et al. (2026) scale the design using LatentMoE and Multi-Token Prediction.

  • A conventional decoder-only Transformer typically alternates self-attention and dense feed-forward layers. Nemotron 3 replaces both sides of this pattern selectively. Most sequence mixing is performed by Mamba-2 rather than attention, and every conventional feed-forward network is replaced with a sparsely activated MoE layer. Attention remains periodically inserted into the stack as a global communication mechanism. The resulting conceptual block is therefore closer to:

      Mamba-2 → MoE → Mamba-2 → MoE → ... → Attention → MoE

    • than the conventional structure:

        Attention → FFN → Attention → FFN → ...

  • The goal is to retain the large representational capacity and global routing capabilities of Transformers while shifting the majority of autoregressive inference toward operations whose memory requirements do not grow with the entire generated history. NVIDIA Nemotron 3: Efficient and Open Intelligence by NVIDIA et al. (2025) describes the hybrid as a balance between sparse expert scaling, all-to-all information routing through attention, and fixed-state sequence modeling through Mamba-2.

Model dimensions

  • The architecture scales substantially from Nano to Ultra, but several structural choices remain stable. Mamba state size remains 128, Mamba uses eight groups, attention head dimension remains 128, and the models use only two KV heads even as the number of query heads grows. Super and Ultra additionally use a latent expert space and two shared-weight MTP layers. Nemotron 3 Nano by NVIDIA et al. (2025), Nemotron 3 Super by NVIDIA et al. (2026), and Nemotron 3 Ultra by NVIDIA et al. (2026) provide the corresponding architectural configurations.
Configuration Nano 30B-A3B Super 120B-A12B Ultra 550B-A55B
Decoder layers 52 88 108
Model dimension 2,688 4,096 8,192
Query heads 32 32 64
KV heads 2 2 2
Attention head dimension 128 128 128
Mamba state dimension 128 128 128
Mamba groups 8 8 8
Mamba heads 64 128 256
Mamba head dimension 64 64 64
Expert hidden dimension 1,856 2,688 5,120
Routable experts 128 512 512
Activated experts per token 6 22 22
Shared expert intermediate size 2 shared experts 5,376 10,240
MoE latent dimension Not used 1,024 2,048
Shared-weight MTP layers Not used 2 2
  • The table illustrates an important scaling property: total capacity grows much faster than the amount of computation activated for a particular token. Nano contains 31.6B parameters but activates roughly 3.2B parameters excluding embeddings. Super scales to 120.6B total parameters while using roughly 12.1B excluding embeddings per forward pass. Ultra further scales this principle to approximately 550B total and 55B active parameters. Sparse expert scaling therefore supplies model capacity without requiring dense execution of the full parameter count.

Nano layer pattern

  • Nano contains 52 computational layers arranged into a deliberately nonuniform hybrid pattern. From the layer diagram, the model contains repeated groups of Mamba-2 and MoE layers, with self-attention inserted at selected positions rather than every block. This means the model can propagate information cheaply through many recurrent-state layers before periodically using attention to directly remix information across the sequence. Nemotron 3 Nano by NVIDIA et al. (2025) describes this as an extension of the earlier Nemotron hybrid architecture in which dense FFNs are replaced by sparse MoE layers.

  • The following figure (source) shows the Nemotron 3 Nano layer pattern, with predominantly interleaved Mamba-2 and MoE layers and periodically inserted attention layers.

  • This structure produces 23 Mamba-2 layers, 23 MoE layers, and 6 attention layers across the 52-layer Nano stack. Thus, attention accounts for only a minority of the sequence-processing layers, while Mamba performs most token-to-token state propagation. The architecture retains global attention at carefully spaced points rather than eliminating it entirely. This design is one reason Nemotron 3 should be viewed as a hybrid state-space model rather than simply a Mamba language model.

Sequence mixing

Mamba-2 sequence mixing

  • Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality by Dao and Gu (2024) introduces Mamba-2 and its state-space-duality formulation, enabling recurrent sequence processing while retaining efficient parallel implementations during training. Nemotron 3 uses Mamba-2 for the majority of its sequence mixing because recurrent decoding maintains a bounded state rather than retaining keys and values for every preceding token.

  • At a high level, a selective state-space layer can be represented schematically as

    \[h_t = A_t h_{t-1} + B_t x_t\] \[y_t = C_t h_t\]
    • where the current input updates a recurrent hidden state, and the output is computed from that state. The matrices are input-dependent in selective state-space models, allowing the recurrence to selectively retain or discard information rather than behaving as a fixed linear filter. Transformers are SSMs by Dao and Gu (2024) develops the more general structured state-space formulation used by Mamba-2.

  • The important inference consequence is that the state required by a Mamba layer does not need to store representations for all previous sequence positions. Its recurrent state has fixed architectural dimensions. By contrast, an attention layer maintains cached keys and values for previous tokens during autoregressive decoding. Nemotron therefore reduces long-sequence inference cost in two ways: it reduces the number of attention layers, and it uses highly compressed GQA caches in the attention layers that remain. Nemotron 3 Super by NVIDIA et al. (2026) explicitly describes the Mamba blocks as constant-state sequence-processing layers and the remaining attention layers as global anchors.

Attention as a global anchor

  • Mamba’s recurrent state is efficient, but unrestricted self-attention remains useful for directly connecting arbitrary positions in a long context. Super therefore describes its selected attention layers as global anchors. The model repeatedly performs efficient local-to-global state propagation through Mamba-2 and sparse expert computation, then periodically applies full attention to permit direct interaction between tokens throughout the available context. Nemotron 3 Super by NVIDIA et al. (2026) states that this interleaving is intended to preserve global dependency modeling while moving most computation to Mamba and sparse MoE components.

  • This architecture can be interpreted as a hierarchy of communication mechanisms:

    • Mamba-2 carries compressed sequence state forward efficiently.

    • MoE layers transform each token using a large sparsely selected parameter pool.

    • Periodic attention layers globally remix information between sequence positions.

  • The separation is useful because each mechanism addresses a different bottleneck. Mamba targets sequence-length-dependent inference cost, MoE targets parameter-capacity scaling, and attention supplies high-bandwidth global interaction.

Grouped-Query Attention and KV-cache reduction

  • The attention layers use Grouped-Query Attention rather than conventional multi-head attention. GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints by Ainslie et al. (2023) introduces GQA as an intermediate design between multi-head attention and multi-query attention, allowing many query heads to share a much smaller set of key-value heads.

  • Nano and Super use 32 query heads but only 2 KV heads, while Ultra uses 64 query heads and still only 2 KV heads. The head dimension is 128 throughout the family.

  • Ignoring implementation-specific alignment and metadata, the KV-cache storage of one attention layer scales approximately as

    \[M_{\mathrm{KV}} \propto 2 T n_{\mathrm{KV}} d_h b\]
    • where the factor of two represents keys and values, sequence length determines the number of cached positions, the number of KV heads determines how many independent key-value streams are stored, head dimension specifies each stream’s width, and the final factor denotes bytes per stored element.

  • Consequently, Nano and Super maintain only 2 KV streams for 32 query heads. Relative to a conventional 32-head configuration with one KV head per query head, this reduces the number of cached KV-head streams by a factor of 16 for each remaining attention layer. Ultra’s 64-query-head and 2-KV-head design produces a corresponding 32-to-1 reduction in KV-head streams relative to a 64-KV-head multi-head implementation. This GQA reduction compounds with the small number of attention layers, making the aggregate attention cache substantially smaller than it would be in a similarly deep all-attention Transformer. GQA by Ainslie et al. (2023) motivates this sharing specifically as an inference-efficiency mechanism, while the Nemotron reports specify the two-KV-head configurations used in practice.

Sparse capacity

Sparse MoE in Nano

  • Nano replaces the conventional FFN after sequence-mixing layers with granular sparse experts. DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models by Dai et al. (2024) develops granular expert segmentation together with shared experts, allowing a token to combine a finer-grained selection of specialized experts while reserving shared capacity for commonly required knowledge. Nemotron 3 Nano adopts both granular routed experts and shared experts.

  • For a schematic MoE layer, let the router assign scores to the expert set:

    \[s_i(x) = \sigma\left(g_i(x)\right)\]

    • and let the routed output select only the highest-scoring experts:

      \[y_{\mathrm{routed}} = \sum_{i \in \operatorname{TopK}(s(x))} s_i(x)E_i(x)\]

  • A shared-expert component is then combined with the routed computation:

    \[y = y_{\mathrm{routed}} + \sum_{j=1}^{N_{\mathrm{shared}}} E^{\mathrm{shared}}_j(x)\]

  • The equations are a schematic representation of the routing pattern. Nano specifically uses a learned MLP router with sigmoid gating, squared-ReLU expert activations, 128 routable experts, top-6 expert activation, and 2 shared experts. It uses no positional embeddings, no dropout, and no bias in linear layers; normalization uses RMSNorm, and input embeddings and output projection weights are untied. Nemotron 3 Nano by NVIDIA et al. (2025) specifies these implementation choices.

  • Sparse routing means that increasing the number of experts increases total parameter capacity without proportionally increasing per-token expert computation. Nano therefore exposes 128 potential routed experts but evaluates only 6 of them for a token, plus its shared experts. This is the basis for the distinction between roughly 31.6B total parameters and approximately 3.2B active parameters excluding embeddings.

LatentMoE in Super and Ultra

  • The principal MoE innovation introduced with Super is LatentMoE: Toward Optimal Accuracy per FLOP and Parameter in Mixture of Experts by Elango et al. (2026), which redesigns expert computation around memory bandwidth and inter-device communication rather than optimizing FLOPs alone. Standard MoE layers route the full model-hidden representation to experts. At large hidden dimensions, this produces two important costs: expert matrices are expensive to read from memory for latency-sensitive inference, and all-to-all token dispatch becomes expensive when experts are distributed across devices.

  • LatentMoE inserts a learned projection before routed expert computation. For an input representation in the full hidden dimension, the model first maps it into a smaller expert latent space:

    \[z = W_{\downarrow}x\]
    • where \(W_{\downarrow} \in \mathbb{R}^{\ell \times d}\) and the latent width is smaller than the model width \(\ell < d\).

  • Routing and routed expert computation then occur in this compressed representation:

    \[u = \sum_{i \in \operatorname{TopK}(r(z))} r_i(z)E_i(z)\]

  • The aggregated routed result is projected back to the model dimension:

    \[y_{\mathrm{routed}} = W_{\uparrow}u\]

    • with:

      \[W_{\uparrow} \in \mathbb{R}^{d \times \ell}\]

  • LatentMoE by Elango et al. (2026) uses this projection to reduce routed weight movement and all-to-all communication while reinvesting the savings into a larger expert pool and a larger active-expert set.

  • The following figure (source) shows the standard MoE and LatentMoE computation paths. LatentMoE moves routed expert computation and all-to-all communication into a lower-dimensional latent space while leaving shared experts, routing, and other non-routed operations at the full model width.

  • The reduction in routed communication and expert-weight traffic is approximately governed by the compression ratio \(\frac{d}{\ell}\) and the design uses the saved budget to increase the expert pool according to the conceptual scaling relation \(N' = N\frac{d}{\ell}\) with a corresponding increase in the number of routed experts \(K' = K\frac{d}{\ell}\), while preserving approximately comparable inference cost. The exact production configurations are empirically chosen rather than simply applying these equations mechanically.

  • Super uses a model dimension of 4,096 and a latent MoE dimension of 1,024, giving a 4-to-1 dimensional reduction for routed computation. Its expert pool contains 512 experts with 22 activated per token. Ultra scales the model dimension to 8,192 and the latent dimension to 2,048, retaining the same 4-to-1 compression ratio and the same 512-expert, top-22 routing configuration.

  • An ablation reported in NVIDIA Nemotron 3: Efficient and Open Intelligence by NVIDIA et al. (2025) compares an approximately 73B-parameter standard MoE using 128 experts and top-6 routing against a similarly sized LatentMoE using 512 experts and top-22 routing. With comparable active and total parameter counts, LatentMoE improves the reported aggregates across MMLU-Pro, MMLU, code, math, and commonsense evaluation categories, motivating its adoption in Super and Ultra.

Speculative prediction

Multi-Token Prediction

  • Super and Ultra add Multi-Token Prediction as both a training objective and an inference mechanism. Better & Faster Large Language Models via Multi-token Prediction by Gloeckle et al. (2024) proposes augmenting conventional next-token training with prediction of multiple future tokens from each position, providing a denser learning signal and enabling the resulting heads to draft several tokens for speculative inference.

  • A conventional autoregressive objective predicts one next token:

    \[\mathcal{L}_{\mathrm{NTP}} = -\sum_t \log p(x_{t+1}\mid x_{\leq t})\]

  • A multi-token objective can instead be expressed as a sum over several future prediction horizons:

    \[\mathcal{L}_{\mathrm{MTP}} = -\frac{1}{H} \sum_{h=1}^{H} \sum_t \log p_h(x_{t+h}\mid x_{\leq t})\]

  • Nemotron 3 modifies the conventional independent-head MTP formulation. Super trains multiple prediction positions using shared MTP parameters, exposing the same predictor to several future offsets. The shared-weight formulation reduces the mismatch encountered when a head trained for one fixed offset is recursively reused on its own generated states. At inference, the shared predictor can therefore draft autoregressively beyond the number of physical training heads.

  • Ultra retains two shared-weight MTP heads. Its report further specifies that the shared MTP module consists of a single attention layer followed by a single MoE layer. Nemotron 3 Ultra by NVIDIA et al. (2026) uses these heads for native speculative decoding while keeping the same general architecture inherited from Super.

MTP as an internal drafter

  • MTP becomes particularly useful because future-token predictions can act as speculative draft tokens. Fast Inference from Transformers via Speculative Decoding by Leviathan et al. (2022) formalizes the broader draft-and-verify approach, in which a cheaper predictor proposes several tokens and the target model verifies them together.

  • Nemotron’s MTP heads play the role of an internal drafter:

      Main model hidden state
          ↓
  Shared MTP predictor
          ↓
  draft₁ → draft₂ → draft₃ → ...
          ↓
  Main model verifies candidate block
          ↓
  Accept valid prefix and continue

  • This avoids loading and executing a completely independent draft language model. The physical MTP module is comparatively small, and the same prediction module can be recursively reused to construct longer drafts. Super’s shared-head design specifically aims to improve acceptance stability as this recursive draft becomes longer.

  • On SPEED-Bench with a draft length of seven, Super reports an average accepted length of 3.45 tokens per verification step, compared with 3.33 for Qwen3-Next and 2.70 for DeepSeek-R1 under the reported setup. The advantage is particularly relevant because successful speculative decoding reduces the number of expensive main-model forward passes required to emit a given number of output tokens. Nemotron 3 Super by NVIDIA et al. (2026) reports that the shared-head design remains comparatively stable at deeper draft positions.

  • The following figure (source) shows the serving trade-off produced by MTP on Nemotron 3 Super, comparing MTP disabled with draft depths of one and three on an NVFP4 checkpoint.

  • The important systems effect is that MTP changes the throughput-latency frontier rather than simply increasing a single benchmark number. Longer accepted drafts amortize target-model verification over multiple output tokens, which is particularly valuable at low concurrency where ordinary one-token-at-a-time decoding underutilizes the accelerator.

Scaling and compression

Super hybrid stack

  • Super expands Nano from 52 to 88 layers and replaces standard granular MoE with LatentMoE. The model retains the hybrid pattern rather than reverting to full attention: Mamba-2 and LatentMoE constitute most of the stack, while attention layers are inserted periodically as global anchors. Nemotron 3 Super by NVIDIA et al. (2026) describes the architecture as jointly optimizing long-context modeling and inference throughput.

  • The attention layers use 32 query heads, 2 KV heads, and a head dimension of 128. As in Nano, Super omits explicit positional embeddings, dropout, and biases in linear layers; it uses RMSNorm and untied embedding and output-projection matrices. Sparse routing activates 22 of 512 experts for each token. The complete architecture reaches approximately 120.6B total parameters while keeping the active budget at approximately 12.7B including embeddings.

  • The following figure (source) shows the Nemotron 3 Super hybrid layer pattern, where LatentMoE layers are interleaved primarily with Mamba-2 blocks and occasional attention layers.

Ultra scaling

  • Ultra retains the Super architecture rather than introducing an unrelated backbone. Nemotron 3 Ultra by NVIDIA et al. (2026) scales the model to 108 layers, increases model width from 4,096 to 8,192, doubles the number of Mamba heads from 128 to 256, increases query heads from 32 to 64, and increases the expert hidden dimension from 2,688 to 5,120. The number of KV heads remains fixed at two, and the expert topology remains 512 total with 22 routed experts per token.

  • The LatentMoE width grows proportionally from 1,024 to 2,048, preserving the 4-to-1 relationship between model width and routed latent width. The shared-expert intermediate size doubles from 5,376 to 10,240. These choices indicate that the primary route to additional capability is wider hidden representations, larger expert computation, greater depth, and a substantially larger total parameter pool, rather than increasing the number of routed experts or KV heads.

  • Ultra also retains two shared-weight MTP heads, allowing the speculative-decoding mechanism introduced in Super to scale with the target model rather than depending on an external drafter. The resulting architecture therefore preserves the same three-way decomposition across the family: Mamba for efficient sequence state, LatentMoE for sparse capacity, and attention for periodic global synchronization.

Bolt deployment-budgeted compression

  • Nemotron 3.5 Bolt 74B-A9B 24GB: A Hybrid Vision-Language Model for Local AI by NVIDIA et al. (2026) introduces a different scaling operation from Nano-to-Super-to-Ultra growth: it compresses the 120B-A12B Nemotron 3.5 Super VL parent into a 74B-A9B model under an explicit deployment-memory constraint. The compressed model retains 40 Mamba layers, 40 MoE layers, 8 attention layers, hidden width 4,096, LatentMoE width 1,024, and shared-expert intermediate width 5,376, preserving compatibility with the parent’s vision stack. Compression instead acts primarily on routed-expert count, routed-expert intermediate width, and active top-\(K\), which falls from 22 to 9.

  • The architecture search is based on Star Elastic: Many-in-One Reasoning LLMs with Efficient Budget Control by Taghibakhshi et al. (2026). Before differentiable search, routed experts and intermediate channels are ordered by importance; expert ordering uses the router-aware criterion developed in REAP the Experts: Why Pruning Prevails for One-Shot MoE compression by Lasby et al. (2025), while the channel ordering follows Star Elastic. Because the permutations are paired with corresponding router outputs or projection dimensions, they preserve the parent function before masking.

  • For candidate size \(c_j\) on one compression axis, Bolt uses a nested prefix mask

    \[m_j[i] = \mathbf{1}\!\left[i\le c_j\right]\]

    • where smaller candidates retain prefixes of the experts or channels retained by larger candidates. A budget-conditioned router produces Gumbel-softmax probabilities \(p_j\), giving the soft mask

      \[\widetilde{m} = \sum_j p_j m_j\]

      • and the corresponding masked activation

        \[\widetilde{Z} = Z\odot\widetilde{m}\]

    • This makes the discrete architecture choice differentiable during search, allowing the distillation signal to train both the student weights and the architecture router.

  • The deployment constraint is converted into a target parameter count \(b\) using the intended quantization formats, recurrent-state precision, sequence length, concurrency, and other runtime assumptions. If \(P(\psi;b)\) is the differentiable parameter-count estimate produced by router parameters \(\psi\), the relative budget error is

    \[\delta = \left|\frac{P(\psi;b)}{b}-1\right|\]

    • with budget penalty

      \[\mathcal{L}_{\mathrm{budget}}(\psi;b) = \begin{cases} 0, & \delta\le\epsilon\\ \delta, & \delta>\epsilon \end{cases}\]

    • and joint compression objective

      \[\min_{\theta,\psi} \underbrace{\mathcal{L}_{\mathrm{KD}}(\theta,\psi;b)}_{\text{teacher-student distribution matching}} + \underbrace{\lambda\mathcal{L}_{\mathrm{budget}}(\psi;b)}_{\text{deployment-budget constraint}}\]

    • where only knowledge distillation updates the student weights \(\theta\), while both terms update the architecture router. Bolt uses \(\lambda=1\) and a relative budget tolerance \(\epsilon=0.05\).

  • The final layer-wise architecture is intentionally nonuniform. Routed-expert counts are selected from \(\{240,304,368,432,496\}\) and routed-expert intermediate widths from \(\{1536,2048,2304,2560\}\), allowing different depths to spend parameter capacity differently. The search target is below 24GB at batch size 1 and 128K context under the planned low-bit expert representation and runtime-state precisions.

  • The following figure (source) shows the Star Elastic compression pipeline in which a memory-budget-conditioned router, differentiable Gumbel-softmax masks, logit knowledge distillation, and a budget penalty jointly optimize student weights and per-layer MoE capacity.

  • Search and recovery use a 70% blend of the Super 3.5 VL SFT data and 30% Ultra-v3 pre-training data. Expert and channel importance is estimated from 8M sampled tokens; the router converges during the first 8B tokens at 8K context, after which the architecture is fixed and distilled for another 64B tokens at 8K and 48B tokens at 128K, for 120B total compression-and-recovery tokens. This schedule performs most recovery at short context, where training is cheaper, before extending the recovered student to the longer deployment regime.

Component interactions

  • The efficiency of Nemotron 3 does not come from a single mechanism. Each architectural component targets a different source of inference cost:
Component Primary function Systems effect
Mamba-2 Sequence modeling Replaces most attention with fixed-size recurrent state
Sparse MoE Parameter scaling Increases total capacity without activating all parameters
Shared experts Common computation Keeps broadly useful capabilities outside routed specialization
GQA Global attention Reduces KV-cache storage in the attention layers that remain
Attention anchors Global communication Periodically permits direct interactions across the full sequence
LatentMoE Expert computation Reduces expert-weight traffic and all-to-all communication
MTP Future-token prediction Adds denser supervision and enables internal speculative drafting
  • This composition is particularly well matched to reasoning agents. Long reasoning traces amplify attention and KV-cache costs; tool-using agents require strong global context integration; sparse experts allow the model to retain large capacity without dense execution; and speculative decoding reduces the number of expensive target-model steps required for long outputs. The architecture therefore treats long-context reasoning throughput as a first-class design constraint rather than an optimization added after model training. NVIDIA Nemotron 3: Efficient and Open Intelligence by NVIDIA et al. (2025) frames the hybrid architecture around this accuracy-to-inference-throughput objective, while Nemotron 3 Super by NVIDIA et al. (2026) extends the approach through hardware-aware LatentMoE and native MTP.

Pre-training

Training recipe

Configuration Nano 30B-A3B Super 120B-A12B Ultra 550B-A55B
Main pre-training tokens 25T 25T 20T
Phase 1 23.5T, 94% 20T, 80% 15T, 75%
Phase 2 1.5T, 6% 5T, 20% 5T, 25%
Phase 1 objective Broad diversity Broad diversity Broad diversity
Phase 2 objective Higher-quality data Higher-quality data Higher-quality data
Peak learning rate \(10^{-3}\) \(4.5\times10^{-4}\) \(2.5\times10^{-4}\)
Warmup tokens 8.4B 200B 200B
Final learning rate \(10^{-5}\) \(4.5\times10^{-6}\) \(2.5\times10^{-6}\)
Base sequence length 8,192 8,192 Inherits Super recipe unless modified
MTP auxiliary scale None 0.3 0.1
Long-context CPT 121B tokens 51B tokens 33B tokens
  • The most notable evolution is the increasing importance of the quality-focused phase. Nano reserves only the final 6% of pre-training for its phase-2 blend, Super expands this to 20%, and Ultra moves the transition to approximately 15T tokens, leaving 25% of its 20T-token horizon for the higher-quality distribution.

Autoregressive objective

  • At its core, base-model pre-training remains causal next-token prediction. For tokens \(x_1,\ldots,x_T\), the standard language-model objective is

    \[\mathcal{L}_{\mathrm{NTP}} = -\frac{1}{T} \sum_{t=1}^{T} \log p_{\theta} \left( x_t \mid x_{<t} \right)\]

  • For Super and Ultra, this is augmented by the MTP objective described in the architecture section. A schematic representation of the joint objective is

    \[\mathcal{L} = \mathcal{L}_{\mathrm{NTP}} + \lambda_{\mathrm{MTP}}\mathcal{L}_{\mathrm{MTP}} + \lambda_{\mathrm{LB}}\mathcal{L}_{\mathrm{LB}}\]
    • where \(\mathcal{L}_{\mathrm{MTP}}\) supplies supervision for future-token prediction and \(\mathcal{L}_{\mathrm{LB}}\) is the small conventional MoE load-balancing term used alongside bias-based loss-free expert balancing. Super sets the MTP loss scaling factor to \(0.3\), whereas Ultra reduces it to \(0.1\). Both Nano and Super report a conventional load-balancing-loss coefficient of \(10^{-4}\).

Data curriculum

Diversity-to-quality curriculum

  • The common data strategy is a two-phase curriculum. Phase 1 prioritizes broad coverage across web, code, mathematics, scientific material, multilingual text, academic documents, Wikipedia, and synthetic instruction-like data. Phase 2 retains broad capability coverage but reallocates probability mass toward data judged to be higher quality. The underlying principle is to use most of the training horizon to acquire broad knowledge and then perform the final portion of optimization on a cleaner distribution. Nemotron 3 Nano by NVIDIA et al. (2025) introduces this recipe for the family, while Nemotron 3 Super by NVIDIA et al. (2026) and Nemotron 3 Ultra by NVIDIA et al. (2026) progressively increase the fraction allocated to the quality-focused phase.

  • This curriculum is notable because the pre-training distribution is not restricted to conventional unsupervised text. Synthetic SFT-style examples for general instruction following, STEM, mathematics, reasoning, and code are explicitly mixed into pre-training. This effectively introduces structured reasoning supervision before the dedicated post-training stage, providing the base model with exposure to solution traces and instruction-like formats while it is still learning its general representation.

Nano data mixture

  • Nano’s 25T-token corpus spans fifteen high-level categories. Web data is divided into quality tiers such as crawl-medium, crawl-medium-high, crawl-high, and synthetic versions of the higher-quality crawl groups. Other categories include code, Nemotron-CC-Code, mathematics, academic material, Wikipedia, multilingual data, Crawl++, and general, STEM, and code SFT-style data. Its multilingual component spans 19 languages.

  • The corpus includes several significant data refreshes. Nemotron-CC-v2.1 contributes approximately 2.5T new English tokens; Nemotron-CC-Code-v1 contributes approximately 428B high-quality code-oriented tokens extracted from Common Crawl while preserving source code and mathematical structure; additional GitHub data refreshes expand source-code coverage; and specialized synthetic datasets target STEM reasoning and scientific coding.

  • The data shift between Nano’s two phases is substantial even though phase 2 is only 1.5T tokens. For example, STEM SFT increases from 11.1% to 22.3%, math from 6.4% to 12.5%, code SFT from 3.3% to 6.7%, and Wikipedia from 0.6% to 1.3%. Lower-quality crawl data is reduced or removed while high-quality synthetic crawl remains heavily represented. Code remains fixed at 14%, preserving coding coverage throughout the curriculum.

  • The following figure (source) shows the phase-1 and phase-2 data mixtures used for Nemotron 3 Nano pre-training, illustrating the shift from broad web coverage toward more heavily weighted STEM, mathematics, code-SFT, and Wikipedia data in the final phase.

Super data mixture

  • Super retains the basic Nano taxonomy but expands the corpus to sixteen high-level categories and adds or refreshes specialized datasets. Its first 20T tokens emphasize diversity, while the final 5T-token phase shifts toward higher-quality material. One visible example is document data: the first phase allocates 6.1% to general FinePDFs, while phase 2 allocates 14.3% specifically to high-quality FinePDFs. Code remains at 14%, and synthetic high-quality crawl remains at 22.4% in both phases.

  • The Super pipeline also illustrates how synthetic data is constructed rather than simply collected. For one synthetic multiple-choice corpus, seed questions are drawn from established MCQ sources, new questions are generated using Qwen3-235B-A22B, DeepSeek-V3 produces answers and supporting reasoning, multiple solutions are sampled independently, and majority voting is used to retain internally consistent examples. This pipeline produces approximately 3.5M MMLU-style examples totaling roughly 1.6B tokens. Nemotron 3 Super by NVIDIA et al. (2026) reports that targeted pre-training ablations using this synthetic data improve several knowledge, mathematics, and code evaluations without material regressions elsewhere.

Ultra data mixture

  • Ultra expands the corpus to nineteen high-level categories and incorporates refreshed code, legal, synthetic, web, document, multilingual, and reasoning data. Approximately 49% of phase-1 tokens and 38% of phase-2 tokens come from filtered or synthetic web-crawl categories. The reduction in crawl share during phase 2 creates room for higher-quality document and specialized data.

  • Code remains a major component at 14% in both Ultra phases. FinePDFs shifts from 4.9% unfiltered data in phase 1 to 10.7% high-quality plus 2.5% medium-quality FinePDFs in phase 2. Synthetic medium-quality crawl falls from 10.3% to 3.8%, while the high-quality synthetic crawl component remains 22.4%. Phase 2 also introduces a small legal component and increases general SFT-style data from 1.6% to 4.0%.

  • Ultra further refreshes GitHub code with 173B new tokens through September 30, 2025 and adds multiple legal datasets covering contract clauses, terms of service, and federal regulations. In a smaller-model ablation starting from a 14.9T-token Nano checkpoint, adding the legal data to a 100B-token continuation increases the reported average over more than 100 LegalBench subtasks from 64.6 to 74.7, providing an empirical basis for including domain-specialized data in the Ultra mixture.

Optimization and low precision

Warmup-Stable-Decay scheduling

  • All three core text models use Warmup-Stable-Decay rather than a full-horizon cosine schedule. MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies by Hu et al. (2024) introduced WSD as a training schedule consisting of an initial warmup, an extended constant-learning-rate phase, and a final decay phase, allowing the high-learning-rate portion of training to be extended without redefining the complete schedule.

  • A generic WSD schedule can be represented as

    \[\eta(t) = \begin{cases} \eta_{\max}\dfrac{t}{T_{\mathrm{warm}}}, & 0\leq t<T_{\mathrm{warm}},\\[6pt] \eta_{\max}, & T_{\mathrm{warm}}\leq t<T_{\mathrm{decay}},\\[6pt] D(t), & T_{\mathrm{decay}}\leq t\leq T, \end{cases}\]

  • The concrete schedules differ with scale:

    • Nano: warmup over 8.4B tokens to \(10^{-3}\), a long high-learning-rate phase, and decay over the final 5T tokens to \(10^{-5}\). AdamW uses weight decay \(0.1\), \(\beta_1=0.9\), and \(\beta_2=0.95\).

    • Super: warmup over 200B tokens to \(4.5\times10^{-4}\), followed by the stable phase and a 5T-token minus-square-root decay to \(4.5\times10^{-6}\). AdamW again uses weight decay \(0.1\), \(\beta_1=0.9\), and \(\beta_2=0.95\).

    • Ultra: warmup over 200B tokens to \(2.5\times10^{-4}\) and decay over the final 5T tokens to \(2.5\times10^{-6}\). The remaining main-run hyperparameters largely follow the Super recipe unless explicitly modified.

Batch construction and MoE balancing

  • Nano and Super train initially at sequence length 8,192 with a global batch of 3,072 sequences, corresponding to approximately 25.2M tokens per optimizer step. This large token batch is important for efficient sparse expert execution because each step supplies enough routed tokens to populate the expert kernels effectively.

  • Expert load balance is maintained with a combination of bias-based routing control and a small conventional auxiliary loss. Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts by Wang et al. (2024) introduces dynamic expert-specific routing biases that adjust according to recent expert loads without injecting balancing gradients directly into the main training objective. Nemotron applies this mechanism with an update rate of \(10^{-3}\) while retaining a conventional load-balancing loss with coefficient \(10^{-4}\). Super additionally uses sigmoid routing scores with expert biasing.

  • This combination is important in large sparse models. Pure top-\(K\) routing can lead to some experts receiving disproportionately many tokens while others become underutilized. Dynamic expert biasing modifies which experts enter the top-\(K\) set without placing a large auxiliary gradient on the representations being learned, while the residual small load-balancing loss provides an additional stabilizing signal.

NVFP4 pre-training

  • Nano is principally trained in BF16, whereas Super is the first Nemotron 3 model to move the main pre-training workload into NVFP4. Ultra subsequently uses the same general low-precision training recipe at substantially larger scale. The objective is to exploit FP4 tensor-core throughput during forward propagation, activation-gradient computation, and weight-gradient computation, rather than training a conventional BF16 model and applying FP4 only after training. Nemotron 3 Super by NVIDIA et al. (2026) reports stable NVFP4 pre-training through 25T tokens, and Nemotron 3 Ultra by NVIDIA et al. (2026) extends the approach to the 550B-parameter scale.

  • The precision policy is deliberately heterogeneous rather than forcing every tensor into FP4:

Layer or operation Super pre-training precision Motivation
Most linear layers NVFP4 Maximize low-precision GEMM throughput
Final 15% of network BF16 Improve training stability
Latent projections BF16 Small step-time contribution, preserve fidelity
MTP layers BF16 Preserve future-token prediction quality
QKV and attention projections BF16 Preserve the few global attention layers
Mamba output projection MXFP8 Avoid observed NVFP4 underflow
Embeddings BF16 Preserve embedding fidelity

  • For NVFP4 matrix multiplications, weights use two-dimensional block scaling so that forward and backward passes operate on consistently quantized representations. Activations and gradients use one-dimensional blocks along the GEMM reduction axis. Inputs to weight-gradient computation undergo Random Hadamard Transforms, and stochastic rounding is applied to gradient tensors. The underlying NVFP4 representation uses E2M1 values in 16-element micro-blocks, E4M3 micro-block scales, and an additional FP32 global scale.

  • This mixed-precision arrangement illustrates a general principle of the Nemotron 3 stack: low precision is applied aggressively where it materially improves GEMM efficiency, but numerically sensitive or relatively cheap operations remain at higher precision.

Numerical-health monitoring

  • The Super run reveals a concrete consequence of pushing training to FP4. As training progresses, some expert-layer channels acquire very small magnitudes, and the corresponding gradients increasingly underflow to exact zero in NVFP4. By the end of pre-training, approximately 7% of parameter weight-gradient elements are reported as zero-valued. Controlled Nano-scale comparisons show that moving an NVFP4 run back to BF16 causes these zero-gradient counts to return toward baseline, supporting the interpretation that many representable small BF16 gradients are being quantized to zero under NVFP4. Nemotron 3 Super by NVIDIA et al. (2026) analyzes these channel-magnitude and underflow patterns as part of its pre-training stability validation.

  • Importantly, the presence of these zeros did not by itself imply training failure. Instead, the analysis motivated keeping several sensitive operations in BF16 or MXFP8 and explicitly validating low-precision runs against higher-precision continuation experiments. Ultra formalizes this validation further by branching checkpoints at 5T, 10T, and 16T tokens into BF16 and training them for an additional 74B tokens. The average relative training-loss gap between the NVFP4 trajectory and these BF16 continuations remains below 0.4% in the reported comparisons.

Checkpoint merging for evaluation

  • A distinctive part of the Super pre-training workflow is that intermediate model quality is not judged solely from individual checkpoints. During the stable WSD phase, benchmark scores fluctuate even though the underlying training trajectory continues to improve. Super therefore evaluates weighted offline merges of recent checkpoints as a smoother proxy for the model that would result after learning-rate annealing. Nemotron 3 Super by NVIDIA et al. (2026) uses this mechanism to assess checkpoints without repeatedly launching dedicated decay branches.

  • Checkpoints are saved approximately every 25B tokens. Sliding merge windows of 125B, 250B, and 500B tokens are evaluated, with merge weights chosen to emulate a minus-square-root learning-rate decay. Across twelve downstream benchmarks, the best offline merge typically exceeds its corresponding raw stable-phase checkpoint by approximately 2 to 4 average points. The approach can eliminate several expensive intermediate decay runs, estimated at roughly 4T tokens or approximately 16% of the pre-training FLOP budget for the described evaluation strategy.

  • The following figure (source) shows trained checkpoints versus the best offline checkpoint merge throughout Super pre-training. The advantage of merging is largest during the constant-learning-rate phase and narrows as the actual final learning-rate decay takes effect.

  • The final Super base checkpoint selected for alignment is itself a 500B-token merge. Ultra adopts the same concept throughout training, using a 500B sliding window at 25B-token checkpoint intervals. At the end of Ultra pre-training, merges spanning 125B to 1T tokens and different checkpoint orderings are evaluated; a 500B-window merge with a balanced knowledge, mathematics, and code profile is selected for long-context continuation.

Long-context continuation

  • The base pre-training runs use much shorter sequences than the final advertised context window. Rather than training the complete 20T to 25T-token run at hundreds of thousands or millions of tokens per sequence, Nemotron 3 performs a comparatively short long-context continued-pretraining stage after the main base-model training. This avoids paying long-context training costs throughout the entire pre-training horizon. NVIDIA Nemotron 3: Efficient and Open Intelligence by NVIDIA et al. (2025) also notes that the Mamba-heavy architecture does not depend on RoPE in its attention layers, avoiding one common source of positional extrapolation difficulty during context extension.

Nano long-context continuation

  • Nano’s long-context phase uses 121B tokens with a constant learning rate of \(10^{-5}\) and global batch size 48. Training runs on H100 GPUs with 8-way context parallelism, 8-way tensor parallelism, 8-way expert parallelism, and 4-way pipeline parallelism. The data blend consists of 20% long-document QA, 1% synthetic retrieval-oriented examples, and 79% downsampled phase-2 pre-training data.

  • A key implementation finding is that training exclusively on 512K sequences slightly degrades short-context benchmarks. The final recipe therefore mixes 512K and 4K sequences. This restores or improves shorter-context tasks such as MMLU-Pro and code evaluation while preserving the gains from long-context training.

Super long-context continuation

  • Super extends the procedure directly to 1M-token sequences. Its long-context stage uses a constant learning rate of \(4.5\times10^{-6}\) and global batch size 16, with 64-way context parallelism, 2-way tensor parallelism, and 64-way expert parallelism on GB200 GPUs.

  • The first stage trains exclusively at 1,048,576-token context for 34B tokens. A second 17B-token stage alternates between 1M-token and 4K-token sequences after the pure long-context stage produces a minor regression on mathematics benchmarks. The data blend consists of 20% long-document QA and 80% downscaled phase-2 pre-training data. Thus, Super’s long-context extension totals 51B tokens, only about 0.2% of its 25T-token main pre-training horizon.

Ultra long-context continuation

  • Ultra’s long-context phase is shorter still at 33B tokens, but it directly trains at 1M tokens for most iterations. It uses a constant learning rate of \(2.5\times10^{-6}\) and trains on GB200 GPUs using 32-way context parallelism, 8-way tensor parallelism, 128-way expert parallelism, and 2-way pipeline parallelism.

  • The Ultra blend consists of 46% long-context data and 54% phase-2 data. No RULER-style examples are included. Instead, 92% of iterations use 1M-token sequences and 8% use 4K sequences, with each iteration restricted to one sequence-length regime rather than mixing lengths within the same optimizer step. The 4K iterations use mathematics and code SFT-style data specifically to preserve short-context reasoning metrics. Each iteration processes 25,165,824 tokens.

  • The evolution from Nano to Ultra therefore converges on a general recipe:

    \[\text{Main pre-training at moderate sequence length} \;\longrightarrow\; \text{Select / merge a strong base checkpoint} \;\longrightarrow\; \text{Long-context CPT} \;\longrightarrow\; \\ \substack{\text{Mostly very-long sequences}\\+\;\text{small fraction of short sequences}} \;\longrightarrow\; \text{Preserve short-context capability while learning 1M context}\]

  • The short-sequence interleaving is an important implementation detail. Context extension is not treated as a monotonic sequence-length curriculum. Instead, long and short contexts are deliberately revisited to reduce catastrophic specialization toward the long-sequence regime.

Nemotron 3.5 continued training

MTP warmup before model distillation

  • Nemotron 3.5 Super and Lightning: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Models for Agentic AI by NVIDIA et al. (2026) begins the 3.5 training path by adapting the internal MTP drafter before changing the backbone. Both models receive a dedicated 100B-token MTP warmup with backbone parameters frozen. Super initializes a single recursively shared MTP head from the first of Nemotron 3 Super’s two original heads and discards the second; Lightning initializes from the Super 3 Base MTP module after fusing LatentMoE projections and pruning the hidden, expert, and routed-expert dimensions to the smaller 30B-A3B architecture.

  • Recursive drafting creates a train-inference conditioning mismatch because later MTP steps see ground-truth-derived hidden states during standard teacher-forced training but increasingly synthetic MTP-generated states at inference. Super 3.5 addresses this with hidden-state mixing: MTP step \(k\) samples input states from the set of earlier MTP steps \(1,\ldots,k-1\) rather than depending only on step \(k-1\). The goal is to expose the drafter to inference-like hidden-state mixtures before speculative decoding is used in deployment.

One-trillion-token teacher distillation

  • After MTP warmup, both base models undergo 1T tokens of distillation from the next-larger Nemotron form factor. Lightning 3.5 is distilled from Nemotron 3 Super Base using top-128 teacher logits plus a sink token representing residual vocabulary probability mass. Its objective combines forward KL and hard-token cross-entropy with weights 0.9 and 0.1, respectively; the learning rate rises from \(10^{-5}\) to \(10^{-4}\) over 200B tokens and linearly returns to \(10^{-5}\) over the remaining 800B tokens.

  • Super 3.5 is distilled from Nemotron 3 Ultra Base using the same 1T-token data horizon but replaces forward KL with the skew reverse-KL family introduced in DistiLLM: Towards Streamlined Distillation for Large Language Models by Ko et al. (2024), using skew parameter \(\alpha=0.1\). Its learning-rate anchors are inherited from Super 3 Base, with \(4.5\times10^{-6}\) as the minimum and \(4.5\times10^{-5}\) as the maximum.

Indic continued pre-training in Super 3.5

  • Super then receives a 100B-token Indic continued-pretraining stage. Ten billion tokens are allocated to selected synthetic and natural Indic text, while 90B tokens preserve the preceding training mixture’s relative proportions. The broader Nemotron-Pretraining-Indic-v1 corpus contains approximately 2.83T synthetic output tokens across 22 Indian languages and 34 language-script configurations, substantially more than is consumed by the actual Super CPT run.

  • For twelve priority languages, English source documents are translated as complete documents with a 4,096-token source limit, scored for translation quality on a 0-to-100 scale, retained at scores of at least 75, and otherwise rewritten using the source plus identified errors before a second quality check. Native-script outputs can then be romanized and passed through the same check-rewrite-check logic. Ten additional lower-resource languages use a separate translation backbone with quality assessment and language-specific thresholds.

  • The following figure (source) shows the four-stage Indic synthetic-data pipeline: document translation, translation-quality scoring, targeted rewriting of low-scoring candidates, final quality verification, and a parallel romanization path with retained provenance metadata.

  • The Indic CPT stage improves all five reported Indic evaluations, with gains ranging from 4.98 to 9.82 percentage points, while the tested English benchmark categories do not regress. Separate 200B-token Lightning research ablations evaluate 10%, 20%, and 30% Indic mixtures, but the released Lightning 3.5 checkpoint does not include this Indic CPT stage.

Long-context continuation in Super and Lightning 3.5

  • Both models perform 1M-context continued pre-training using the Ultra long-context data blend. Lightning trains for 21B tokens at a constant learning rate of \(10^{-5}\) using 8-way context, tensor, and expert parallelism plus 2-way pipeline parallelism on GB200. Super trains for 35B tokens at \(4.5\times10^{-6}\) using 16-way context parallelism, 8-way tensor parallelism, and 64-way expert parallelism on GB300.

Software-engineering mid-training as a distinct stage

  • Super 3.5 introduces a 65.4B-token software-engineering mid-training stage at 256K context. The motivation is to occupy the gap between short-context pre-training, which sees little agent interaction, and SFT, which is dominated by synthetic behavior-shaping data. Mid-training instead emphasizes natural software-engineering artifacts at long context while keeping explicit agent traces as a minority of the data, leaving most behavior shaping to later post-training. Lightning 3.5 does not use this stage.

  • The token mixture is 41% issue-pull-request pairs with repository context and merged patches, 20% whole repositories, 10% agentic trajectories, 3% commit diffs and edit histories, 1% build and CI logs, 20% Phase-2 pre-training replay, and 5% long-context data. Tool responses in agent traces are loss-masked so that supervision applies to the model’s own turns, and the sources are decontaminated against SWE-bench and Terminal-Bench.

  • Mid-training resets the learning rate, warms linearly to \(10^{-5}\), and then holds it constant. In a controlled comparison followed by the same experimental SFT, DeepSWE pass@1 rises from 6.4% to 10.4%. The larger effect is robustness across agent harnesses: across five non-OpenHands harnesses, the mid-trained checkpoint improves SWE-bench Verified by 8.6 points on average and Terminal-Bench 2.1 by 6.8 points on average; under the Codex harness, SWE-bench Verified rises from 48.4% to 61.8%.

  • The 256K software-engineering stage reduces RULER-1M from 79.9 to 33.5, illustrating that capability can regress when the active sequence-length distribution moves away from the deployment target. Super therefore performs a final 9B-token 1M-context recovery stage using the same long-context recipe, restoring RULER-1M to 81.0.

Ultra-scale stability

  • Ultra provides unusually detailed evidence of the numerical and optimization problems that can emerge at 550B total parameters. Two major divergence events occur during the main pre-training run. Nemotron 3 Ultra by NVIDIA et al. (2026) reports that both are visible as simultaneous increases in training cross-entropy and weight-gradient norm.

  • The first divergence occurs at approximately 8T tokens and is traced to a throughput optimization that reduced local gradient accumulation precision for the shared output layer from FP32 to BF16. Ultra’s two MTP blocks each contribute a comparatively small weighted gradient because the total MTP loss coefficient is only \(0.1\), or approximately \(0.05\) per MTP block. With BF16’s limited mantissa precision, these smaller MTP weight-gradient contributions can disappear when accumulated together with the larger main-language-model gradient. MTP-2 begins spiking before the primary training and validation losses, providing an early signal of the instability. Reverting to full FP32 gradient reduction restores stable training.

  • The second divergence occurs near 16T tokens. Extensive ablations do not isolate a single root cause, but beginning learning-rate annealing immediately after rolling back to the 15T-token checkpoint prevents the divergence under both investigated 5T and 10T decay horizons. The practical response is to shorten Ultra’s planned pre-training horizon and finish at 20T tokens.

  • The following figure (source) shows Ultra’s training and validation loss trajectories, including the two divergence regions, the rollback with restored FP32 gradient reduction, and the final annealed training trajectory.

  • The episode illustrates why low-precision training cannot be understood only in terms of forward-pass loss parity. A model can show small NVFP4-versus-BF16 loss gaps while still encounter scale-dependent numerical failures in gradient accumulation, auxiliary objectives, routed experts, or distributed reductions. Nemotron 3 therefore combines low-precision kernels with selective high-precision paths, BF16 continuation experiments, gradient-health monitoring, checkpoint rollback, and explicit numerical ablations.

Pre-training as hardware-software co-design

  • Taken together, the Nemotron 3 pre-training recipe is designed around the same systems constraints as its architecture. Sparse experts require active load balancing; NVFP4 reduces matrix-multiplication cost but requires careful protection of sensitive layers; MTP increases training supervision while adding a numerically smaller auxiliary gradient; WSD permits long stable-phase training and supports checkpoint-merging-based evaluation; and 1M-context capability is learned in a comparatively small CPT phase rather than imposed on the entire pre-training run.

  • The progression from Nano to Super to Ultra also shows that scaling is not achieved by mechanically reusing one recipe. The data-quality phase grows from 6% to 25% of pre-training, the peak learning rate falls with model scale, long-context parallelism changes substantially, low-precision safeguards become more selective, and Ultra’s training horizon is shortened in response to observed optimization instability. Nemotron 3 Nano by NVIDIA et al. (2025), Nemotron 3 Super by NVIDIA et al. (2026), and Nemotron 3 Ultra by NVIDIA et al. (2026) collectively present pre-training as an iterative combination of data curriculum, optimization, numerical precision, sparse routing, and distributed systems design.

Post-training

Pipeline overview

Stage Nano Super Ultra
General SFT Yes Two-stage Two-stage
Reasoning on/off Yes Yes Yes
Explicit reduced-effort mode No Low-effort Medium-effort
Budget control Yes Yes Yes
Multi-environment RLVR Synchronous Asynchronous, scaled Asynchronous, scaled further
Agentic RL Included in mixed RL PivotRL + dedicated SWE-RL Domain-specific teachers + RL
RLHF GenRM Principle-following GenRM Specialized chat teacher
Multi-teacher distillation No No MOPD
MTP repair after alignment No MTP healing MTP Boosting
  • This progression reflects a broader shift in the optimization target. Nano establishes broad reasoning and agentic competence; Super makes long-horizon interaction a major part of the training distribution; Ultra separates capability development into specialized teachers and then attempts to combine those capabilities through on-policy distillation.

Recurring stage pattern

  • Although the exact recipe changes across Nano, Super, and Ultra, the practical workflow follows a recurring progression from demonstration, to exploration, to consolidation.

  • SFT provides the initial foundation. A common construction pattern is generate-then-filter: begin with diverse prompts drawn from public or synthetic sources, sample several candidate trajectories from one or more strong teacher models, verify them with domain-appropriate checks, and retain only useful trajectories. Deterministic verification is preferred when possible. Mathematics can use exact-answer or symbolic checks, code can use unit tests, formal proofs can use a compiler or proof checker, and structured tasks can use schema or state validation. Open-ended tasks instead require a reward model or LLM judge.

  • A single prompt can yield several valid SFT examples rather than only one. When multiple completions are correct, diversity becomes useful: near-duplicate reasoning traces can be removed, different valid strategies can be retained, and trajectories can be drawn from a pool of teacher models rather than from one teacher alone. This reduces the risk that the SFT distribution collapses around one stylistic or reasoning pattern. The same generation and filtering pipeline can also be refreshed as stronger teachers become available.

  • RL then shifts the source of trajectories from the teacher toward the current policy. The policy samples its own solutions, receives task-specific rewards, and explores behaviors that may not appear in the supervised data. In this stage, prompt selection becomes as important as the policy-gradient algorithm. Prompts that are already solved almost always provide little learning signal, while prompts that are almost never solved may be noisy, infeasible, or too difficult for the current policy. Nemotron therefore profiles prompt difficulty and uses curriculum mechanisms so that the training distribution evolves with the policy.

  • At larger scale, the challenge becomes consolidation. A single mixed RL run must divide its finite rollout budget across many capabilities, and the domains may interfere with one another or impose incompatible environment costs. Ultra addresses this by allowing specialized teachers to become strong in narrow domains and then using MOPD to merge those skills back into one general student. The resulting post-training system is therefore better viewed as a capability-acquisition pipeline than as one monolithic alignment stage.

Supervised fine-tuning

Generate, verify, diversify, and refresh

  • The practical objective of SFT data construction is not simply to maximize the number of demonstrations. It is to produce trajectories that are correct, diverse, structurally valid, and useful for the behaviors expected during later RL or deployment.

  • A generic SFT generation pipeline can be written as:

    \[x \sim \mathcal{D}_{\mathrm{prompt}}, \qquad y_{1:K} \sim \pi_{\mathrm{teacher}}(\cdot\mid x)\]

    • followed by domain-specific verification:

      \[v(x,y_k) \in \{0,1\}\]

      • or a graded judge score. The retained set is then:

        \[\mathcal{S}(x) = \operatorname{SelectDiverse} \left( \left\{ y_k : v(x,y_k)\ \text{passes} \right\} \right)\]

  • SelectDiverse is important because several teacher samples can be correct while following nearly identical reasoning paths. Retaining only duplicate solutions wastes dataset capacity. Conversely, preserving multiple genuinely different valid strategies can improve behavioral coverage and reduce overfitting to one canonical trace.

  • This pattern appears in several Nemotron data pipelines. Formal-proof examples retain compiler-verified proofs, CUDA examples must compile and numerically match a reference, software-engineering examples execute in repository environments, and Super’s long-context construction generates multiple reasoning traces, groups them by semantic answer, and retains a concise trace from the dominant answer group. The general principle is that generation creates candidate behavior, while verification and diversity selection determine what becomes supervision.

Nano: agentic SFT rather than conventional instruction tuning

  • Nano’s SFT distribution is explicitly oriented toward multi-step and multi-turn behavior. It includes competition mathematics, code, tool-integrated reasoning, conversational tool use, long-context reasoning, formal proofs, terminal interaction, software engineering, science, instruction following, general chat, multilingual data, safety, and generative solution selection. Nemotron 3 Nano by NVIDIA et al. (2025) reports more than 18 million SFT samples and uses dataset-dependent sampling rates rather than training each source for the same number of epochs.

  • The largest portions of the reported blend are chat at 28.6%, code at 20.7%, science at 12.8%, mathematics at 9.9%, multilingual data at 7.4%, and mathematics with tools at 4.9%. Smaller but strategically important components include software engineering, conversational agents, terminal use, formal proofs, and long-context reasoning.

  • The following figure (source) shows the SFT data blend for Nemotron 3 Nano, illustrating how general conversation and code are combined with explicit reasoning, tool-use, software-engineering, formal-proof, long-context, and multilingual supervision.

  • The SFT run uses 13,000 optimization steps with global batch size 64, sequence packing to 256K tokens, a learning rate of \(5\times10^{-5}\), 800 warmup steps, and a sequence-level MoE load-balancing regularizer with coefficient \(10^{-4}\).

Preserving reasoning across agent steps

  • The Nano chat template distinguishes multi-step interactions within a turn from separate user turns. During a tool-using step sequence, reasoning generated earlier in the same user turn can remain available to subsequent assistant steps. Once a new user turn begins, previous-turn reasoning is removed from the materialized prompt. This allows tool-using agents to reuse relevant intermediate reasoning without indefinitely accumulating reasoning traces across the entire conversation.

  • Tool calls are represented using XML-style special tags, reducing escaping complexity and providing a structured boundary between reasoning, assistant content, tool calls, and tool observations.

Verifiable and filtered SFT trajectories

  • A recurring principle is to retain trajectories that can be checked either automatically or by a judge. Formal proof data, for example, begins with roughly 580K natural-language mathematical theorems, converts them into Lean statements, generates candidate proofs, and retains compiler-verified solutions. The final reported formal-proof set contains approximately 300K successful examples.

  • Software-engineering examples use real repository issues and containerized environments, while trajectories are distilled through multiple agent harnesses. CUDA examples are retained only when generated CUDA code compiles and numerically agrees with the PyTorch reference. Conversational tool-use trajectories are filtered by judges that assess whether user, agent, and environment actions remain consistent with their intended goals.

  • Across domains, malformed examples, pathological repeated reasoning, invalid tool structures, and other detected synthetic-generation failure modes are removed before SFT.

Reasoning control in SFT

  • Nemotron 3 does not treat reasoning length purely as an emergent inference behavior. It explicitly modifies the SFT distribution so that the model learns several reasoning regimes.

  • For Nano, reasoning traces are removed from a random 10% of examples to teach reasoning-off behavior. Another 3% of reasoning traces are truncated to randomly selected reasoning budgets while retaining the original post-reasoning answer. This teaches the model to transition from incomplete internal reasoning into a final answer instead of assuming that every reasoning trace must run to its natural termination. Nemotron 3 Nano by NVIDIA et al. (2025) introduces both reasoning on/off and token-budget control in this way.

  • Super extends the control space to three modes: reasoning-off, regular reasoning, and low-effort reasoning. Low-effort examples constitute approximately 2% of SFT samples and initially cover mathematics, STEM, and instruction following. For reasoning-off training, Super removes reasoning from 3% of examples. It then runs a short 350-step semi-on-policy stage for budget control, generating model rollouts and truncating 12% of their reasoning traces to random budgets. Nemotron 3 Super by NVIDIA et al. (2026) uses this combination to make reasoning mode and reasoning budget independently controllable.

  • Ultra replaces low-effort with medium-effort reasoning. Medium-effort demonstrations cover mathematics, STEM, and instruction following. Budget-truncated examples are again included, but Ultra masks the closing reasoning delimiter from the SFT loss for truncated samples, preventing the model from learning that a randomly truncated reasoning path necessarily corresponds to a natural reasoning endpoint. Nemotron 3 Ultra by NVIDIA et al. (2026) introduces this modification.

Two-stage SFT in Super

  • Super identifies a specific failure mode of ordinary token-averaged SFT: long outputs contribute many more loss terms than short outputs, so heavily reasoning-oriented data can cause the model to become disproportionately optimized for long responses and degrade long-input, short-output behavior. Nemotron 3 Super by NVIDIA et al. (2026) addresses this with a two-stage loss-normalization strategy.

  • For output token position \(t\) in conversation \(c\), define:

    \[\ell_t = -\log p_{\theta} \left( y_t \mid x,y_{<t} \right)\]

  • Stage 1 averages over all output tokens in the packed global batch:

    \[\mathcal{L}_{\mathrm{tok}} = \frac{ \displaystyle \sum_{c\in\mathcal{B}} \sum_{t\in\mathcal{O}_c} \ell_t }{ \displaystyle \sum_{c\in\mathcal{B}} |\mathcal{O}_c| }\]

  • This gives long outputs proportionally greater influence because they contribute more tokens. Stage 2 instead normalizes inside each conversation before averaging conversations:

    \[\mathcal{L}_{\mathrm{samp}} = \frac{1}{|\mathcal{B}|} \sum_{c\in\mathcal{B}} \left( \frac{1}{|\mathcal{O}_c|} \sum_{t\in\mathcal{O}_c} \ell_t \right)\]

  • The second objective assigns equal aggregate weight to each conversation regardless of output length. Super reports that this recovers long-input, short-output performance while retaining the reasoning behavior learned during Stage 1.

  • Stage 1 uses 256K packed sequences, global batch size 64, a constant learning rate of \(10^{-5}\), and 30K warmup samples. Stage 2 increases packing to 512K, includes long-context examples up to 512K, uses global batch size 32, and retains the same learning rate. The shared-weight MTP heads continue training during SFT with an auxiliary scaling factor of 0.3.

Agentic SFT scaling in Super

  • The SFT corpus for Super grows to more than 7 million samples while significantly increasing the relative share and diversity of agentic data. Long-context examples are constructed at 128K, 256K, and 512K lengths, with questions explicitly designed to require cross-document navigation and approximately four to seven retrieval or reasoning steps. Multiple reasoning traces are generated for each question, semantically grouped by answer, and the shortest trace from the majority answer group is retained.

  • Software-engineering SFT uses real GitHub issues and containerized repositories, while a separate agentic programming pipeline creates approximately 20K seed CLI queries spanning 24 action classes, augments them with AGENTS.md-style specifications, mixes them with approximately 3K repository-based SWE tasks and 10K web-development tasks, and records trajectories from multiple agentic models and CLI harnesses.

  • The following figure (source) shows the complete Nemotron 3 Super post-training sequence, including SFT, multiple RLVR rounds, agentic-focused training, dedicated software-engineering RL, RLHF, and MTP healing.

Ultra SFT and long-sequence packing

  • Ultra retains the two-stage structure but adjusts sequence lengths and optimization. Stage 1 packs sequences to 294,912 tokens, uses global batch size 64, trains over 204,800 samples, and applies cosine learning-rate decay from a peak of \(1.5\times10^{-5}\) to \(10^{-6}\). Stage 2 increases the packed length to approximately 515K tokens, injects additional 512K-context examples, and trains another 19,200 samples with a peak learning rate of \(10^{-5}\) and minimum of \(2\times10^{-6}\). Two shared MTP layers remain active with auxiliary scaling factor 0.1.

  • Ultra uses length-aware best-fit packing instead of naive concatenation. Each incoming conversation is assigned to the partially filled sequence whose remaining capacity best matches its length. Conversations are neither split nor truncated. A fixed-size pool of open packs limits memory usage, identical prompts are prevented from appearing inside the same pack, and completed packs are shuffled to avoid source-locality effects.

  • This matters at context lengths approaching half a million tokens: poor packing can waste large fractions of training compute on padding, while naive per-dataset concatenation can produce long stretches dominated by a single source distribution.

Reinforcement learning from verifiable rewards

Why train environments jointly

  • The defining RL choice in Nemotron 3 is to optimize across heterogeneous environments in a unified training mixture rather than sequentially optimizing one domain at a time. Nano reports that single-environment RL often produces unrecoverable regressions on unrelated benchmarks, whereas simultaneous multi-environment RL yields substantially smoother capability improvements.

  • This creates a multi-task RL problem in which a batch can contain mathematical reasoning, code, question answering, structured output generation, instruction following, long-context QA, and agentic tool-use tasks. Rewards may come from deterministic unit tests, schema validation, database-state comparison, proof checkers, or LLM judges depending on the environment.

  • DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models by Shao et al. (2024) introduces Group Relative Policy Optimization, which removes the learned value model used by PPO and estimates relative advantages from groups of sampled responses. Nemotron 3 adopts GRPO as the core policy-optimization mechanism, with additional masking and asynchronous-training modifications at larger scale.

Scaling multi-domain RL: optimization and systems tension

  • Unified RL becomes harder as more domains are added. Each training batch has finite rollout capacity, so every additional environment reduces the number of samples available to the others unless the total batch grows proportionally. This can weaken the learning signal for individual capabilities. Updates can also interfere: a gradient step that improves one domain can degrade another, and reward scales or verifier characteristics can differ substantially across tasks.

  • The systems problem is equally important. A short mathematics rollout may finish quickly and be verified with an exact-answer check, while a software-engineering trajectory can require many tool calls, hundreds of interaction steps, a repository sandbox, and a large hidden test suite. Other domains may require separate judge models or reward models to be hosted during training. If these trajectories share a synchronous batch, the wall-clock duration is driven by the slowest examples \(T_{\mathrm{batch}} \approx \max_i T_i\), so finished workers can sit idle while a small number of long-running trajectories continue. These stragglers motivate asynchronous rollout generation, dedicated environment pools, and in some cases separating especially expensive domains from the main mixed-environment run.

  • This explains why post-training quality cannot be reduced to the choice between PPO, GRPO, or another policy objective. Dataset composition, environment latency, verifier cost, sampling policy, and scheduler design can determine how much useful learning signal reaches the model for a fixed training budget.

Unified RL versus cascade RL

  • There are two broad ways to manage heterogeneous domains. Unified multi-environment RL, used throughout Nemotron 3, samples several capabilities within the same overall optimization process. Its advantage is that the policy is continually exposed to the complete mixture, reducing the risk that a later stage overwrites an earlier capability. The Nemotron 3 white paper reports that simultaneous multi-environment training is more stable than earlier staged approaches and is less prone to capability degradation.

  • A different strategy is cascade or sequential RL, where incompatible capability groups are optimized in separate stages. Nemotron-Cascade 2: Post-Training LLMs with Cascade RL and Multi-Domain On-Policy Distillation by Yang et al. (2026) develops this direction together with multi-domain on-policy distillation. Sequentialization makes each stage easier to optimize and can isolate domains with very different rollout or verifier requirements, but capabilities learned early can regress as later stages move the policy distribution.

  • The trade-off is therefore structural rather than purely algorithmic:

    \[\text{unified RL} \Rightarrow \text{harder balancing and orchestration, less sequential forgetting}\]

    • while:

      \[\text{cascade RL} \Rightarrow \text{simpler per-stage optimization, greater pressure to recover earlier skills}\]

  • MOPD provides a third mechanism: train specialists independently, then consolidate them through dense teacher supervision rather than requiring every domain to compete directly inside one RL batch.

Difficulty-aware RL curriculum

  • Before RLVR, Nano profiles each training task using the SFT checkpoint. Tasks for which the model already obtains a 100% pass rate are removed. Remaining examples are organized according to observed pass rate, which acts as a proxy for difficulty.

  • This profiling can be understood as an empirical estimate of the current policy’s success probability. For prompt \(x\), sample \(K\) rollouts and compute:

    \[\widehat{p}_{\mathrm{pass}}(x) = \frac{1}{K} \sum_{k=1}^{K} \mathbf{1} \left[ R(x,y_k)\ \text{passes} \right]\]

  • Prompts with \(\widehat{p}_{\mathrm{pass}}(x)=1\) are often removed because all sampled responses already succeed and therefore provide little relative learning signal. Very low-pass-rate prompts can also be poor early-training choices because they may be noisy, unsolvable, or outside the policy’s current exploration frontier. The useful region moves as the policy improves, which motivates re-profiling the prompt pool rather than treating difficulty as a static metadata field.

  • Within each domain, the curriculum defines a Gaussian target distribution over pass rates. Early training emphasizes higher-pass-rate examples, and the target mean decreases over training so that increasingly difficult examples are sampled. Domain ratios remain fixed, preserving cross-domain balance while difficulty changes within each domain. Once performance plateaus, the task pool is re-profiled using the improved RL checkpoint and a new curriculum is built.

  • This distinction is important: the curriculum does not simply increase the fraction of one difficult domain. It changes difficulty within each domain while maintaining a controlled mixture across domains.

Nano’s synchronous GRPO implementation

  • Nano uses synchronous GRPO with masked importance sampling. Each optimizer step samples 128 prompts and produces 16 generations per prompt, giving an effective training batch of 2,048 trajectories. Because the complete group is generated before the update, the reported setup remains on-policy. Maximum generation length is 49K tokens.

  • The MoE router parameters are frozen during RL, while loss-free expert balancing continues updating expert-routing bias values. This isolates policy optimization from potentially destabilizing changes to the sparse-routing network.

  • Nano executes two multi-environment RLVR stages, one after SFT and another after RLHF. In experiments comparing RLVR to a substantially longer SFT run, the RLVR trajectory matches or exceeds the more heavily fine-tuned checkpoint across the tested domains, motivating the use of verifiable RL rather than simply increasing SFT epochs.

Super scales RL breadth and switches to asynchronous optimization

  • Super grows the RL mixture to 21 environments and 37 datasets spanning mathematics, code, STEM, formal proofs, instruction following, safety, long context, conversational tool use, terminal interaction, and general reasoning. The same difficulty-aware curriculum is retained, including filtering prompts that the SFT checkpoint already solves consistently.

  • Unlike Nano, Super decouples rollout generation from model training. Inference workers continuously produce trajectories and place them in a rollout buffer, while separate learner GPUs consume completed batches. Updated weights are pushed to rollout workers as soon as a new policy becomes available. The system permits at most one learner-step of policy lag.

  • Each RLVR step samples 256 prompts with 16 responses per prompt, producing a batch of 4,096 trajectories. Generation initially supports up to 49K tokens and later extends to 64K.

  • This is an explicit throughput versus on-policy trade-off. Synchronous RL avoids stale trajectories but stalls the learner while generation completes. Asynchronous RL overlaps these phases, but some trajectories originate from slightly older policies and therefore require mechanisms to limit or mask excessive importance-weight mismatch.

Low-effort reasoning during RL

  • Super does not learn low-effort behavior only from SFT demonstrations. During RLVR, a fraction of prompts are explicitly converted into low-effort mode and rewarded according to both correctness and output length. The initial mixture uses mathematics, STEM, and competitive coding and makes roughly 2% of RL prompts low-effort; later training reduces this to approximately 1% and focuses primarily on math and STEM.

  • The coding prompts used for low-effort optimization are held out from SFT. This provides a direct test of whether the model learns an underlying compute-control behavior rather than memorizing concise demonstrations. The resulting control generalizes beyond the task families explicitly included in the low-effort RL subset.

Dedicated end-to-end software-engineering RL

  • Super separates software-engineering RL from the mixed-environment stage because SWE trajectories are substantially longer, involve repository state and external execution, and would become stragglers if mixed directly with shorter reasoning problems.

  • During SWE-RL, each rollout launches an isolated Apptainer container containing the target repository. A modified OpenHands agent executes a multi-turn tool loop, produces a patch, and the environment runs hidden ground-truth tests to assign a binary reward. Tool diversity is introduced through OpenCode- and Codex-style agent interfaces implemented over the same underlying OpenHands runtime.

  • Container execution includes explicit safeguards because Apptainer shares the host kernel. A memory watchdog kills runaway agent processes before they exhaust node memory, and a command blocklist intercepts dangerous operations such as indiscriminate process termination.

PivotRL for cheaper agentic post-training

  • PivotRL: High Accuracy Agentic Post-Training at Low Compute Cost by Yi et al. (2026) addresses the high cost of fully online long-horizon RL by reusing expert SFT trajectories and identifying informative intermediate turns, called pivots, where the current policy is uncertain about the next action. Local on-policy exploration is concentrated around these turns, and functionally equivalent actions can receive credit even when they do not exactly reproduce the demonstration.

  • Nemotron 3 Super applies PivotRL to agentic programming, search, terminal use, and conversational tool use. This creates an intermediate regime between static behavioral cloning and expensive end-to-end environment RL: the system retains SFT trajectories as scaffolding but applies RL at the decision points where the policy can most benefit from additional exploration.

Reinforcement learning from human feedback

Generative reward models

  • Nano uses a generative reward model rather than a conventional scalar Bradley-Terry reward model. The reward model receives a conversation and two candidate responses, reasons about their strengths and weaknesses, predicts an individual helpfulness score for each response, and predicts a preference ranking. The GenRM is initialized from Qwen3-235B-A22B-Thinking-2507 and itself trained with GRPO.

  • HelpSteer3: Human-Annotated Feedback and Edit Data to Empower Inference-Time Scaling in Open-Ended General-Domain Tasks by Wang et al. (2025) supplies part of the human-feedback data used in this reward-model pipeline, alongside a commercially usable subset of Arena preference data and synthetic safety examples.

  • For a training pair, Nano defines the GenRM reward as:

    \[R = -C_1 I_{\mathrm{format}} - |P_{h1}-G_{h1}| - |P_{h2}-G_{h2}| - C_2|P_r-G_r|\]
    • where the predicted helpfulness scores and ranking are compared with their corresponding labels. The format-violation term receives coefficient 10 and the ranking-error term coefficient 1.

Linear-cost pairwise comparison

  • During policy RLHF, each prompt produces 16 candidate responses. Comparing every possible response pair would require 120 GenRM comparisons for each prompt. Nano instead arranges responses in a cycle so that every response is compared only against its successor, with the final response compared against the first. This reduces the number of comparisons from quadratic in the group size to linear while keeping the response set connected through pairwise judgments. Each response appears in both ordering positions to reduce positional bias.

Group Relative Length Control

  • Optimizing only the GenRM score creates another failure mode: response length increases significantly because additional reasoning can improve reward even when open-ended chat questions do not require long reasoning chains. Nano introduces Group Relative Length Control to reward concision relative to other responses for the same prompt rather than imposing an absolute response-length threshold.

  • For reasoning length, the response-specific normalized preference for shorter reasoning is:

    \[w_i^{(\mathrm{think})} = 1- \frac{ \ell_i^{(\mathrm{think})} - \ell_{\min}^{(\mathrm{think})} }{ \ell_{\max}^{(\mathrm{think})} - \ell_{\min}^{(\mathrm{think})} }\]

  • The weights are centered within the response group:

    \[\widetilde{w}_i^{(\mathrm{think})} = w_i^{(\mathrm{think})} - \frac{1}{N} \sum_{j=1}^{N} w_j^{(\mathrm{think})}\]

  • The same construction is applied independently to final-answer length. The adjusted reward is:

    \[R_i = R_i^{(\mathrm{base})} + \lambda_{\mathrm{think}} \widetilde{w}_i^{(\mathrm{think})} + \lambda_{\mathrm{answer}} \widetilde{w}_i^{(\mathrm{answer})}\]

  • Both length coefficients are set to 0.5.

  • A further quality gate awards an additional concision bonus only when the shortest reasoning or answer belongs to the top 20% of responses by base reward. This avoids teaching the model that brevity itself is desirable when the short response is low quality. The reported procedure reduces verbosity by approximately 30% without reducing accuracy.

Super’s principle-following RLHF

  • Super retains a GenRM-based RLHF architecture but changes the reward model into a principle-following judge. Rather than evaluating only generic helpfulness, the reward model can judge responses against explicit principles for areas such as safety and identity. The GenRM is used during the broader multi-environment RL process and again in a dedicated final RLHF stage.

  • This makes reward-model behavior itself configurable: targeted principles can be changed between training rounds without necessarily retraining the complete reward model.

Multi-teacher on-policy distillation

Specialize first, consolidate second

  • MOPD changes the scaling strategy for multi-domain post-training. Rather than forcing one policy to become maximally strong in every domain under one mixed RL objective, separate teachers can be optimized independently. A software-engineering teacher can spend nearly all of its rollout budget on repository interaction, while a chat teacher can optimize open-ended conversational behavior and a search teacher can focus on long-running browsing and context management.

  • The student then generates the trajectories used for distillation. For each student token, the corresponding domain teacher evaluates how probable that action would be under the teacher policy. This creates a dense learning signal at every realized token rather than assigning one sparse outcome reward to the entire trajectory. Conceptually, RL is used to create specialists, while MOPD is used to consolidate their local policies into one general model.

  • Because the rollouts are generated by the student, the objective remains on-policy from the student’s perspective. This is a key distinction from ordinary offline distillation on a fixed teacher-generated corpus: the teacher is asked to supervise the states that the student actually visits. The benefit is stronger relevance of the supervision; the limitation is that the teacher cannot easily transfer a capability whose trajectories the student almost never reaches, which leads directly to the support-mismatch problem discussed below.

Why RLVR alone becomes insufficient

  • As the number of RL environments grows, every domain receives a smaller fraction of each mixed-environment batch. Ultra therefore separates two objectives: unified RLVR provides broad improvements across the complete task distribution, while specialized teachers push individual domains farther than a single balanced policy can efficiently reach. Nemotron 3 Ultra by NVIDIA et al. (2026) trains more than ten such teachers and consolidates them through MOPD.

  • Nemotron-Cascade 2: Post-Training LLMs with Cascade RL and Multi-Domain On-Policy Distillation by Yang et al. (2026)

  • DistiLLM: Towards Streamlined Distillation for Large Language Models by Ko et al. (2024) develops the preceding multi-domain on-policy distillation framework, using strong intermediate domain-specific teachers to recover regressions and combine capabilities across an RL training process.

  • Ultra generalizes this idea into an iterative student-teacher system. A unified RLVR student generates its own trajectories, domain-specific teachers score those trajectories at the token level, the student is updated, and new teacher rounds are then branched from the improved student. Two complete MOPD iterations are performed.

  • The following figure (source) shows the two-iteration MOPD pipeline. The first iteration distills STEM, chat, instruction-following, terminal-use, conversational-tool-use, software-engineering, search, office-work, usability, and agentic-safety teachers; the second refreshes several teachers from the improved MOPD1 student before producing the final Ultra checkpoint.

On-policy reverse-KL distillation

  • Let the student policy be \(\pi_{\theta}\) and let domain \(i\) have teacher policy \(\pi_{T_i}\) and prompt distribution \(\mathcal{D}_i\). For a student-generated trajectory, MOPD trains on prefixes induced by the student itself. The fully on-policy objective is

    \[\mathcal{J}_{\mathrm{MOPD}}(\theta) = \sum_{i=1}^{N} \lambda_i \, \mathbb{E}_{ q\sim\mathcal{D}_i,\, y\sim\pi_{\theta}(\cdot\mid q) } \left[ \sum_{t=1}^{H} \left( \log\pi_{T_i}(y_t\mid s_t) - \log\pi_{\theta}(y_t\mid s_t) \right) \right]\]

  • Equivalently, the student minimizes:

    \[D_{\mathrm{KL}} \left( \pi_{\theta}(\cdot\mid s_t) \parallel \pi_{T_i}(\cdot\mid s_t) \right)\]
    • at prefixes generated by the student.

  • This differs fundamentally from environment RLVR. A typical verifier may provide one sparse trajectory-level reward, such as whether a patch passes hidden tests. MOPD instead asks the teacher how compatible each realized student token is with the teacher’s policy, producing a dense learning signal throughout the trajectory.

Asynchronous MOPD

  • Ultra runs student rollout generation, teacher scoring, and learner updates as a pipeline. This introduces policy staleness, because a trajectory may have been generated by an older behavior policy while the learner has already advanced. Ultra therefore separates the stale behavior policy from a proximal reference policy used as the optimization trust-region center.

  • The sampled distillation advantage is:

    \[\widehat{A}_t = \operatorname{sg} \left[ \ell_t^{T_i} - \ell_t^{\mathrm{prox}} \right]\]
    • where the stop-gradient prevents backpropagation through the advantage construction.

  • The stale-policy correction and learner-policy ratio are:

    \[c_t = \operatorname{sg} \left[ \frac{ \pi_{\mathrm{prox}}(y_t\mid s_t) }{ \pi_{\mathrm{behav}}(y_t\mid s_t) } \right]\] \[r_t(\theta) = \frac{ \pi_{\theta}(y_t\mid s_t) }{ \pi_{\mathrm{prox}}(y_t\mid s_t) }\]

  • The asynchronous clipped objective is:

    \[\mathcal{J}_{\mathrm{async\text{-}MOPD}}(\theta) = \mathbb{E} \left[ \sum_{t=1}^{H} m_t c_t \min \left( r_t(\theta)\widehat{A}_t, \operatorname{clip} \left( r_t(\theta),1-\epsilon,1+\epsilon \right) \widehat{A}_t \right) \right]\]
    • where the token mask suppresses unreliable token-level updates.

  • MOPD uses generation lengths up to 192K tokens. Each training batch contains 1,024 prompts with one student rollout per prompt; experiments with multiple rollouts per prompt did not yield additional gains in the reported setup.

Domain-specialized teachers

  • The teacher pool is intentionally heterogeneous because different agentic domains benefit from different optimization procedures.

  • The software-engineering teacher uses SFT, PivotRL by Yi et al. (2026), and final end-to-end SWE-RL. Repository interactions can extend to 200 agent turns and 192K generated tokens. Unfinished trajectories are masked, malformed reasoning or tool calls receive negative token-level advantage, and the training environment removes future Git history and blocks network routes that could expose the gold patch.

  • The office-work teacher targets file-grounded professional tasks where success depends on producing artifacts such as documents, reports, or spreadsheets, rather than merely returning the correct textual conclusion. Training uses strong-model trajectories, light SFT, and subsequent pivot-style RL.

  • The search teacher explicitly learns context management for long-running research trajectories, including strategies that discard or compress old search observations as the context budget fills. This addresses a practical agent failure mode where noisy search results exhaust the context window before sufficient evidence has been collected.

  • The terminal-use teacher uses difficult long-timeout tasks and repeated PivotRL rounds. The conversational tool-use teacher expands sequential and dependent actions to reduce premature termination. The model-usability teacher covers structured schemas, document extraction, inline citations, and free-form formatting.

  • Ultra additionally trains a dedicated agentic-safety teacher against indirect prompt injection in tool observations. The environment embeds malicious instructions inside otherwise plausible tool responses, while deterministic verification checks whether the agent improperly executes the attacker’s requested action.

MOPD warmup and teacher-student support mismatch

  • A major empirical finding is that specialized teachers cannot always be merged effectively when their SFT histories differ strongly from the student’s. A teacher may know how to score trajectories in its own behavioral distribution, but student-generated trajectories can lie outside that distribution and therefore receive unreliable token-level guidance.

  • Ultra addresses this with a brief MOPD warmup. Before distillation from a teacher, the student performs light SFT on examples from that teacher’s training distribution. The goal is not to directly transfer the whole capability through SFT, but to shift the student’s trajectory distribution sufficiently close to the teacher’s support that subsequent on-policy teacher scores become meaningful.

  • The effect is especially large in agentic domains:

Benchmark RLVR student MOPD without warmup MOPD with warmup Teacher
GDPVal 28.9 35.3 46.7 49.5
BrowseComp 31.0 33.0 44.4 51.0
HLE 25.6 26.3 26.7 32.1
  • The weak HLE gain is informative. For agentic tasks, the teacher often improves the choices made along trajectories that the student can already generate. For hard self-contained reasoning, the teacher may have acquired entirely new reasoning paths through additional off-policy SFT and RL data. On-policy distillation cannot efficiently transfer a trajectory that the student almost never samples in the first place.

Capability recovery through MOPD

  • Ultra reports strong gains from RLVR to the second MOPD iteration across agentic and instruction-following tasks:
Benchmark SFT RLVR MOPD 1 MOPD 2 Specialized teacher
Terminal Bench 2.0 34.5 44.5 50.8 54.0 50.0
GDPVal 23.2 28.9 46.7 46.7 49.5
SWE-Bench Verified 63.5 65.8 70.1 71.7 72.5
TauBench Telecom 55.7 82.7 91.2 92.9 94.0
BrowseComp 14.3 31.0 41.0 44.4 51.0
OmniScience Non-Hallucination 4.8 46.3 77.9 78.7 87.0
IFBench 62.3 78.4 80.0 81.7 83.0
HLE 19.7 25.6 25.9 26.7 32.1
  • MOPD sometimes exceeds the relevant teacher, as in Terminal Bench and Multi-Challenge. The Ultra report attributes this to cross-domain transfer when signals from several specialized teachers are consolidated into a shared student. The smaller improvement on HLE is consistent with the support-mismatch limitation discussed above.

Inference-aware post-training

  • Ultra exposes three reasoning modes: reasoning-off, regular, and medium-effort. Regular and medium-effort can additionally be combined with explicit inference-time reasoning budgets. The resulting control plane separates two concepts: reasoning style determines the model’s learned tendency to expend compute, while the token budget imposes an explicit upper bound at inference.

  • Medium-effort behavior is seeded during SFT and refined during RLVR. Approximately 2.5% of Ultra’s RLVR prompts use medium-effort mode across mathematics, STEM, and coding, with length-aware reward modifications. The reported model generalizes this efficiency behavior to tasks outside the explicitly optimized domains.

  • On the reported Artificial Analysis Intelligence Index V4 comparison, medium-effort Ultra generates approximately 2.5 times fewer tokens than regular Ultra at a cost of roughly 7% lower accuracy. Nemotron 3 Ultra by NVIDIA et al. (2026) uses this result to illustrate that reasoning expenditure is an explicit operating point rather than a fixed property of the checkpoint.

  • The following figure (source) shows the accuracy-efficiency trade-off for Nemotron 3 Ultra’s regular and medium-effort reasoning modes relative to other open models.

MTP repair after policy training

  • Policy post-training changes the backbone distribution that the MTP drafter is expected to approximate. Super therefore ends its pipeline with MTP healing: backbone parameters are frozen and the MTP heads are retrained on responses generated from RLVR prompts using a next-token likelihood objective. This recovers speculative-drafting accuracy without altering the aligned policy itself.

  • Ultra develops this into MTP Boosting. Instead of ordinary teacher-forced MTP training, it deliberately exposes the draft head to mixtures of backbone and recursively MTP-generated hidden states, better approximating the distribution encountered during autoregressive speculative decoding. Only the MTP head is updated, leaving the post-trained backbone fixed.

  • The boosted head is trained for 12K steps with global batch size 64 on on-policy responses capped at 8K tokens. Rather than cross-entropy against one-hot target tokens, it minimizes a temperature-scaled forward KL divergence against the frozen backbone distribution:

    \[\mathcal{L}_{\mathrm{MTP}}(\theta) = \frac{ T^2 }{ N_{\mathrm{mtp}}|\mathcal{A}| } \sum_{k=1}^{N_{\mathrm{mtp}}} \sum_{t\in\mathcal{A}} D_{\mathrm{KL}} \left( \sigma \left( \frac{z_{t+k}}{T} \right) \parallel \sigma \left( \frac{z^{\mathrm{mtp},k}_{t+k}}{T} \right) \right)\]

  • Ultra uses temperature 2 and seven recursive MTP steps.

  • The resulting average accepted draft length on SPEED-Bench increases from 4.387 to 4.584 tokens under greedy decoding, with reported relative speculative-decoding speedup improvements ranging from 3.15% on summarization to 5.82% on coding.

Nemotron 3.5 specialization

Super 3.5 turns a text base into a multimodal agent

  • Nemotron 3.5 Super and Lightning: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Models for Agentic AI by NVIDIA et al. (2026) uses post-training not only to align Super 3.5 but also to introduce its vision-language capability. After warming the vision projector, Super uses two-stage SFT: the first stage jointly trains on vision, reasoning, chat, long-context, safety, and agentic data, while a smaller second stage changes the blend to reinforce capabilities that remain underlearned. Lightning 3.5 follows a lighter single-stage text-only SFT path.

  • Super’s vision corpus extends the Nano Omni data with chart understanding, grounded perception, multi-hop visual reasoning, GUI and web navigation, multimodal coding, and long-document vision examples. Synthetic examples are filtered for answerability, grounding, reasoning consistency, hallucinated labels or values, and unproductive self-correction. Agentic vision data adds executable image and video tool-use trajectories in persistent environments; the reported corpus contains 14,169 trajectories and 81,366 assistant actions.

  • SFT uses packed sequences up to 524,288 tokens with global batch size 32. The learning rate warms to \(10^{-5}\) and stays constant, using Adam with \(\beta_1=0.9\), \(\beta_2=0.95\), weight decay 0.1, and gradient clipping at 1.0. Two shared-weight MTP layers remain active with auxiliary-loss scale 0.3, preserving the internal drafter while the model learns multimodal and agentic behavior.

  • The following figure (source) shows the Nemotron 3.5 Super post-training pipeline, where a shared SFT/RL trunk branches into reasoning, agentic, alignment, video, and vision-tool-use teachers before two stages of MOPD consolidate those capabilities into the final student.

RLVR, RLHF, and specialized teachers

  • Super jointly applies RLVR and RLHF, routing each environment to a deterministic verifier or learned reward model as appropriate. The refreshed RL mixture includes Olympiad-level competitive programming with executable binary rewards and a finance-agent environment whose tools cover SEC filing search, document parsing, evidence retrieval, and source-supported submission. The same finance tool interface is used across synthetic-data generation, SFT, RL, and evaluation to reduce train-evaluation interface mismatch.

  • The main RL stage uses a global batch size of 8,192 with 16 rollouts per sample and increases maximum generation length from 80K to 128K. Router replay and an FP32 LM head reduce discrepancies between training-engine and inference-engine log probabilities by more than an order of magnitude, making the optimization stack itself part of the post-training recipe.

  • Specialized teachers are then branched from the shared RL checkpoint. The agentic teacher uses full end-to-end GRPO on executable Terminal, SWE, and Search tasks; the reasoning teacher uses a 25,600-example curriculum dominated by science with 131,072-token maximum generation; vision teachers specialize in video reasoning and tracking, visual tool use, and visual reasoning; and an alignment teacher uses RLHF with a reasoning-length penalty. These teachers separate domain optimization before capability consolidation.

Two-stage MOPD and MTP boosting

  • Super 3.5 consolidates its specialists in two MOPD stages. Stage 1 emphasizes agentic, reasoning, alignment, and vision-tool teachers, while Stage 2 shifts weight toward alignment and multimodal specialists and uses the Stage-1 model as a self-anchor for capabilities whose teachers are retired. Stage 2 additionally smooths the teacher target with a proximal-teacher mixture, bounding negative token advantages after the student has already absorbed one round of specialist guidance. Lightning does not use MOPD.

  • The final MTP drafter is optimized separately from the backbone. Super first performs a 4K-step MTP-only warm start from the SFT checkpoint and then 25K final boosting steps against the frozen final backbone; Lightning performs 48K boosting steps against its final backbone. Unlike Ultra’s temperature-scaled forward-KL MTP boosting, Super and Lightning switch to LK Losses: Direct Acceptance Rate Optimization for Speculative Decoding by Samarin et al. (2026), which directly targets speculative-decoding acceptance rather than treating KL divergence as a proxy.

  • On Super, MTP boosting raises average acceptance length at draft length 7 from 4.287 to 4.718 under greedy decoding and from 3.787 to 4.612 under probabilistic sampling. The relative gain is largest at deeper draft positions, where recursive train-inference mismatch is most severe.

Bolt recovery after structural compression

  • Nemotron 3.5 Bolt 74B-A9B 24GB: A Hybrid Vision-Language Model for Local AI by NVIDIA et al. (2026) uses a recovery pipeline because architecture compression is applied primarily to the text tower of the Super 3.5 VL parent. The trained parent vision tower is first reattached to the compressed LLM, and only the vision-to-language projector is adapted on an image-only blend of approximately 6.6M samples per epoch. A subsequent vision-heavy SFT stage jointly trains the language model, vision adapter, and vision encoder to recover multimodal quality.

  • The following figure (source) shows Bolt’s post-compression training journey: vision-adapter training and vision-heavy SFT, an 80:20 SLERP merge, end-to-end MCP-Atlas RL, RLHF and SWE-RL, followed by a final 75:25 checkpoint merge into the released BF16 model.

  • Bolt’s end-to-end RL focuses on local-agent interaction efficiency rather than only static accuracy. MCP training uses 32 prompts with 16 generations per prompt and improves the reported public-task MCP-Atlas score from approximately 48 to 56. SWE-RL uses 64 prompts with 16 generations per prompt, a maximum sequence length of 196,608 tokens, and learning rate \(3\times10^{-6}\); the report also observes transfer to PinchBench, reaching up to 90.4 when averaged over ten repeats.

  • This pipeline illustrates a distinct role for post-training after compression. Distillation recovers broad predictive behavior during architecture search, but multimodal adapter training, vision-heavy SFT, interactive RL, RLHF, and checkpoint merging are still required to recover the behavioral and tool-use capabilities expected from the parent model.

Full-agent trajectory training

  • The post-training distribution becomes increasingly agentic across the Nemotron series. Early supervised examples can be represented as a prompt followed by an answer, optionally with a reasoning trace. Tool-enabled SFT expands this structure into a trajectory containing model thoughts or decisions, tool calls, observations, environment state, and subsequent actions. For software engineering, the relevant example may include repository search, file inspection, edits, execution, tests, and patch revision rather than one final code block.

  • RL follows the same progression. A simplified task can often be converted into a single-response problem with a cheap verifier, but end-to-end agentic RL instead lets the policy interact with a live sandbox over many steps. The environment must preserve state, expose tools, execute commands, run verifiers, and return observations until the task terminates. The reward can then reflect whether the complete trajectory produced a correct final state.

  • This change in horizon substantially alters the infrastructure requirement. Large-scale agentic RL may need thousands of isolated environments, fast environment initialization and recovery, coordinated tool execution, judge or test infrastructure, and schedulers that tolerate highly variable trajectory durations. The challenge is therefore not only to make the policy learn from long-horizon experience, but also to keep expensive learner GPUs and inference workers utilized while those experiences are being generated.

  • Nemotron 3 Super and Ultra address different points on this spectrum. PivotRL reduces cost by concentrating on informative intermediate decisions from expert trajectories. Dedicated SWE-RL performs genuine end-to-end repository interaction. Ultra’s teacher pool allows particularly expensive domains to be optimized separately, while MOPD provides a mechanism for transferring those capabilities back into one general policy.

Post-training infrastructure

NeMo Gym and NeMo RL

  • The model-training algorithms depend on a distributed system capable of running heterogeneous interactive environments. NVIDIA NeMo Gym separates an RL environment into agent servers, model servers, and resource servers. Agent servers execute environment-specific interaction loops; model servers expose inference and retain token-level log probabilities needed for policy updates; resource servers execute verifiers and compute rewards.

  • NVIDIA NeMo RL operates the training loop and uses Megatron-Core for distributed model optimization. Rollouts are routed through NeMo Gym and vLLM, while Ray and Slurm coordinate policy workers, inference workers, environments, sandboxes, and judge models.

  • The abstraction is important because environments have very different execution profiles. Cheap verifiers can run inline, whereas software-engineering tasks require containers and test execution, and open-ended domains may depend on separately hosted judge models. Keeping these resources behind standardized services allows rollout workers to request the environment-specific computation they need without hard-coding every domain into the learner loop.

In-flight model updates

  • Super and Ultra use asynchronous RL in which generation continues while learner updates run on separate GPU pools. Super also supports in-flight weight updates: generation workers can adopt newer model parameters without waiting for all active trajectories to terminate. A single trajectory may consequently contain tokens generated by slightly different policy snapshots, and the KV cache is not recomputed when an update occurs.

  • The benefit is higher hardware utilization; the cost is additional policy mismatch. The algorithms therefore restrict policy lag and mask updates when training and inference probabilities diverge too strongly.

MTP-accelerated RL rollouts

  • Rollout generation is typically slower than the learner update in Ultra’s asynchronous system, particularly because a small number of very long generations determine when a rollout batch completes. MTP speculative decoding is therefore used during RL and MOPD, not only during final serving.

  • Ultra evaluates draft lengths of zero, three, five, and seven tokens during RLVR. A five-token MTP draft gives the best reported operating point, reducing mean rollout-generation time by a factor of 1.46 relative to ordinary decoding. Most of the improvement comes from long-tail trajectories that emit many tokens and finish when effective batch concurrency has already fallen.

  • This creates a useful feedback loop between architecture and alignment: MTP is initially introduced to accelerate inference, but because RL requires enormous amounts of inference to generate training trajectories, better decoding directly reduces the cost of producing the post-training data that improves the model.

Scaling to thousands of GPUs

  • Ultra’s RL infrastructure is reported at more than 3,000 GPUs in parts of the production stack. At this scale, bottlenecks arise outside the core model kernels. Ray’s Global Control Service can become overwhelmed by simultaneous actor registrations, Slurm launch overhead can scale with the number of nodes, and expert-parallel groups can accidentally span NVLink domains and force all-to-all communication onto InfiniBand.

  • Topology-aware rank assignment keeps expert-parallel groups within the same GB200 NVL72 NVLink domain, producing a reported 20% end-to-end throughput improvement. Explicit NUMA binding of policy and vLLM workers to the CPU socket local to their GPUs contributes another reported 10% improvement.

  • Checkpointing is also moved off the critical path. A synchronous save initially blocks training for roughly 60 seconds. Asynchronous checkpointing, overlapped transfers, persistent checkpoint workers, background finalization, and sharded optimizer-state saving reduce exposed checkpoint time to less than one second.

  • Finally, persistent caches for compiled FlashInfer, Triton, PyTorch Inductor, and vLLM artifacts reduce reported cold initialization from 38.8 minutes to approximately 0.4 minutes. This is particularly important for large RL jobs because hardware or software failures can make restart latency a meaningful fraction of total cluster time.

  • Overall, Nemotron 3’s post-training system evolves from conventional SFT plus RL into a distributed capability-acquisition pipeline. SFT establishes tool formats, reasoning modes, and broad behavioral priors; mixed-environment RLVR improves verifiable capabilities without sequential domain forgetting; GenRM-based RLHF addresses open-ended behavior; PivotRL reduces the rollout cost of long-horizon agent training; specialized Ultra teachers push individual domains; MOPD consolidates their token-level guidance; and MTP is retrained and reused to accelerate both deployment and the RL process itself. The result is a post-training stack in which data design, reward design, inference efficiency, and distributed systems are tightly coupled rather than independent stages.

Training Objectives Across Pre-, Mid-, and Post-Training

Objective map

  • The Nemotron reports do not use mid-training as a single universal stage name. In this primer, mid-training refers to continued-training phases that sit between broad foundation-model pre-training and behavior-oriented post-training. In the core Nano, Super, and Ultra lineage, this includes long-context continued pre-training and MTP-focused continuation. In NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4, the model card explicitly separates pre-training from a continued pre-training stage for MTP before SFT and RL. Nemotron-Labs-Diffusion by Fu et al. (2026) provides a second example in which the mid-training objective itself changes from pure autoregressive learning to a joint autoregressive-diffusion objective.

  • Across these stages, the optimization signal evolves from token likelihood, to context- or auxiliary-head specialization, to behavior optimization and teacher-policy matching:

Stage Representative Nemotron objective What the objective is trying to acquire
Pre-training next-token NLL, plus MTP and small MoE balancing terms where applicable broad language, code, math, knowledge, and sequence modeling
Mid-training / CPT the same language-model objective on a changed context/data distribution, or an MTP-specific continuation objective long-context competence or stronger future-token drafting
SFT output-token NLL with stage-dependent loss normalization, plus retained MTP loss in Super/Ultra instruction following, reasoning format, tool use, agent trajectories
RLVR GRPO-style expected-reward optimization with verifiable rewards and masked importance sampling capabilities that can be checked by deterministic or structured verifiers
RLHF GRPO-style policy optimization with GenRM-derived preference rewards open-ended quality, chat behavior, safety, and usability
MOPD on-policy reverse-KL distillation from domain-specialized teachers consolidation of specialist capabilities into one student
MTP boosting forward-KL distillation from the frozen backbone into the MTP head higher speculative-decoding acceptance without changing backbone quality
  • The key change is therefore not simply the dataset. Pre-training and most continued pre-training optimize predictive likelihood; SFT restricts that likelihood objective to curated assistant outputs; RL replaces supervised token targets with rewards; MOPD replaces sparse trajectory reward with dense token-level teacher guidance; and MTP boosting finally freezes the backbone and trains only the speculative drafter to reproduce the backbone distribution.

Pre- and mid-training objectives

Autoregressive likelihood

  • The base objective throughout the main Nemotron 3 text family is autoregressive next-token prediction. For a token sequence \(x=(x_1,\ldots,x_T)\), a normalized form of the objective is

    \[\mathcal{L}_{\mathrm{AR}}(\theta) = \mathbb{E}_{x\sim\mathcal{D}} \left[ -\frac{1}{T} \sum_{t=1}^{T} \underbrace{ \log p_{\theta}(x_t\mid x_{<t}) }_{\text{next-token log-likelihood}} \right]\]

    • where

      \[\underbrace{\mathcal{D}}_{\text{pre-training mixture}} ,\qquad \underbrace{x_{<t}}_{\text{causal prefix}} ,\qquad \underbrace{p_{\theta}(x_t\mid x_{<t})}_{\text{model probability of the observed token}}\]
      • define the data distribution, the visible history, and the prediction being optimized. The Nemotron 3 Ultra report explicitly refers to the main training loss as the cross-entropy loss for the next token when analyzing Ultra’s training divergence.

  • Nano primarily uses this autoregressive objective together with MoE load-balancing machinery. Nemotron 3 Nano reports aux-loss-free expert balancing together with a small conventional load-balancing loss coefficient of \(10^{-4}\). Super retains this small expert-balancing term and adds native Multi-Token Prediction.

Multi-Token Prediction

  • Super and Ultra augment ordinary next-token prediction with an auxiliary MTP objective. The MTP head predicts future tokens several positions ahead while sharing parameters across prediction steps. Nemotron 3 Super reports an MTP loss scaling factor of \(0.3\), while Nemotron 3 Ultra uses a total MTP scale of \(0.1\), split as \(0.05\) for each of its two shared-weight MTP blocks.

  • The reports specify the auxiliary weighting and future-token prediction role, but do not print a separate closed-form equation for the pre-training MTP head. A compact notation for the reported objective is

    \[\mathcal{L}_{\mathrm{pre}} = \underbrace{ \mathcal{L}_{\mathrm{AR}} }_{\text{next-token language modeling}} + \underbrace{ \lambda_{\mathrm{MTP}} \sum_{k=1}^{K} \mathcal{L}_{\mathrm{MTP}}^{(k)} }_{\text{multi-step future-token prediction}} + \underbrace{ \lambda_{\mathrm{LB}}\mathcal{L}_{\mathrm{LB}} }_{\text{small MoE balancing regularizer}}\]

    • with the auxiliary future-token term written schematically as

      \[\mathcal{L}_{\mathrm{MTP}}^{(k)} = -\frac{1}{|\mathcal{T}_k|} \sum_{t\in\mathcal{T}_k} \underbrace{ \log p_{\theta,\phi}\!\left(x_{t+k}\mid h_t^{(k)}\right) }_{\text{predict token }k\text{ positions ahead}}\]
      • where \(\phi\) denotes the shared MTP-head parameters and \(h_t^{(k)}\) denotes the representation used by the \(k\)-step predictor. This notation makes explicit the future-token NLL described in the reports; it is not a separately printed Nemotron equation.

  • For Super,

    \[\underbrace{\lambda_{\mathrm{MTP}}=0.3}_{\text{Super auxiliary MTP scale}}\]

    • while Ultra uses

      \[\underbrace{\lambda_{\mathrm{MTP}}=0.1}_{\text{Ultra total MTP scale}} = \underbrace{0.05}_{\text{MTP block 1}} + \underbrace{0.05}_{\text{MTP block 2}}\]

  • The lower Ultra scale matters numerically. The Ultra report observes that the MTP gradients into the shared output layer are comparatively small, which made reduced-precision gradient accumulation particularly sensitive during the first major pre-training instability.

Objective-preserving continued pre-training

  • For the core Nano, Super, and Ultra models, long-context extension changes the training distribution and sequence length much more than it changes the objective itself. The model continues to optimize the autoregressive objective, with MTP retained when present, but now over a long-context mixture.

  • This can be written as

    \[\mathcal{L}_{\mathrm{LC}} = \underbrace{ \mathbb{E}_{x\sim\mathcal{D}_{\mathrm{LC}}} }_{\text{long-context continued-pretraining distribution}} \left[ \underbrace{ \mathcal{L}_{\mathrm{pre}}(x) }_{\text{same predictive objective}} \right]\]

    • with a data mixture of the form

      \[\mathcal{D}_{\mathrm{LC}} = \underbrace{ \alpha\,\mathcal{D}_{\mathrm{long}} }_{\text{long documents, retrieval, long-context QA}} + \underbrace{ (1-\alpha)\,\mathcal{D}_{\mathrm{retain}} }_{\text{short-context data retained to reduce regression}}\]

  • The second term is practically important. Nano initially found that using only 512K-sequence CPT degraded short-context benchmarks, so it mixed 512K and 4K sequences. Ultra similarly performs most long-context iterations at 1M tokens while retaining a smaller fraction of 4K iterations, and mixes long-context data with Phase-2 pre-training data. In other words, the mid-training design problem is partly distributional: extend the context support without moving the model so far toward long documents that shorter-task accuracy degrades.

Nemotron 3.5 MTP warmup and distillation

  • Nemotron 3.5 Super and Lightning: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Models for Agentic AI by NVIDIA et al. (2026) makes knowledge distillation an explicit base-model training stage for both 3.5 form factors. Before backbone distillation, a 100B-token warmup freezes the language-model backbone and updates only the MTP head using future-token cross-entropy. A compact representation is

    \[\mathcal{L}_{\mathrm{MTP-warm}} = -\frac{1}{|\mathcal{A}|K} \sum_{t\in\mathcal{A}} \sum_{k=1}^K \underbrace{\log p_{\phi}(x_{t+k}\mid h_t^{(k)})}_{\text{future-token supervision}}\]
    • where the backbone is fixed and \(\phi\) denotes the trainable MTP parameters. Super additionally mixes hidden states from earlier recursive MTP steps so the auxiliary head is trained on states closer to those it will encounter during autoregressive drafting.

  • Lightning 3.5 then distills Super 3 Base using top-128 teacher logits plus a sink token for residual probability mass outside the retained teacher support. If \(p_T\) is the reconstructed teacher distribution, \(p_S\) the student distribution, and \(y\) the observed token, the reported 0.9-to-0.1 blend can be written as

    \[\mathcal{L}_{\mathrm{Lightning-KD}} = \underbrace{0.9\,D_{\mathrm{KL}}\!\left(p_T\Vert p_S\right)}_{\text{teacher-distribution matching}} + \underbrace{0.1\left[-\log p_S(y)\right]}_{\text{hard-token cross-entropy}} + \underbrace{0.1\,\mathcal{L}_{\mathrm{MTP}}}_{\text{auxiliary MTP scale reported for base distillation}}\]
    • This combination preserves distribution-level information while maintaining direct supervision on rare observed tokens.

  • Super 3.5 replaces the ordinary forward-KL term with skew reverse KL at \(\alpha=0.1\). DistiLLM: Towards Streamlined Distillation for Large Language Models by Ko et al. (2024) introduces skew KL objectives to stabilize language-model distillation. With teacher distribution \(p_T\) and student distribution \(p_S\), the skew reverse-KL form is

    \[\mathcal{L}_{\mathrm{SRKL}}^{(\alpha)} = D_{\mathrm{KL}} \!\left( p_S \;\middle\Vert\; (1-\alpha)p_T+\alpha p_S \right)\]
    • and Super uses \(\alpha=0.1\) together with the inherited MTP objective and a smaller MoE load-balancing coefficient than Lightning.

  • These objectives clarify the role of 3.5 mid-training. The 1T-token distillation stage changes the target distribution rather than simply replaying next-token labels, while the later long-context and Indic stages return to continued language modeling over specialized data distributions. Super’s software-engineering mid-training then uses long natural artifacts and a small amount of loss-masked agent data before behavior is shaped more aggressively by SFT and RL.

Joint autoregressive-diffusion training

  • Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decoding by Fu et al. (2026) changes the objective itself. It first strengthens the left-to-right prior with pure AR training, then turns on block-wise diffusion supervision.

  • Its autoregressive objective is:

    \[\mathcal{L}_{\mathrm{AR}}(\theta) = \mathbb{E}_{x\sim\mathcal{D}} \left[ -\sum_{i=1}^{|x|} \underbrace{ \log p_{\theta}(x_i\mid x_{<i}) }_{\text{left-to-right token prediction}} \right]\]

  • The diffusion objective partitions the sequence into \(B\) contiguous blocks and corrupts one block at noise level \(t\sim\mathcal{U}[0,1]\) while keeping earlier blocks clean. In compact form,

    \[\mathcal{L}_{\mathrm{diff}}(\theta) = \underbrace{ \mathbb{E}_{t\sim\mathcal{U}[0,1],\, \tilde{x}_t^{\,b}\sim q(\cdot\mid x^b)} }_{\text{sample noise level and corrupted block}} \left[ -\frac{1}{t} \sum_{b=1}^{B} \underbrace{ \log p_{\theta} \!\left( x^b \mid \tilde{x}_t^{\,b}, x^{<b} \right) }_{\text{denoise current block conditioned on clean prefix}} \right]\]

    • and the joint objective is:

      \[\mathcal{L}_{\mathrm{NLD}}(\theta) = \underbrace{ \mathcal{L}_{\mathrm{AR}}(\theta) }_{\text{preserve left-to-right language modeling}} + \underbrace{ \alpha\, \mathcal{L}_{\mathrm{diff}}(\theta) }_{\text{learn block-wise parallel denoising}}\]

  • The two-stage schedule is particularly simple:

    \[\underbrace{\alpha=0}_{\text{Stage 1: pure AR}} \qquad\longrightarrow\qquad \underbrace{\alpha=0.3}_{\text{Stage 2: joint AR + diffusion}}\]

  • The report uses \(\alpha=0.3\) across subsequent training stages as well. This is a useful contrast with the core Nemotron 3 long-context CPT stages: in Nano, Super, and Ultra, mid-training primarily changes sequence length and data support, whereas Nemotron-Labs-Diffusion deliberately changes the optimization target.

Post-training objectives

Supervised fine-tuning

  • SFT still uses token likelihood, but the supervised support changes from arbitrary pre-training tokens to curated assistant outputs. Super makes the loss normalization explicit.

  • For a packed batch \(\mathcal{B}\) containing conversations \(c\), let \(\mathcal{O}_c\) be the output-token positions for conversation \(c\), and define

    \[\ell_t = \underbrace{ -\log p_{\theta}(y_t\mid x,y_{<t}) }_{\text{assistant-token negative log-likelihood}}\]

  • Super Stage 1 minimizes one global average over all output tokens:

    \[\mathcal{L}_{\mathrm{tok}} = \frac{ \displaystyle \sum_{c\in\mathcal{B}} \sum_{t\in\mathcal{O}_c} \underbrace{ \ell_t }_{\text{token-level supervision}} }{ \underbrace{ \displaystyle\sum_{c\in\mathcal{B}}|\mathcal{O}_c| }_{\text{all supervised output tokens in the batch}} }\]

  • This means long responses contribute proportionally more supervised tokens and therefore more total weight.

  • Super Stage 2 switches to equal conversation weighting:

    \[\mathcal{L}_{\mathrm{samp}} = \frac{1}{ \underbrace{|\mathcal{B}|}_{\text{number of conversations}} } \sum_{c\in\mathcal{B}} \underbrace{ \left( \frac{1}{|\mathcal{O}_c|} \sum_{t\in\mathcal{O}_c} \ell_t \right) }_{\text{mean loss for one conversation}}\]

  • The distinction is subtle but important. The first objective gives each output token equal weight globally. The second gives each conversation equal weight after normalizing for response length, preventing long reasoning traces from dominating the gradient. Super reports that this second stage recovers long-input-short-output performance while retaining the reasoning behavior learned in Stage 1.

  • Super also keeps MTP active during SFT:

    \[\mathcal{L}_{\mathrm{SFT,Super}} = \underbrace{ \mathcal{L}_{\mathrm{tok/samp}} }_{\text{assistant-output SFT loss}} + \underbrace{ 0.3\,\mathcal{L}_{\mathrm{MTP}} }_{\text{preserve multi-token prediction}}\]

  • Ultra follows the same two-stage SFT structure but uses a smaller MTP scale:

    \[\mathcal{L}_{\mathrm{SFT,Ultra}} = \underbrace{ \mathcal{L}_{\mathrm{tok/samp}} }_{\text{assistant-output SFT loss}} + \underbrace{ 0.1\,\mathcal{L}_{\mathrm{MTP}} }_{\text{two shared MTP layers}}\]

  • Ultra also masks the </think> token from the SFT loss in truncated reasoning-budget examples, preventing the model from treating an artificially inserted stopping marker as an ordinary supervised target.

RLVR

  • Nano, Super, Ultra, and Lightning use GRPO-style reinforcement learning across multiple environments. The Nemotron reports specify GRPO together with masked importance sampling, but they do not reproduce the complete base GRPO equation. The following form is therefore a compact representation of the named algorithm with the extra token mask described by the Nemotron implementations:

    \[\mathcal{J}_{\mathrm{RLVR}}(\theta) = \mathbb{E} \left[ \frac{1}{G} \sum_{i=1}^{G} \frac{1}{|o_i|} \sum_{t=1}^{|o_i|} \underbrace{ m_{i,t} }_{\text{masked importance-sampling gate}} \; \min \left( \underbrace{ \rho_{i,t}(\theta)\,\hat{A}_i }_{\text{unclipped policy improvement}} , \underbrace{ \operatorname{clip} \!\left( \rho_{i,t}(\theta), 1-\epsilon, 1+\epsilon \right) \hat{A}_i }_{\text{proximal clipped update}} \right) \right]\]

    • where:

      \[\underbrace{ \rho_{i,t}(\theta) = \frac{ \pi_{\theta}(o_{i,t}\mid s_{i,t}) }{ \pi_{\mathrm{behav}}(o_{i,t}\mid s_{i,t}) } }_{\text{learner-to-rollout policy ratio}}\]
      • and \(\hat{A}_i\) is the group-relative advantage derived from the task reward. In Nano, rollouts and updates are synchronous and effectively on-policy. Super and Ultra use asynchronous rollout generation, so masking the importance ratio becomes more important for limiting the effect of stale trajectories.

  • The reward source is environment-specific. For mathematics, code, formal reasoning, structured output, and software engineering, rewards can often be generated by exact-match checks, compilers, unit tests, symbolic verifiers, schema constraints, or final environment state. This is the essence of RLVR: the policy objective is generic, while the learning signal is supplied by a domain verifier.

GenRM-based RLHF

  • When correctness cannot be determined reliably with a programmatic verifier, Nemotron uses a generative reward model. Nano trains its GenRM with GRPO using a reward that combines format validity, two helpfulness-score errors, and pairwise-ranking error:

    \[R_{\mathrm{GenRM}} = \underbrace{ -C_1 I_{\mathrm{format}} }_{\text{format-violation penalty}} - \underbrace{ |P_{h1}-G_{h1}| }_{\text{response 1 helpfulness error}} - \underbrace{ |P_{h2}-G_{h2}| }_{\text{response 2 helpfulness error}} - \underbrace{ C_2|P_r-G_r| }_{\text{pairwise-ranking error}}\]
    • with \(\underbrace{C_1=10}_{\text{format penalty weight}}, \qquad \underbrace{C_2=1}_{\text{ranking-error weight}}\) in the Nano report.

  • Once the GenRM is trained, the policy can use its preference signal as the reward inside the same group-relative RL machinery. Nano additionally modifies this reward to control verbosity. Its group-relative length-control reward is

    \[R_i = \underbrace{ R_i^{(\mathrm{base})} }_{\text{GenRM quality reward}} + \underbrace{ \lambda_{\mathrm{think}} \tilde{w}_i^{(\mathrm{think})} }_{\text{relative reasoning-length adjustment}} + \underbrace{ \lambda_{\mathrm{answer}} \tilde{w}_i^{(\mathrm{answer})} }_{\text{relative answer-length adjustment}}\]
    • with \(\lambda_{\mathrm{think}}=\lambda_{\mathrm{answer}}=0.5\) in the reported setup. Because the length terms are centered within each response group, they encourage relative conciseness without imposing one absolute reasoning length on every task.

  • Super extends the same idea with a principle-following GenRM, uses GenRM supervision during the multi-environment RL stage, and performs a separate RLHF-only stage near the end of post-training.

Multi-teacher on-policy distillation

  • Ultra changes the final capability-consolidation objective from sparse reward maximization to dense teacher-policy matching. Let \(\pi_{\theta}\) be the student, \(\pi_{T_i}\) the teacher for domain \(i\), and \(\mathcal{D}_i\) that domain’s prompt distribution. For a student-generated trajectory \(y\), the fully on-policy MOPD objective is

    \[\mathcal{J}_{\mathrm{MOPD}}(\theta) = \sum_{i=1}^{N} \underbrace{ \lambda_i }_{\text{domain sampling / loss weight}} \; \mathbb{E}_{ q\sim\mathcal{D}_i,\, y\sim\pi_{\theta}(\cdot\mid q) } \left[ \sum_{t=1}^{H} \left( \underbrace{ \log \pi_{T_i}(y_t\mid s_t) }_{\text{teacher score of the realized student token}} - \underbrace{ \log \pi_{\theta}(y_t\mid s_t) }_{\text{student score of the same token}} \right) \right]\]

  • Maximizing this quantity is equivalent, at each student-induced prefix, to minimizing

    \[\underbrace{ D_{\mathrm{KL}} \left( \pi_{\theta}(\cdot\mid s_t) \;\Vert\; \pi_{T_i}(\cdot\mid s_t) \right) }_{\text{reverse KL from student to the relevant teacher}}\]

  • The practical Ultra implementation is asynchronous, so the rollout policy may lag behind the learner. Ultra defines the dense distillation advantage

    \[\hat{A}_t = \underbrace{ \operatorname{sg} \left[ \log\pi_{T_i}(y_t\mid s_t) - \log\pi_{\mathrm{prox}}(y_t\mid s_t) \right] }_{\text{teacher-vs-proximal token advantage}}\]

    • together with

      \[\underbrace{ c_t = \operatorname{sg} \left[ \frac{ \pi_{\mathrm{prox}}(y_t\mid s_t) }{ \pi_{\mathrm{behav}}(y_t\mid s_t) } \right] }_{\text{behavior-to-proximal correction}} , \qquad \underbrace{ r_t(\theta) = \frac{ \pi_{\theta}(y_t\mid s_t) }{ \pi_{\mathrm{prox}}(y_t\mid s_t) } }_{\text{proximal-to-current policy ratio}}\]

      • and maximizes

        \[\mathcal{J}_{\mathrm{async\text{-}MOPD}}(\theta) = \mathbb{E} \left[ \sum_{t=1}^{H} \underbrace{ m_t }_{\text{IcePop token mask}} \underbrace{ c_t }_{\text{staleness correction}} \min \left( \underbrace{ r_t(\theta)\hat{A}_t }_{\text{unclipped distillation update}} , \underbrace{ \operatorname{clip} \!\left( r_t(\theta), 1-\epsilon, 1+\epsilon \right)\hat{A}_t }_{\text{proximal clipped update}} \right) \right]\]

  • This objective highlights the conceptual difference between RLVR and MOPD. RLVR often assigns one sparse environment reward to a whole rollout and derives a group-relative advantage from that outcome. MOPD supplies a teacher-derived learning signal at each realized token, making specialization and later consolidation separable.

MTP boosting

  • After the main Ultra policy is trained, the backbone is frozen and only the MTP head is optimized. The objective no longer uses gold-token cross-entropy. Instead, the drafter is trained to match the frozen backbone’s full next-token distribution under the noisy hidden-state conditions it encounters during autoregressive drafting.

  • For assistant-token positions \(\mathcal{A}\), MTP steps \(k\in\{1,\ldots,N_{\mathrm{mtp}}\}\), distillation temperature \(T\), backbone logits \(z_{t+k}\), and MTP logits \(z_{t+k}^{\mathrm{mtp},k}\), Ultra uses

    \[\mathcal{L}_{\mathrm{MTP\text{-}boost}}(\theta) = \underbrace{ \frac{T^2}{N_{\mathrm{mtp}}|\mathcal{A}|} }_{\text{temperature and averaging scale}} \sum_{k=1}^{N_{\mathrm{mtp}}} \sum_{t\in\mathcal{A}} \underbrace{ D_{\mathrm{KL}} \left( \sigma(z_{t+k}/T) \;\Vert\; \sigma(z_{t+k}^{\mathrm{mtp},k}/T) \right) }_{\text{forward-KL match from backbone to MTP drafter}}\]

    • with

      \[\underbrace{T=2}_{\text{distillation temperature}}, \qquad \underbrace{N_{\mathrm{mtp}}=7}_{\text{drafting steps during boosting}}\]
      • in the Ultra report.

  • This final objective is deliberately head-only. Because the backbone is frozen, MTP boosting can improve speculative-decoding acceptance without risking regression in the already post-trained reasoning and agentic policy.

  • Nemotron 3.5 changes the final MTP objective again. Super uses a 4K-step MTP-only run from the SFT checkpoint to initialize a 25K-step final boosting phase, while Lightning uses 48K final boosting steps; in both cases the backbone is frozen and only the MTP head is updated. The 3.5 report replaces Ultra’s forward-KL objective with LK Losses: Direct Acceptance Rate Optimization for Speculative Decoding by Samarin et al. (2026), which optimizes speculative-decoding acceptance more directly than KL-based distribution matching.

Compression and multimodal objectives

Bolt compression and low-bit distillation

  • Bolt introduces optimization objectives that are tied directly to a deployment budget. During Star Elastic compression, the student and architecture router jointly minimize teacher-student KD plus a parameter-budget penalty

    \[\min_{\theta,\psi} \underbrace{\mathcal{L}_{\mathrm{KD}}(\theta,\psi;b)}_{\text{recover parent behavior}} + \underbrace{\lambda\mathcal{L}_{\mathrm{budget}}(\psi;b)}_{\text{meet target deployment size}}\]

  • The later 2-bit Quantization-Aware Distillation stage changes the discrepancy function again. Quantization-Aware Distillation for NVFP4 Inference Accuracy Recovery by Xin et al. (2026) develops QAD as teacher-student distribution matching for quantized LLMs and VLMs; Bolt extends the approach to 2.0625-bit expert weights and reports that a probability-overlap Jaccard objective outperforms the KL, reverse-KL, Jensen-Shannon, logit-MSE, and intermediate-cosine alternatives tested for this much more aggressive setting.

  • For quantized student distribution \(p_S\) and frozen-teacher distribution \(p_T\) over vocabulary \(\mathcal{V}\), Bolt’s Jaccard loss is

    \[\mathcal{L}_{\mathrm{Jaccard}} = 1 - \frac{ \sum_{v\in\mathcal{V}} \min\!\left(p_S(v),p_T(v)\right) }{ \sum_{v\in\mathcal{V}} \max\!\left(p_S(v),p_T(v)\right) }\]
    • The distributions use softmax temperature 1, and the supervised language-model cross-entropy term is omitted. QAD uses SFT data, RL rollouts, and on-policy responses from the BF16 teacher; the selected learning-rate schedule warms to \(10^{-5}\) over 1.2B tokens and then cosine-decays toward \(10^{-7}\) over 25B tokens.

Nano Omni multimodal objective

  • Nemotron 3 Nano Omni follows the same general progression but adds modality-specific alignment objectives and multimodal RL. Its seven-stage SFT curriculum progressively trains the vision projector, vision-language stack, audio projector and encoder, and finally the joint omni-modal model while extending context length. The supervised stages continue to use language-model-style output supervision, but different modules are frozen or unfrozen according to the stage.

  • The first RL stage uses Mixed Preference Optimization, described in the report as the combination

    \[\mathcal{L}_{\mathrm{MPO}} = \underbrace{ \mathcal{L}_{\mathrm{DPO}} }_{\text{pairwise preference supervision}} + \underbrace{ \lambda_{\mathrm{BCO}}\mathcal{L}_{\mathrm{BCO}} }_{\text{absolute quality supervision}}\]
    • where the report specifies DPO as the preference loss and BCO as the quality loss but does not publish an additional Nemotron-specific coefficient or expanded formula.

  • Image-RL then returns to outcome-based RL. Each prompt receives a scalar reward combining task correctness and output-format quality:

    \[R_{\mathrm{image}} = \underbrace{ w_{\mathrm{out}}R_{\mathrm{outcome}} }_{\text{answer / grounding correctness}} + \underbrace{ w_{\mathrm{fmt}}R_{\mathrm{format}} }_{\text{reasoning and answer-format compliance}}\]
    • where the exact weights are task-dependent and are not specified as one global pair of constants in the report. The outcome component is supplied by string matching, symbolic math verification, multiple-choice checks, or GUI-coordinate scoring depending on the prompt. Omni-RL then extends the same outcome-driven principle to mixed image, video, audio, and text reasoning.

Full-pipeline objective progression

  • The complete progression can be summarized as a change in what constitutes a desirable model update:

    \[\underbrace{ \text{predict the observed next token} }_{\text{pre-training}} \;\longrightarrow\; \underbrace{ \text{predict under longer contexts or with auxiliary future-token heads} }_{\text{mid-training / CPT}} \;\longrightarrow\; \\ \underbrace{ \text{imitate curated assistant trajectories} }_{\text{SFT}} \;\longrightarrow\; \underbrace{ \text{increase verifier or preference reward} }_{\text{RLVR / RLHF}} \;\longrightarrow\; \underbrace{ \text{match specialized teacher policies on student states} }_{\text{MOPD}} \;\longrightarrow\; \underbrace{ \text{match the frozen backbone distribution with the speculative drafter} }_{\text{MTP boosting}}\]

  • This progression is one of the clearest ways to understand the Nemotron training stack. Pre-training acquires broad predictive structure. Mid-training modifies the support on which that predictive structure must operate. SFT imposes desired behaviors through curated trajectories. RL optimizes outcomes that can be verified or judged. MOPD consolidates domain specialists with dense token-level supervision. MTP boosting then optimizes inference behavior without modifying the final policy itself.

Quantization and Low-Precision Inference

Quantization strategy across the Nemotron 3 family

Model Main quantized checkpoint Principal weight/activation format Attention KV cache Mamba SSM cache
Nano FP8 Primarily W8A8 Sensitive attention layers retained in BF16 FP8 Not the central optimization in Nano
Super FP8 W8A8 on MoE and Mamba GEMMs BF16 FP8 FP16 + stochastic rounding
Super NVFP4 Mostly W4A4 with selective FP8/BF16 promotion QKV BF16; output FP8/BF16 FP8 FP16 + stochastic rounding
Ultra NVFP4 Routed experts NVFP4; selected operators FP8/BF16 BF16 FP8 FP16 + stochastic rounding
Ultra on Hopper Same NVFP4 checkpoint W4A16 execution BF16 FP8 FP16 + stochastic rounding
  • The distinction between formats is important. In notation such as W8A8, the first number refers to the nominal weight precision and the second to activation precision. W4A16 therefore denotes 4-bit-stored weights dequantized for GEMMs operating with higher-precision activations, whereas W4A4 uses low precision for both matrix operands.

NVFP4 numerical representation

  • Introducing NVFP4 for Efficient and Accurate Low-Precision Inference describes NVIDIA’s FP4 format as a two-level scaled representation built around E2M1 4-bit floating-point values. Groups of 16 values share an E4M3 FP8 scale, and a second FP32 scaling value provides additional tensor-level dynamic range. This combination reduces the local block size relative to coarser microscaling formats while preserving a wider effective dynamic range than raw E2M1 values alone.

  • For a block of real values \(x\), a simplified block-scaled quantizer can be represented as

    \[q_i = Q_{\mathrm{E2M1}} \left( \frac{x_i}{s_b s_g} \right)\]

    • with reconstruction:

      \[\widehat{x}_i = q_i s_b s_g\]
      • where \(s_b\) is a block-level E4M3 scale and \(s_g\) is a higher-level FP32 scale. The actual NVFP4 implementation includes constraints associated with representable scale values and hardware-supported quantization paths, but this decomposition captures the key idea: low-bit values are given a locally adaptive numerical range rather than sharing one scale across an entire tensor.

  • Nemotron 3 Super by NVIDIA et al. (2026) uses 16-element blocks along the final dimension for inference-time NVFP4 and quantizes block scales to E4M3 under a statically calibrated FP32 global scale.

Nano: selective FP8 post-training quantization

Calibration from reasoning data

  • Nano is post-trained in BF16 and then converted to FP8 through post-training quantization using NVIDIA Model Optimizer and Megatron-LM. Its calibration corpus contains 1,000 examples drawn from the model’s post-training reasoning SFT distribution. Nemotron 3 Nano by NVIDIA et al. (2025) reports that these SFT examples produce slightly better accuracy recovery than calibration using CNN/DailyMail, while replacing them with on-policy generations from the BF16 model does not provide a further measurable benefit.

  • This result illustrates an important practical property of PTQ calibration: calibration data does not need to reproduce the entire training distribution, but it should activate tensor ranges representative of deployment. Reasoning-oriented post-training examples more closely resemble Nemotron’s intended inference behavior than a generic summarization corpus.

Selective rather than uniform FP8 conversion

  • The Nano sensitivity analysis shows that the six self-attention layers are among the most quantization-sensitive components. The six Mamba layers immediately preceding those attention layers are also sensitive. The final deployment recipe therefore leaves both sets in BF16 while quantizing most other eligible weights and activations to FP8. Mamba Conv1D operations remain BF16, while the attention KV cache is FP8. Nemotron 3 Nano by NVIDIA et al. (2025) identifies this configuration as the best accuracy-efficiency compromise among the tested mixed-precision layouts.

  • Conceptually, the recipe is

      6 attention layers                         → BF16
  6 Mamba layers feeding those attention     → BF16
  Remaining eligible weights/activations      → FP8
  Attention KV cache                          → FP8
  Mamba Conv1D                                → BF16

  • The significance of this result is broader than retaining twelve layers at higher precision. Hybrid architectures do not necessarily have uniform quantization sensitivity across layer types. A relatively small set of globally important attention anchors can determine enough of the model’s information-routing behavior that quantizing them aggressively causes disproportionate quality loss.

Accuracy-throughput trade-off

  • Nano’s final FP8 checkpoint retains approximately 99% median benchmark accuracy relative to BF16 while materially increasing inference throughput. The throughput gain comes not only from faster low-precision arithmetic, but also from reduced model and cache footprint, which enables larger batches under a fixed H100 memory budget. Nemotron 3 Nano by NVIDIA et al. (2025) evaluates the ablation at an 8K-token input and 16K-token output sequence length using the maximum feasible batch size for each configuration.

  • The following figure (source) shows the Nano quantization ablation over attention precision, KV-cache precision, and selective high-precision layers, plotting median accuracy recovery against throughput improvement over the BF16 baseline.

  • One of the clearest results is the importance of FP8 KV caching. Lower cache precision permits a substantially larger batch before exhausting GPU memory. However, quantizing all compute layers indiscriminately causes greater accuracy degradation, which motivates keeping the sensitive attention and adjacent Mamba layers in BF16.

Super: separate FP8 and NVFP4 deployment checkpoints

Hopper-oriented FP8 checkpoint

  • Super creates two deployment checkpoints using PTQ: W8A8 FP8 for Hopper and W4A4 NVFP4 for Blackwell. For FP8 calibration, 256 SFT examples are processed at a context length of 65,536 tokens. Routed and shared MoE GEMMs and Mamba linear layers are quantized to FP8, while several numerically sensitive components remain at higher precision. Nemotron 3 Super by NVIDIA et al. (2026) gives the following operator-level recipe.
Operator Super FP8 checkpoint
Embeddings BF16
Attention QKV and output GEMMs BF16
KV cache / first attention BMM path FP8
Second attention BMM BF16
Routed MoE experts FP8
Shared MoE experts FP8
LatentMoE projections BF16
Router FP32
Mamba linear GEMMs FP8
Mamba SSM cache FP16
Mamba Conv1D BF16
Output layers BF16
  • The model therefore does not treat an “FP8 checkpoint” as an assertion that every operator uses FP8. FP8 is applied to the computationally dominant GEMMs and the KV cache, while the router, embeddings, attention projections, latent projections, convolution, and output layer remain at higher precision where the expected runtime benefit is smaller or sensitivity is higher.

NVFP4 weight-scale calibration

  • Super’s Blackwell-oriented checkpoint applies a substantially more aggressive NVFP4 recipe. The baseline NVFP4 procedure sets the scale of each 16-element block from its maximum absolute value. Super instead evaluates alternative scale-selection rules and finds that weight-block scales selected to minimize reconstruction mean squared error work better overall, while activation scales continue using dynamic max-based scaling. Nemotron 3 Super by NVIDIA et al. (2026) chooses this asymmetry because weight-scale optimization is an offline PTQ operation, whereas activation-scale computation occurs at runtime and therefore must remain inexpensive.

  • For a weight block \(w\) and candidate scale \(s\), the offline scale-selection problem can be written schematically as:

    \[s^\star = \arg\min_s \left\| w - sQ_{\mathrm{FP4}} \left( \frac{w}{s} \right) \right\|_2^2\]
  • By contrast, max-based activation scaling approximates $$s(x) \propto \max_i x_i $$, which avoids an expensive per-token or per-batch optimization over possible scales.

AutoQuantize for layer-wise precision assignment

  • After improving FP4 scale calibration, Super still finds that some operators should not be left in NVFP4. NVIDIA Model Optimizer AutoQuantize searches over operator-level precision choices while satisfying an effective-bit budget. It estimates the sensitivity of quantizing each operation, incorporates deployment cost, and solves a knapsack-style precision-allocation problem. AutoQuantize’s sensitivity calculation uses a second-order approximation inspired by Optimal Brain Surgeon and General Network Pruning by Hassibi et al. (1993), which uses curvature information to estimate the effect of perturbing trained parameters.

  • For a parameter or operator perturbation \(\Delta w\) around a local optimum, second-order sensitivity can be approximated from

    \[\Delta\mathcal{L} \approx \frac{1}{2} \Delta w^\top H \Delta w\]
    • where \(H\) is a local Hessian or approximation to curvature. In quantization, \(\Delta w\) corresponds to the error induced by replacing high-precision parameters with quantized values. Operators whose quantization perturbations align with high-curvature directions are consequently more sensitive.

  • Super asks AutoQuantize to choose among

    \[\{\mathrm{NVFP4},\mathrm{FP8},\mathrm{BF16}\}\]
    • for eligible backbone GEMMs under an effective-precision budget of 4.75 bits. Routed sparse-expert GEMMs remain NVFP4 throughout, while attention projections, Mamba projections, latent projections, and shared experts receive selectively higher precision.

  • The resulting recipe is:

Operator Super NVFP4 checkpoint
Routed expert GEMMs NVFP4
Shared expert GEMMs NVFP4 / FP8 / BF16
Attention QKV projections BF16
Attention output projections FP8 / BF16
LatentMoE projections FP8 / BF16
Mamba projections FP8 / BF16
KV cache FP8
Mamba SSM cache FP16
Mamba Conv1D BF16
Router FP32
Embeddings / output BF16

Calibration cost and quality recovery

  • The full mixed-precision Super PTQ process completes in less than two hours on one eight-GPU B200 node, using 512 SFT samples with sequence length 4,096 for the NVFP4 calibration procedure. The resulting checkpoint achieves approximately 99.8% median accuracy relative to BF16 while retaining near-FP4 deployment characteristics. Across the reported suite, most individual benchmark movements are small and occur in both directions rather than showing a systematic collapse from quantization. Nemotron 3 Super by NVIDIA et al. (2026) reports these comparisons across reasoning, agentic, instruction-following, long-context, and multilingual evaluations.

Nemotron 3.5 Super and Lightning: mixed-precision PTQ followed by QAD

  • Nemotron 3.5 Super and Lightning: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Models for Agentic AI by NVIDIA et al. (2026) extends Nemotron quantization from fixed deployment recipes into a PTQ-plus-QAD workflow. The released Super 3.5 checkpoint uses an aggressive mixed-precision recipe with NVFP4 routed and shared-expert GEMMs, FP8 attention QKV/output projections, FP8 Mamba linear layers, FP8 KV cache, FP16 Mamba SSM state with stochastic rounding, BF16 vision components and embeddings, and an FP8 LM head. The measured whole-model storage is 4.82 bits per original parameter when quantization-scale overhead is included and runtime caches are excluded.

  • Deployment precision is hardware-aware. Super’s NVFP4 checkpoint executes W4A4 on Blackwell and W4A16 on non-Blackwell platforms, while Lightning is optimized around W4A16 across GPU SKUs to reduce weight traffic without forcing low-precision activations. The recipe-development target is a usable point on the throughput-interactivity frontier, not merely minimum checkpoint size.

  • Calibration uses image-text rather than text-only examples so the language backbone observes projected visual-token distributions. Super calibrates with 1,024 SFT samples and selects MSE weight-scale calibration over maximum-based and MSE Four-Over-Six alternatives for the tested aggressive recipe. AutoQuant is also evaluated as a sensitivity-constrained assignment over NVFP4, FP8, and BF16, revealing that the LM head is highly sensitive, attention and Mamba projections form the next tier, and the large MoE expert matrices are comparatively robust.

  • Quantization-Aware Distillation starts from the calibrated PTQ student and freezes embeddings, unquantized language-model parameters, normalization, the vision encoder/projector, the MTP module, and router-bias updates. Only quantized language-model GEMM weights are optimized. The loss is KL divergence to the full-precision teacher, while hard-target cross-entropy, MTP cross-entropy, MoE load-balancing loss, and intermediate-feature distillation are disabled. This directly follows the distribution-matching strategy developed in Quantization-Aware Distillation for NVFP4 Inference Accuracy Recovery by Xin et al. (2026).

  • The released 4.82-BPE Super checkpoint uses the fixed aggressive PTQ recipe followed by 200 QAD iterations on the SFT blend. Across 36 benchmarks, the released quantized checkpoint reaches 99.62% median score recovery relative to the BF16 reference while increasing mean per-benchmark verbosity by only 0.44%.

Nemotron 3.5 Bolt: 2.0625-bit expert quantization

  • Bolt deliberately moves beyond the near-lossless quantization target used for Super. Nemotron 3.5 Bolt 74B-A9B 24GB: A Hybrid Vision-Language Model for Local AI by NVIDIA et al. (2026) quantizes both routed and shared expert projections to IQ2_XXS at 2.0625 bits per element, while Mamba input/output projections use NVFP4, attention projections, LatentMoE projections, embeddings, and LM head use Q8_0, and router plus Mamba-convolution weights remain FP32. The design premise is that, under a fixed memory budget, a larger compressed network at approximately two-bit expert precision can preserve more capability than a substantially smaller network kept at four-bit precision.

  • The report’s deployment accounting gives 22.01GB for model weights, including 17.05GB of routed-expert weights and 0.45GB of shared-expert weights. With recurrent state, KV cache, activation buffer, and the speculative decoder, the subtotal is 23.83GB excluding the separately itemized 1.68GB vision encoder and projector. These numbers make the memory target an explicit systems constraint rather than a checkpoint-file-size proxy.

  • Bolt extends QAD to this 2-bit regime and finds the training schedule materially more sensitive than in the 4-bit case. Its selected schedule warms to \(10^{-5}\) for 1.2B tokens and cosine-decays toward \(10^{-7}\) over 25B tokens. The final loss is the Jaccard probability-overlap objective defined in the training-objectives section, with temperature 1 and no supervised CE term.

  • The following figure (source) shows Bolt’s Quantization-Aware Distillation workflow, which uses the BF16 model as a teacher for the low-bit student and incorporates an additional QAD stage after later RL healing so the final quantized model tracks the final behavior distribution rather than only an earlier checkpoint.

  • The low-bit recipe is paired with post-training rather than treated as an isolated conversion. On-policy BF16-teacher rollouts, SFT data, and RL trajectories are included in QAD, and Terminal-environment rollouts are generated specifically to prevent agentic regressions during quantization. This is consistent with the broader Nemotron principle that quantization quality must be evaluated on the behavioral distribution the deployed model will actually encounter.

Recurrent-state quantization

Recurrent quantization error

  • One of the most important quantization findings in Nemotron 3 concerns the Mamba SSM cache. An attention KV-cache value is generally written once and later read by attention. A Mamba state is recurrent: the quantized state from one decoding step becomes an input to the state update at the next step. Quantization error can therefore accumulate sequentially. Nemotron 3 Super by NVIDIA et al. (2026) shows that naively storing this state in FP16 can cause large increases in output verbosity even when benchmark accuracy appears relatively stable.

  • Let the exact recurrent update be

    \[h_t = A_t h_{t-1} + B_t x_t\]

  • If cache quantization introduces additive error \(e_t\) at every step, the stored state follows

    \[h_{q,t} = A_t h_{q,t-1} + B_t x_t + e_t\]

  • Unrolling gives

    \[h_{q,t} = h_t + \sum_{i=0}^{t} \left( \prod_{j=i+1}^{t} A_j \right)e_i\]

  • Thus, an error introduced many tokens earlier remains transformed by all subsequent recurrent transitions rather than disappearing after its local step.

  • This makes cache quantization qualitatively different from ordinary weight PTQ. A small systematic rounding bias can become a long-horizon behavioral drift.

Why naive FP16 storage increases verbosity

  • In Super’s experiments, simply converting the FP32 SSM cache to FP16 increases LiveCodeBench completion length by approximately 37% when weights and activations remain at BF16, and by approximately 40% when combined with W8A8 execution. Accuracy does not fall proportionally, meaning the main observed failure can appear as substantially longer generation rather than straightforward task failure.

  • This is particularly relevant for reasoning models because verbosity is itself a deployment cost. A cache recipe that preserves pass rate while increasing generated token count by 40% may erase a substantial fraction of the throughput gain sought from quantization.

Stochastic rounding

  • Super attributes much of the drift to deterministic round-to-nearest behavior. If \(Q_{\mathrm{RTN}}\) repeatedly rounds similar state values in the same direction, the resulting error can have a persistent bias. Stochastic rounding instead chooses neighboring representable values probabilistically such that the expected quantized value matches the original value:

    \[\mathbb{E} \left[ Q_{\mathrm{SR}}(x) \right] = x\]

  • The quantization noise therefore has approximately zero mean:

    \[\mathbb{E}[e_t] \approx 0\]
    • making recurrent errors less likely to accumulate coherently in one direction.

  • Super finds that FP16 storage with stochastic rounding restores both accuracy and verbosity close to the FP32-cache baseline. It selects FP16 stochastic rounding using a five-round Philox pseudorandom-number generator because it avoids loading block scales, Blackwell provides conversion support for stochastic rounding, and the pseudorandom sequence can be generated efficiently through cuRAND. Nemotron 3 Super by NVIDIA et al. (2026) reports that this recipe removes the severe verbosity inflation of naive FP16 cache storage.

Ultra: mixed NVFP4 at 550B parameters

Final operator-level precision recipe

  • Ultra starts from the same mixed-precision philosophy but performs a new empirical search at 550B total parameters. The final checkpoint uses NVFP4 only for routed experts, FP8 for shared experts and Mamba linear projections, and BF16 for attention linears, LatentMoE projections, embeddings, MTP heads, and the output classifier. The KV cache uses FP8 and the Mamba cache uses FP16 with stochastic rounding. Nemotron 3 Ultra by NVIDIA et al. (2026) summarizes the final deployment recipe as follows:
Ultra operator Quantized precision
Embeddings BF16
Output classifier BF16
MTP layers BF16
Routed MoE experts NVFP4
Shared experts FP8 per-tensor
Mamba mixer linears FP8 per-tensor
Attention linears BF16
LatentMoE projections BF16
Mamba Conv1D BF16
KV cache FP8
Mamba SSM cache FP16 + stochastic rounding
  • This is less uniformly FP4 than Super’s initial AutoQuantize search might suggest. At Ultra scale, the dominant routed-expert parameter mass can be compressed to FP4 while relatively small but sensitive components remain at eight or sixteen bits without greatly increasing the aggregate model footprint.

Bits per element as a deployment budget

  • Ultra characterizes mixed precision using effective bits per element, or BPE. BPE is not a single numeric datatype; it summarizes the average storage budget after different tensor groups have been assigned different precisions. If tensor group \(g\) contains \(N_g\) parameters stored at effective width \(b_g\), a simplified model-level average is:

    \[\mathrm{BPE} = \frac{ \sum_g N_g b_g }{ \sum_g N_g }\]

  • Ultra evaluates recipes ranging from approximately 4.85 to 7.19 BPE. Most reasoning and instruction-following benchmarks remain within evaluation noise across this entire range. The major discriminating benchmark is AA-LCR: moving from 4.85 to 5.03 BPE increases long-context reasoning by roughly 2.4 points, after which performance plateaus even when precision is raised to 7.19 BPE. Nemotron 3 Ultra by NVIDIA et al. (2026) therefore selects 5.03 BPE as the smallest recipe that recovers the observed long-context loss.

  • This is an important deployment principle: once the sensitive operators have been promoted, adding bits to already robust operators can increase memory consumption without producing measurable quality gains.

Four-Over-Six FP4 scaling

The gap between 4 and 6

  • Four Over Six: More Accurate NVFP4 Quantization with Adaptive Block Scaling by Cook et al. (2025) observes that FP4’s nonuniform representable grid creates particularly large errors near the top of the E2M1 range. Positive E2M1 values include a gap between 4 and 6, so values falling inside that interval can experience unusually large relative rounding error.

  • Instead of always scaling each block to use 6 as its largest effective FP4 value, Four-Over-Six evaluates two candidate grids for every block \(M\in\{4,6\}\).

  • For each block \(w_b\), it chooses

    \[M_b^\star = \arg\min_{M\in\{4,6\}} \left\| w_b - \widehat{w}_b^{(M)} \right\|_2^2\]

  • The method therefore sacrifices some representable maximum range in blocks whose values are better approximated by the denser effective grid associated with \(M=4\).

Ultra’s selection of Four-Over-Six

  • Ultra evaluates ordinary max-based scaling, per-block MSE scaling, and Four-Over-Six. At the selected 5.03-BPE mixed-precision operating point, reported median accuracy recovery is 96.78% for ordinary max calibration, 98.40% for MSE scaling, and 98.50% for Four-Over-Six. Nemotron 3 Ultra by NVIDIA et al. (2026) therefore adopts Four-Over-Six for routed-expert FP4 weights while retaining dynamic max scaling for activations.

  • Across all 49,152 routed-expert projection weights in the reported tensor analysis, Four-Over-Six reduces median reconstruction MSE by 16.4% relative to standard max calibration. Ordinary per-block MSE optimization reduces reconstruction error even further, but the extra reconstruction improvement does not translate into a consistent downstream benchmark advantage, which is why Ultra selects Four-Over-Six rather than choosing solely by tensor-level MSE.

  • This highlights the distinction between quantization reconstruction error and model-level quality:

    \[\arg\min Q\text{-MSE} \neq \arg\max \text{downstream accuracy}\]
    • in general. Weight-level error is a useful proxy, but the best deployment recipe ultimately needs model-level evaluation.

One-terabyte BF16 quantization

  • Ultra’s BF16 representation is approximately 1.1 TB, making the PTQ procedure itself a distributed-systems problem. NVIDIA Model Optimizer supports both Hugging Face and Megatron-LM workflows, but the Ultra recipe uses Megatron-LM to exploit expert, data, and context parallelism during calibration. Nemotron 3 Ultra by NVIDIA et al. (2026) reports that expert parallelism shards the MoE experts, data parallelism increases calibration throughput, and context parallelism permits long calibration sequences without relying on single-process CPU offloading.

  • Under the reported configurations, a Hugging Face workflow using four B300 GPUs and CPU offloading requires approximately 2 hours 45 minutes, while a 16-B300 Megatron-LM setup finishes in approximately 42 minutes. Model loading alone falls from roughly 40 minutes to under two minutes, and loading plus calibration from approximately 85 minutes to nine minutes.

  • The result reinforces the same hardware-software co-design principle seen throughout Nemotron 3: quantization is not merely a numerical transform. For very large MoE models, the calibration and export implementation must itself understand expert partitioning, multi-node memory, and long-context execution.

SSM-cache optimization

When the Mamba cache dominates

  • Mamba’s cache is constant with sequence length, whereas attention’s KV cache grows linearly with sequence length. Constant-size does not necessarily mean small, however. For Ultra at batch size one, the FP32 Mamba state occupies more memory than its FP8 KV cache up to approximately a 64K-token context. Nemotron 3 Ultra by NVIDIA et al. (2026) therefore treats SSM-cache compression as a meaningful inference optimization even though Mamba avoids the asymptotic KV-cache growth of attention.

  • The following figure (source) shows the cache-memory crossover between Ultra’s FP8 attention KV cache and Mamba state stored at 32-, 16-, and 8-bit precision.

  • At shorter contexts, the Mamba state dominates cache footprint; as context grows, the attention cache eventually overtakes it because the latter scales with sequence length while the former remains fixed.
Why INT8 can outperform FP8

  • Ultra’s cache ablations show that bit width alone is not sufficient to characterize accuracy. Block-scaled INT8 with stochastic rounding preserves FP32-cache behavior better than FP8 E4M3 in the reported tests. The explanation offered is effective mantissa precision: although both are eight-bit formats, the integer representation with block scaling provides more useful local resolution for this particular state distribution.

  • Without cache checkpointing, the reported mean effects relative to an FP32 state are:

Cache recipe Mean accuracy change Mean verbosity increase
FP16 round-to-nearest −1.07% +9.98%
FP16 stochastic rounding +0.26% +0.36%
INT8 block-128 round-to-nearest −1.42% +9.91%
INT8 block-128 stochastic rounding +0.15% +1.28%
FP8 block-128 round-to-nearest −4.68% +25.17%
FP8 block-128 stochastic rounding −0.61% +3.12%
  • The signs in the accuracy column above are written as change relative to the FP32 baseline for readability; the source table reports “accuracy drop,” where negative values denote a small improvement. The dominant pattern is that stochastic rounding dramatically reduces degradation for every investigated format.
Periodic cache checkpointing

  • Ultra also investigates reducing the number of times recurrent state is quantized. Instead of writing a quantized Mamba state after every token, state is checkpointed only once every \(C\) steps. Inputs since the previous checkpoint are retained, and the current state is reconstructed by replaying those activations through the recurrent update when needed.

  • If ordinary recurrent storage performs approximately one quantization operation per generated token, checkpointing changes the number of state quantizations from \(O(T)\) to approximately \(O\left(\frac{T}{C}\right)\).

  • This reduces repeated rounding error and cache-write traffic, at the cost of additional activation storage and replay computation.

  • With checkpoint period \(C=8\), the severe degradation of several naive cache formats is substantially reduced. For example, FP8 round-to-nearest drops from a 4.68% mean accuracy loss and 25.17% verbosity increase to approximately 0.46% accuracy loss and 3.69% verbosity increase.

  • The released Ultra checkpoint nevertheless uses FP16 with stochastic rounding because optimized eight-bit checkpointed-state kernels were not yet the selected production path, and FP16 stochastic rounding provides strong quality retention without block-scaling overhead.

Cross-generation checkpoint portability

  • Ultra ultimately ships one NVFP4 checkpoint rather than maintaining a separate Hopper FP8 model. On Blackwell, the checkpoint uses native FP4 arithmetic. On Hopper, which lacks native FP4 tensor cores, the same stored weights execute using W4A16: NVFP4 weights provide a compact representation, while GEMMs operate with BF16 activations. The KV cache remains FP8 and the Mamba state remains FP16 with stochastic rounding. Nemotron 3 Ultra by NVIDIA et al. (2026) chooses this configuration based on end-to-end memory and serving behavior rather than peak tensor-core throughput alone.

  • An FP8 checkpoint might appear preferable on H100 because Hopper supports native FP8 tensor-core computation. At Ultra’s scale, however, the FP8 model is approximately 540 GiB. On an eight-H100 node with approximately 640 GiB aggregate HBM, this leaves only about 10 GiB per GPU for activations, KV cache, and Mamba state. The resulting memory constraint caps batch size and keeps the system in a memory-bandwidth-bound region where FP8’s higher arithmetic throughput cannot be exploited effectively.

  • The NVFP4 checkpoint is approximately 330 GiB, leaving substantially more serving headroom. It can also fit MTP weights within the same eight-GPU H100 node, whereas the larger FP8 model would require additional hardware to accommodate MTP under the reported configuration. As a result, W4A16 plus MTP produces a stronger throughput-versus-user-latency frontier than W8A8 in the evaluated Ultra serving regime.

Why Ultra does not use W4A8 on Hopper

  • A third possibility would retain NVFP4 weights but cast them into FP8 GEMMs, yielding W4A8. Ultra explicitly rejects this path. NVFP4 combines E2M1 values with E4M3 scaling and therefore represents a wider effective numerical range than unscaled E4M3 FP8. A direct FP4-to-FP8 conversion can saturate weights. Avoiding that saturation requires dequantizing through BF16 before converting to FP8:

    \[\mathrm{NVFP4} \rightarrow \mathrm{BF16} \rightarrow \mathrm{FP8}\]

  • The additional conversion operation must occur before GEMMs, eliminating the expected efficiency advantage. Because W8A8 is already slower than W4A16 in the reported memory-bound operating regime, the additional W4-to-BF16-to-FP8 path makes W4A8 unattractive without providing an accuracy benefit.

Quantization as part of model-system co-design

  • The Nemotron 3 quantization progression demonstrates that low precision is not a single scalar choice such as “use FP8” or “use FP4.” At least five separate objects require independent treatment:

    • model weights,

    • runtime activations,

    • attention KV state,

    • recurrent Mamba state,

    • numerically sensitive control or projection layers.

  • Nano shows that global attention anchors should remain at higher precision even when most of the network moves to FP8. Super shows that FP4 scale calibration and layer-wise precision search can recover nearly all BF16 quality while leaving routed experts predominantly at four bits. Ultra shows that effective bit budgets should be selected from downstream evaluations rather than tensor reconstruction alone, that recurrent cache quantization requires unbiased rounding, and that the optimal stored checkpoint may differ from the native arithmetic precision of the target accelerator.

  • The final design is therefore workload- and hardware-dependent:

      Compress the largest parameter mass aggressively
                    ↓
  Identify sensitive operators empirically
                    ↓
  Promote those operators to FP8 or BF16
                    ↓
  Compress KV cache for batch capacity
                    ↓
  Treat recurrent Mamba cache separately
                    ↓
  Use stochastic rounding to avoid recurrent drift
                    ↓
  Evaluate end-to-end throughput × latency × accuracy

  • This is consistent with the broader Nemotron 3 design philosophy: numerical format, model architecture, state representation, speculative decoding, memory capacity, and accelerator topology are optimized jointly rather than as isolated implementation choices.

Inference and Serving

Serving objectives

  • Nemotron 3 is designed around a serving objective that differs from conventional dense-Transformer optimization. The relevant metric is not simply floating-point operations per generated token, but the end-to-end trade-off among model quality, output-token throughput, per-user latency, memory footprint, communication cost, and maximum feasible concurrency. Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning by NVIDIA et al. (2026) explicitly motivates LatentMoE by separating low-latency serving, where expert-weight memory bandwidth is important, from throughput-oriented serving, where distributed all-to-all routing can dominate.

  • This distinction is especially important for agentic workloads. A conventional chatbot may generate hundreds of tokens after a relatively short prompt. A software-engineering, research, or tool-using agent can instead accumulate a long context and generate tens of thousands of tokens over reasoning and action trajectories. The cost of decode therefore becomes a first-class system constraint. Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning by NVIDIA et al. (2026) positions the 550B-A55B model specifically around long-running autonomous workloads and reports throughput comparisons using an 8K-token input and 64K-token output regime rather than a short-response chat regime.

  • At a high level, request latency can be decomposed as

    \[T_{\mathrm{request}} = T_{\mathrm{prefill}} + T_{\mathrm{decode}}\]

    • and for an output containing \(N_{\mathrm{out}}\) tokens,

      \[T_{\mathrm{decode}} \approx \frac{N_{\mathrm{out}}} {r_{\mathrm{user}}}\]
      • where \(r_{\mathrm{user}}\) denotes effective output tokens per second for that user. Consequently, long-output agents magnify even small differences in decode throughput.

Serving cost model

Prefill and decode

  • LLM serving has two qualitatively different phases.

  • During prefill, the model processes many prompt tokens in parallel. Large matrix multiplications can achieve relatively high accelerator utilization, and the workload becomes increasingly compute-intensive as the input grows.

  • During autoregressive decode, the model ordinarily generates one next token for each active sequence per decoding iteration. Arithmetic intensity falls, parameter reads and cache movement become more important, and performance can become memory-bandwidth-bound at low concurrency.

  • Nemotron 3’s components address these regimes differently:

Component Prefill effect Decode effect
Sparse MoE Executes only selected experts Reduces active parameter compute
LatentMoE Reduces expert communication Reduces expert-weight and routing bandwidth
Mamba-2 Linear sequence processing Fixed-size recurrent state rather than full-history KV state
Sparse attention anchors Retain global interaction Limits the number of layers requiring growing KV caches
GQA Reduces attention KV width Reduces KV-cache reads and storage
FP8/NVFP4 Accelerates GEMMs and reduces footprint Reduces memory movement and permits larger batches
MTP Limited role in prompt processing Can emit multiple accepted tokens per expensive verification step
  • Super’s hardware analysis makes this distinction explicit. In latency-oriented MoE serving, loading expert weights can dominate, whereas in high-throughput distributed serving, all-to-all routing traffic can dominate. LatentMoE reduces the hidden dimension used by routed experts, attacking both bottlenecks simultaneously.

KV-cache versus Mamba-state scaling

  • A major serving advantage of the hybrid architecture comes from storing recurrent Mamba state for most sequence-processing layers instead of KV vectors for every prior token.

  • For an attention model, a simplified KV-cache memory model is:

    \[M_{\mathrm{KV}} \approx 2 B T L_{\mathrm{attn}} n_{\mathrm{KV}} d_h b_{\mathrm{KV}}\]
    • where \(B\) is the number of active sequences, \(T\) is cached sequence length, \(L_{\mathrm{attn}}\) is the number of attention layers, \(n_{\mathrm{KV}}\) is the number of key-value heads, \(d_h\) is head dimension, and \(b_{\mathrm{KV}}\) is bytes per cached element. The factor of two accounts for keys and values.

  • The key dependence is:

    \[M_{\mathrm{KV}} = O(BT)\]

  • For a Mamba-2 layer, the recurrent state instead has dimensions fixed by the architecture. Schematically,

    \[M_{\mathrm{SSM}} \propto B L_{\mathrm{Mamba}} n_h d_h d_{\mathrm{state}} b_{\mathrm{state}}\]
    • with no factor proportional to the sequence length \(T\). Thus, \(M_{\mathrm{SSM}} = O(B)\) with respect to context length.

  • Nemotron 3 does not completely eliminate KV caching because periodic attention layers remain, but it reduces both the number of attention layers and the width of their KV representation through Grouped-Query Attention. This combination is particularly useful as agent histories grow from thousands to hundreds of thousands of tokens. Nemotron 3 Ultra by NVIDIA et al. (2026) attributes part of Ultra’s throughput advantage to lower attention cost and smaller KV-cache requirements from the hybrid Mamba-Attention design.

  • There is nevertheless an important practical qualification. The Mamba state is constant-sized, but at shorter contexts it can itself be large. For Ultra, the FP32 Mamba cache exceeds the FP8 KV-cache footprint until approximately 64K context at batch size one, which is why the released serving recipe compresses Mamba state to FP16 with stochastic rounding. This means the relevant advantage is asymptotic scaling with context, not that recurrent state is always smaller at every operating point.

Sparse-parameter serving

  • A sparse MoE model can contain substantially more parameters than it activates for any individual token. Nemotron 3 Super contains approximately 120.6B total parameters but activates approximately 12.7B including embeddings, while Ultra contains approximately 550B total and 55B active parameters. Sparse activation reduces arithmetic, but it does not make inactive expert parameters free: weights still occupy memory and routed tokens must be communicated to the GPUs that hold their selected experts.

  • This motivates the distinction between

    \[\text{accuracy per FLOP}\]

    • and

      \[\text{accuracy per parameter}\]

  • The latter captures costs that FLOP counts ignore, including model footprint, expert-weight memory bandwidth, expert sharding, and inter-GPU routing. Super’s LatentMoE design is specifically motivated by this serving observation.

  • At low concurrency, reading expert matrices can dominate because relatively few token activations amortize each expert-weight fetch. At high concurrency, more tokens can be batched against the same expert weights, but distributed all-to-all routing becomes increasingly important. Nemotron therefore reduces the routed representation dimensionality rather than merely reducing the number of experts.

Family throughput scaling

  • The family-level serving results are measured in generation-heavy configurations designed to expose decode efficiency.

  • Nemotron 3 Nano by NVIDIA et al. (2025) reports 3.3× higher inference throughput than Qwen3-30B-A3B-Thinking-2507 and 2.2× higher throughput than GPT-OSS-20B for an 8K-input/16K-output workload on one H200, using the better available configuration across vLLM and TensorRT-LLM.

  • Nemotron 3 Super by NVIDIA et al. (2026) extends the output length to 64K and reports up to 2.2× the throughput of GPT-OSS-120B and 7.5× that of Qwen3.5-122B. The measurements use B200 GPUs and select the better result from vLLM and TensorRT-LLM for each model.

  • Ultra scales the same design to 550B total parameters. Nemotron 3 Ultra by NVIDIA et al. (2026) reports 5.9×, 4.8×, and 1.6× higher throughput than GLM-5.1-754B-A40B, Kimi-K2.6-1T-A32B, and Qwen3.5-397B-A17B, respectively, in the reported 8K-input/64K-output setting. Ultra is measured using TensorRT-LLM on GB200, while the comparison models use vLLM; both speculative and non-speculative configurations are tested when available and the better result is selected.

  • The following figure (source) shows Nemotron 3 Ultra’s accuracy and throughput comparison against other large open models, including the relative output-token throughput measured for the 8K-input/64K-output serving regime.

  • These numbers should be interpreted as end-to-end deployment results rather than pure architectural speedups. Model precision, serving framework, speculative decoding, batch size, hardware generation, parallelism strategy, and cache representation all contribute to the measured throughput.

Speculative decoding

Draft-and-verify decoding

  • Standard autoregressive decoding requires a target-model invocation for each emitted token. Fast Inference from Transformers via Speculative Decoding by Leviathan et al. (2023) introduced a draft-and-verify formulation in which a cheaper model proposes several candidate tokens that the expensive target model verifies together, allowing multiple output tokens to be committed during one target-model iteration.

  • If the drafter proposes \(k\) tokens,

    \[\hat{x}_{t+1}, \hat{x}_{t+2}, \ldots, \hat{x}_{t+k}\]

    • and the target model accepts a prefix of length \(a\), then one verification step can advance generation by approximately

      \[a+1\]
      • tokens rather than one token, depending on the exact speculative-decoding algorithm.

  • A useful serving statistic is therefore the expected accepted length:

    \[\mathbb{E}[A]\]

  • Higher acceptance amortizes the target-model verification cost across more generated tokens.

MTP as an embedded drafter

  • Better & Faster Large Language Models via Multi-token Prediction by Gloeckle et al. (2024) trains auxiliary future-token predictors and shows that they can also be used to accelerate inference. Nemotron 3 Super and Ultra integrate this mechanism directly into their checkpoints.

  • Rather than loading a second full language model, Nemotron’s shared MTP module recursively drafts candidate tokens:

      target hidden state
        ↓
  shared MTP layer
        ↓
  candidate token 1
        ↓
  shared MTP layer
        ↓
  candidate token 2
        ↓
  ...
        ↓
  target-model verification

  • Because the draft mechanism is part of the target checkpoint, it avoids the parameter footprint and synchronization requirements of a separate general-purpose draft LLM.

  • This matters most when the target model is expensive relative to its MTP head and when draft acceptance remains high. It is less advantageous when concurrency is already high enough to saturate the accelerator with ordinary batched decoding.

Speculative decoding is workload-dependent

  • A central serving lesson from Nemotron is that speculative decoding is not universally faster.

  • At low concurrency, standard autoregressive decoding may underutilize the GPU. Spending additional compute on a cheap drafter can improve per-user generation speed because several candidate tokens can be verified together.

  • At high concurrency, the target model may already operate with large efficient batches. Drafter execution and verification then consume resources that could otherwise serve more independent requests, potentially reducing aggregate throughput.

  • The later NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 model card makes this distinction explicit: DSpark is recommended for interactive, low-concurrency inference, while maximum-throughput batch serving on a single H100 is fastest without speculative decoding.

  • This produces two distinct optimization targets:

    \(\text{interactive serving} \rightarrow \max r_{\mathrm{user}}\) versus \(\text{batch serving} \rightarrow \max r_{\mathrm{GPU}}\)

    • where \(r_{\mathrm{user}}\) is tokens per second observed by one user and \(r_{\mathrm{GPU}}\) is aggregate output tokens per second per accelerator.

  • Optimizing one does not necessarily optimize the other.

Parallel drafters: MTP, DFlash, DSpark, and LiLiCorr

  • Nemotron 3.5 makes draft architecture workload-dependent. Native MTP is embedded in the target checkpoint and recursively proposes future tokens without loading a second model. DFlash: Block Diffusion for Flash Speculative Decoding by Chen et al. (2026) instead generates a draft block in one diffusion-style forward pass, while DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation by Cheng et al. (2026) introduces lightweight sequential dependence across otherwise parallel draft positions. Lightning 3.5 releases both DFlash and DSpark checkpoints in addition to native MTP.

  • Super 3.5 adds LiLiCorr: Lightweight Likelihood Correlation of Parallel Drafts for Speculative Decoding by Rusanovsky et al. (2026). A parallel drafter first produces per-position candidate distributions; LiLiCorr retains the top candidates at each slot and processes the candidate lattice jointly with a small Transformer, producing adjacent-candidate compatibility scores. A cheap left-to-right walk through those precomputed scores restores cross-token coherence without another neural-network evaluation for every selected token.

  • Super’s released LiLiCorr drafter uses a block of 8 positions, corresponding to 7 drafted tokens, and contains 1.13B parameters, of which 37M belong to the correlation head. The drafter body is trained by full-distribution distillation from the target model, and the head combines teacher-forced cross-entropy with a calibration term that matches the target’s renormalized candidate probabilities. This makes the parallel drafter useful under both greedy and probabilistic speculative verification.

  • Bolt uses the same LiLiCorr design principle for local inference. Its drafter also uses 7 drafted slots and a 1.13B-parameter body/head configuration, then applies weight-only NVFP4 quantization to latency-dominant drafter projections. On the report’s SPEED-Bench setup at concurrency 64, drafter quantization preserves acceptance length while reducing drafting latency; the final RTX Spark configuration is optimized around LiLiCorr for low-concurrency local-agent workloads.

Nemotron 3.5 Super serving: interactive agentic throughput

  • Nemotron 3.5 Super and Lightning: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Models for Agentic AI by NVIDIA et al. (2026) evaluates serving against a 20-turn agentic proxy rather than a one-turn synthetic decode alone. Each turn contributes 2,048 new input tokens and 1,024 output tokens, while a 51.2K-token shared prefix and prior conversation state can be reused. An operating point is considered usable when per-user generation remains at or above 50 output tokens/s, after which the objective is to maximize aggregate output tokens/s/GPU.

  • The following figure (source) shows Nemotron 3.5 Super serving performance for BF16 and NVFP4 under this multi-turn workload, reporting both maximum aggregate throughput at the 50-token/s/user interactivity threshold and the per-user rate at minimum latency.

  • Under the reported configuration, NVFP4 raises maximum usable throughput by 1.86x on H100 and 2.43x on GB200 relative to BF16. On one GB200, a ReplaySSM-enabled configuration reaches up to 4,856 output tokens/s/GPU while maintaining more than 50 tokens/s/user.

Prefix reuse in a hybrid Mamba-attention model

  • Prefix caching is more complicated for a hybrid model because attention caches token-local KV state while each Mamba layer has a much larger recurrent checkpoint that summarizes the sequence. On the Super 3.5 serving configuration, one Mamba cache page can occupy roughly the same bytes as about 4K tokens of attention KV state, so the closest reusable Mamba checkpoint can be much coarser than the attention prefix boundary. Speculative decoding additionally needs the target hidden state immediately before the drafted suffix.

  • The serving stack therefore retains an optional additional recurrent checkpoint near the speculative replay boundary. A fine-grained 16-token shared-prefix unit can resume near \(T-16\) instead of replaying from a Mamba page around \(T-L_{\mathrm{page}}\), reducing target recomputation while preserving a valid recurrent state. This makes prefix reuse and speculative decoding composable rather than competing optimizations.

ReplaySSM and checkpointed recurrent-state updates

  • ReplaySSM: Cache SSM Inputs, Not State by Liou et al. (2026) observes that hybrid SSM serving can become state-I/O-bound even though the recurrent state is constant in sequence length. Nemotron 3.5 Super incorporates the same checkpoint-and-replay idea: instead of writing every speculative intermediate Mamba state, the engine stores a checkpoint plus recent \(k_i\), \(v_i\), and decay inputs, then reconstructs state when necessary.

  • For a simplified recurrence \(S_{\mathrm{new}}= \alpha S_{\mathrm{old}}+k^\top v\), buffering accepted recurrence inputs reduces state-related write traffic from approximately \(T+1\) writes for a speculative block of length \(T\) to approximately \(1+\frac{1}{L}\) amortized writes for replay-buffer length \(L\). In the reported GB200 experiments, ReplaySSM improves aggregate output throughput by 4.7% to 6.4% and per-user generation speed by 6.4% to 12.5% across matched configurations.

Disaggregated prefill and decode

  • Super 3.5 is also evaluated with disaggregated serving on 11 NVIDIA VR200 GPUs. A 7-GPU prefill pool builds cache state and transfers it to a 4-GPU decode pool, allowing decode workers to maintain larger decode batches instead of being interrupted by long prefills. At the best reported usable point, disaggregation improves output tokens/s/GPU by 10.3% and median per-user output speed by 61%, while median TTFT increases from 0.49s to 1.12s and remains below the 5s target.

Nemotron 3.5 Bolt serving for local agents

  • Nemotron 3.5 Bolt 74B-A9B 24GB: A Hybrid Vision-Language Model for Local AI by NVIDIA et al. (2026) provides two serving paths for the compressed model: GGUF checkpoints through llama.cpp and ModelOpt-exported checkpoints through vLLM. The local operating point is strongly weight-bandwidth-bound, which is why 2-bit expert storage and a one-pass parallel drafter are central to latency rather than merely reducing disk size.

  • On RTX Spark, the final IQ2_XXS model with LiLiCorr reports 83.99 tokens/s on AIPerf SPEED-Bench-Coding. The same release reports a 600.80-second replay duration on AA-AgentPerf Local, a complementary end-to-end measure over heterogeneous agent trajectories. These figures should be read as full-system measurements because model precision, drafter, runtime, operating system, and hardware are part of the configuration. NVIDIA Nemotron 3.5 Bolt 74B-A9B 24GB documents the public local-serving setup.

Deployment regimes for Nemotron 3.5 Lightning

  • The Lightning model card provides unusually concrete serving configurations, illustrating how the same 30B-A3B checkpoint is deployed differently depending on hardware and service-level objective.
Serving target Recommended configuration Speculation Validated context
DGX Spark GB10 vLLM + DSpark DSpark, 3 draft tokens 1M
1× H100, maximum batch throughput vLLM None 1M
1× H100, interactive vLLM + DSpark DSpark, 3 draft tokens 1M
8× H100, long context vLLM, TP8 + expert parallelism None 1M
1× GB200 vLLM + DSpark DSpark, 5 draft tokens 1M
A100 / Ampere vLLM W4A16 None 1M
Local llama.cpp GGUF Engine-dependent VRAM-dependent
Local Ollama / LM Studio Packaged local model Engine-dependent VRAM-dependent

  • For a single H100 optimized for aggregate throughput, the reference configuration uses up to 256 sequences, a maximum batched-token budget of 16,384, prefix caching, asynchronous scheduling, specialized Mamba execution, and no speculative drafter.

  • For interactive H100 serving, the configuration reduces concurrency to at most 128 active sequences and enables DSpark with three speculative tokens. The model card describes this operating point as targeting more than 40 output tokens per second per user. FP16 Mamba-cache storage is combined with stochastic rounding using five Philox rounds.

  • For eight H100 GPUs, the long-context configuration combines tensor parallelism of eight with expert parallelism. This is a natural fit for the MoE architecture: tensor parallelism shards dense computation, while expert parallelism places different experts across devices.

  • On a single GB200, Lightning uses DSpark with a larger five-token speculative draft. The same model checkpoint can also run through W4A16 kernels on Ampere-class hardware such as A100, trading native FP4 arithmetic for compatibility with older accelerators.

Serving context length is also a memory-allocation decision

  • Although Lightning supports a 1M-token context and the H100 and GB200 recipes serve this window by default, the model card recommends reducing the configured maximum context when memory is constrained or when higher concurrency requires more cache headroom.

  • This exposes a fundamental serving trade-off. If total memory available after storing model weights and recurrent state is \(M_{\mathrm{free}}\), then the approximate number of simultaneously cached attention tokens satisfies

    \[B T \lesssim \frac{ M_{\mathrm{free}} }{ 2L_{\mathrm{attn}} n_{\mathrm{KV}} d_h b_{\mathrm{KV}} }\]

  • Thus, reserving capacity for a 1M-token maximum sequence can constrain batch concurrency even when most requests are substantially shorter. A production server may therefore deliberately configure a smaller context ceiling to increase the number of concurrent users.

Prefix caching

  • The Lightning H100 recipes enable prefix caching. In agentic systems, multiple requests or successive turns can share a substantial prompt prefix containing system instructions, tool schemas, repository context, or previous conversation history. Reusing cached prefix computation avoids repeating the complete prefill for that unchanged prefix.

  • Prefix caching primarily reduces \(T_{\mathrm{prefill}}\) rather than the marginal cost of generating each new decode token. It is therefore complementary to MTP or DSpark, which primarily target the decoding phase.

Asynchronous scheduling

  • The reference vLLM configurations also use asynchronous scheduling. Rather than forcing all requests through globally synchronized generation steps, the serving engine can continuously admit, schedule, and retire sequences as resources become available. This is useful for agent workloads because response lengths can vary dramatically, and synchronization around the longest request would otherwise leave substantial compute capacity idle. The Lightning H100 and GB200 recipes explicitly enable asynchronous scheduling.

Serving engines

  • Nemotron is not tied to a single serving runtime. The supplied deployment recipes cover vLLM, TensorRT-LLM, and SGLang, with different optimization paths available in each runtime.

    • vLLM is used throughout the family for open production serving and exposes Nemotron-specific controls for Mamba cache representation, expert and tensor parallelism, asynchronous scheduling, prefix caching, tool-call parsing, and multiple speculative decoding methods.

  • TensorRT-LLM provides an NVIDIA-specific optimized runtime and is used to obtain Ultra’s reported GB200 throughput. The Lightning TensorRT-LLM recipe configures an FP8 KV cache, FP16 Mamba state with stochastic rounding, NVFP4 GEMMs, MTP speculative decoding with draft length three, chunked prefill, CUDA graphs, and a maximum batch size of eight in the documented single-H100 setup.

  • SGLang exposes corresponding Nemotron serving controls and can run balanced non-speculative configurations or enable MTP, DFlash, and DSpark explicitly. On Hopper, the NVFP4 checkpoint follows a W4A16 path rather than native FP4 arithmetic.

  • This multi-runtime support matters because the fastest runtime depends on the model, hardware, precision, and workload. Super’s benchmark therefore measures both vLLM and TensorRT-LLM and reports the best configuration for each comparison model rather than assuming one serving engine is universally optimal.

MTP across training and serving

  • An unusual property of the Nemotron stack is that the same speculative-decoding mechanism appears in both deployment and post-training infrastructure.

  • During ordinary serving, MTP reduces user-visible latency by allowing multiple output tokens to be accepted per target-model verification.

  • During RL, rollout generation is itself an inference workload, often involving very long trajectories. Ultra therefore enables MTP during RLVR and MOPD generation. With five speculative MTP tokens, the reported mean rollout-generation time improves by 1.46×, with the largest gain occurring in the long-tail trajectories that generate many tokens and reach the end of a batch at low concurrency.

  • This reveals a broader systems feedback loop:

      better speculative decoding
            ↓
  cheaper RL rollouts
            ↓
  more efficient post-training
            ↓
  better agent policy
            ↓
  longer / more capable deployed trajectories
            ↓
  greater value from efficient decoding

  • Thus, inference optimization is not only a deployment concern. It changes the cost of generating the training trajectories required to improve the model itself.

Throughput-latency trade-off

  • The Lightning recipes provide a concrete example of the throughput-latency Pareto frontier.

  • Suppose a GPU can support \(B\) concurrent sequences. Aggregate throughput can be written schematically as

    \[R_{\mathrm{GPU}} = \sum_{i=1}^{B} R_i\]

  • Increasing \(B\) generally improves weight reuse and accelerator utilization until memory, communication, or compute saturation is reached. However, increasing concurrency also means each individual request receives a smaller fraction of the accelerator’s instantaneous capacity.

  • Consequently, \(\arg\max_B R_{\mathrm{GPU}}(B)\) does not generally equal \(\arg\max_B R_{\mathrm{user}}(B)\).

  • Nemotron’s recommended configurations reflect exactly this distinction. A single H100 batch server uses up to 256 sequences and disables speculative decoding, while the interactive configuration caps concurrency at 128 and enables DSpark.

  • For an interactive coding assistant, user-perceived tokens per second and time-to-completion may dominate. For an offline generation service or high-volume sub-agent fleet, aggregate tokens per second per GPU may be the more important objective.

Distributed serving

  • As model size increases, serving requires increasingly sophisticated distributed execution.

  • Tensor parallelism shards individual matrix operations across devices. Expert parallelism partitions MoE experts so that each device stores only a subset of the total expert pool. Data parallelism replicates model execution for separate request groups when sufficient memory is available.

  • For a routed MoE layer, expert parallelism creates an all-to-all communication pattern:

\[\text{token representations} \;\longrightarrow\; \text{router selects experts} \;\longrightarrow\; \text{all-to-all token dispatch} \;\longrightarrow\; \\ \text{expert computation} \;\longrightarrow\; \text{all-to-all result return} \;\longrightarrow\; \text{combine routed outputs}\]

  • The communication volume depends on the routed representation dimension and number of selected experts. This is precisely why LatentMoE’s lower-dimensional expert space matters for distributed serving, not merely for arithmetic FLOPs. Nemotron 3 Super by NVIDIA et al. (2026) identifies all-to-all routing as a principal bottleneck in throughput-oriented MoE inference.

  • For Lightning, an eight-H100 long-context configuration explicitly combines TP8 with expert parallelism. Larger Super and Ultra deployments likewise rely on multi-GPU placement and optimized MoE communication, with Ultra’s published maximum-throughput results obtained on GB200-class systems.

Local deployment

  • The smaller Nemotron branch is also intended to operate outside large data-center clusters. The NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 model card provides deployment paths through llama.cpp, Ollama, and other local runtimes in addition to vLLM, TensorRT-LLM, and SGLang.

  • For Ollama, the packaged configuration dynamically selects a context limit from available VRAM: the documented defaults are approximately 4K below 24 GB, 32K between 24 and 48 GB, and 256K above 48 GB, although users can request a larger context at the cost of possible CPU offloading. Official GGUF weights are also available for llama.cpp.

  • This illustrates another consequence of the hybrid design. A model can architecturally support 1M tokens while a particular local deployment exposes a much smaller practical context because model weights, recurrent state, KV cache, CUDA graphs, and runtime workspace must all compete for the same physical memory.

Model-family routing

  • Nemotron’s tiered family makes inference routing itself an architectural decision. Small, inexpensive models can execute high-volume sub-tasks, while larger models can be reserved for requests requiring greater planning or reasoning capability. The broader NVIDIA Nemotron model ecosystem describes this style of model routing through NeMo Switchyard, in which requests can be assigned to different models according to capability, latency, and cost requirements.

  • A practical agent deployment can therefore operate hierarchically:

\[\text{incoming agent step} \;\longrightarrow\; \text{task / difficulty router} \;\longrightarrow\; \left\{ \begin{array}{ll} \text{Lightning / Nano} & \text{routine, high-volume sub-tasks} \\[4pt] \text{Super} & \text{stronger reasoning + agent work} \\[4pt] \text{Ultra} & \text{difficult planning / orchestration} \end{array} \right.\]
  • The systems objective becomes minimizing total task cost rather than maximizing the capability of the model used for every individual step.

End-to-end serving design

  • Taken together, Nemotron 3’s serving architecture relies on multiple layers of optimization that compound rather than substitute for one another:

    • sparse MoE reduces active computation;

    • LatentMoE reduces expert memory traffic and all-to-all volume;

    • Mamba-2 replaces most sequence-length-dependent attention state with fixed-size recurrence;

    • GQA compresses the KV caches that remain;

    • NVFP4 and FP8 reduce model and cache memory traffic;

    • stochastic rounding makes low-precision recurrent-state storage viable;

    • MTP, DSpark, or DFlash can reduce serial decode steps at low concurrency;

    • high-concurrency workloads can instead disable speculation and rely on batching;

    • prefix caching reduces repeated prefill work;

    • tensor and expert parallelism distribute larger checkpoints across accelerator fabrics;

    • the serving engine itself is selected and configured according to hardware and workload.

  • The resulting throughput gains are therefore best understood as full-stack effects. Nemotron 3 does not obtain its serving profile from Mamba, MoE, quantization, or speculative decoding independently. It combines architecture, numerical representation, cache design, draft generation, scheduling, batching, parallelism, and hardware-aware runtime selection into one inference system.

Evaluation and Capabilities

Evaluation philosophy

Family scaling

Nano operating point

  • Nemotron 3 Nano demonstrates that the 30B-total, approximately 3B-active architecture can remain competitive across heterogeneous capabilities despite its low active parameter count. The model is not uniformly best across every individual benchmark, but it produces a broad capability profile while retaining the throughput advantages discussed earlier.

  • Representative Nano results include:

Capability Benchmark Nemotron 3 Nano
General knowledge MMLU-Pro 78.30
Mathematics AIME25, no tools 89.06
Mathematics AIME25, with tools 99.17
Science reasoning GPQA, no tools 73.04
Science reasoning GPQA, with tools 75.00
Code LiveCodeBench v6 68.25
Formal reasoning MiniF2F pass@1 50.03
Formal reasoning MiniF2F pass@32 79.92
Software engineering SWE-Bench, OpenHands 38.76
Tool use TauBench V2 average 49.04
Function calling BFCL v4 53.76
Instruction following IFBench 71.51
Chat Arena-Hard-V2 average 67.65
Long context RULER 256K 92.92
Long context RULER 512K 91.25
Long context RULER 1M 86.34

  • Two aspects are particularly notable.

  • First, tools can materially change measured reasoning performance. Nano increases from 89.06 to 99.17 on AIME25 when tool use is enabled, and from 73.04 to 75.00 on GPQA. The model should therefore be interpreted partly as a policy for interacting with external computation rather than solely as a closed-book reasoner.

  • Second, Nano’s context degradation is relatively gradual. RULER falls from 92.92 at 256K to 91.25 at 512K and 86.34 at 1M. A nominal 1M-token context limit alone would not establish useful million-token reasoning, but the RULER results show substantial retained capability at the maximum evaluated length.

  • The long-context result is also configuration-specific. RULER is evaluated with reasoning disabled, while AA-LCR uses reasoning-enabled inference. This distinction matters because a model’s ability to retain or retrieve information from a long context is not identical to its ability to perform long-context reasoning with a large generated chain of thought.

Super agentic scaling

  • Super’s principal capability gain is not simply an across-the-board increase in academic benchmark scores. The larger improvement occurs in agentic workloads, reflecting the expanded SFT data, larger RL environment mixture, dedicated software-engineering training, PivotRL, and longer tool trajectories described in the post-training section.

  • Representative results are:

Capability Benchmark Nemotron 3 Super
General knowledge MMLU-Pro 83.73
Mathematics AIME25, no tools 90.21
Mathematics HMMT Feb25, no tools 93.67
Mathematics HMMT Feb25, with tools 94.73
Science reasoning GPQA, no tools 79.23
Science reasoning GPQA, with tools 82.70
Code LiveCodeBench v5 81.19
Science coding SciCode 42.05
Software engineering SWE-Bench, OpenHands 60.47
Software engineering SWE-Bench, OpenCode 59.20
Terminal interaction Terminal Bench Core 2.0 31.00
Tool use TauBench V2 average 61.15
Search BrowseComp 31.28
SQL agent BIRD 41.80
Instruction following IFBench 72.56
Long context RULER 256K 96.83
Long context RULER 512K 95.22
Long context RULER 1M 91.64
Multilingual reasoning MMLU-ProX 79.36

  • The Nano-to-Super jump is especially large for repository-level software engineering: \(38.76 \rightarrow 60.47\) on the reported OpenHands SWE-Bench configuration.

  • Similarly, Terminal Bench increases from Nano’s 8.51 on its hard-subset evaluation to Super’s 25.78 on the corresponding hard subset. These gains are consistent with the explicit increase in software-engineering SFT, terminal environments, long-horizon RL, and dedicated SWE-RL rather than model scaling alone.

  • Super’s evaluation also shows that model ranking is highly benchmark-dependent. It is competitive with GPT-OSS-120B across the reported reasoning tasks but trails Qwen3.5-122B on several academic benchmarks. For example, Super obtains 79.23 on GPQA versus 86.60 for Qwen3.5, and 18.26 versus 25.30 on HLE. Conversely, Super performs strongly on SciCode and remains competitive on multiple agentic evaluations.

  • The intended advantage is therefore not a claim of universal benchmark dominance. It is a different accuracy-throughput-agentic operating point.

Long-context scaling

  • Long-context capability improves consistently across the text family.
RULER Nano Super Ultra BF16
256K 92.92 96.83 Not reported in final Ultra table
512K 91.25 95.22 Not reported in final Ultra table
1M 86.34 91.64 94.7

  • The improvement is notable because the larger models retain the same basic hybrid principle instead of adding proportionally more attention layers. Long-context capability is therefore compatible with a predominantly Mamba-based sequence stack when periodic global attention anchors and dedicated long-context continued pre-training are retained.

  • However, RULER should not be interpreted as proving arbitrary reasoning over all million-token inputs. RULER by Hsieh et al. (2024) deliberately improves upon simple needle retrieval, but it remains a synthetic controlled benchmark. The separate AA-LCR evaluations are useful precisely because long-context reasoning is a broader property than retrieval or aggregation alone.

  • Super scores 58.31 on AA-LCR while Ultra reaches 65.4, compared with Nano’s 35.85. Thus, scaling and post-training produce a considerably larger relative gain on the reasoning-intensive long-context evaluation than on RULER, where Nano was already strong.

Ultra long-horizon agents

  • Ultra’s strongest improvements appear in the agentic domains targeted by its specialized teachers and MOPD pipeline. The final BF16 checkpoint reports:
Capability Benchmark Nemotron 3 Ultra
Terminal agent Terminal Bench 2.1 56.4
Professional workflows GDPVal 46.7
Software engineering SWE-Bench Verified 70.7
Multilingual software engineering SWE-Bench Multilingual 67.7
Search agent ProfBench 56.0
Agentic coding PinchBench 90.0
Tool use TauBench V3 Airline 81.5
Tool use TauBench V3 Retail 86.4
Tool use TauBench V3 Telecom 92.9
Tool use TauBench V3 Banking 22.6
Tool use TauBench V3 average 70.9
Web research BrowseComp 44.4
Competitive programming IOI 2025 570 / 600
Science reasoning GPQA 87.0
Scientific coding SciCode 44.6
Hard reasoning HLE 26.7
Instruction following IFBench 81.7
Long-context reasoning AA-LCR 65.4
Long context RULER 1M 94.7
  • The spread within TauBench is also informative. Airline, retail, and telecom are all above 80, while banking remains at 22.6. A single aggregate “tool-use score” would therefore hide substantial domain variation. Tool competence depends on the domain’s policy constraints, available actions, interaction patterns, and the degree to which those trajectories resemble the agent environments encountered during post-training.

Post-training capability acquisition

  • Ultra provides a useful decomposition of where its final capabilities emerge by reporting evaluation after SFT, after multi-environment RLVR, and after two rounds of MOPD.
Benchmark SFT RLVR MOPD 1 MOPD 2
Terminal Bench 2.0 34.5 44.5 50.8 54.0
GDPVal 23.2 28.9 46.7 46.7
SWE-Bench Verified 63.5 65.8 70.1 71.7
TauBench Telecom 55.7 82.7 91.2 92.9
BrowseComp 14.3 31.0 41.0 44.4
LiveCodeBench 85.5 87.4 90.0 89.0
HLE 19.7 25.6 25.9 26.7
OmniScience non-hallucination 4.8 46.3 77.9 78.7
IFBench 62.3 78.4 80.0 81.7
Multi-Challenge 53.3 60.3 62.8 63.8

  • This table illustrates that different stages perform different roles.

  • RLVR produces especially large gains when the task has a clear verifier. TauBench Telecom rises from 55.7 after SFT to 82.7 after RLVR. BrowseComp rises from 14.3 to 31.0. OmniScience non-hallucination jumps from 4.8 to 46.3.

  • MOPD then produces additional gains in domains where specialized teachers have better action-selection policies. Terminal Bench rises from 44.5 after RLVR to 54.0 after MOPD2, SWE-Bench from 65.8 to 71.7, TauBench Telecom from 82.7 to 92.9, and BrowseComp from 31.0 to 44.4.

  • The gains are much smaller on HLE: \(25.6 \rightarrow 26.7\).

  • This difference supports the limitation discussed in the post-training section. MOPD is effective when a student already samples trajectories similar enough to the teacher that token-level preferences can improve decision making. It is less effective when the teacher’s advantage consists of reasoning trajectories the student almost never discovers.

Agent benchmarks

  • It is useful to separate the major agent evaluations rather than combine them into a single score.

  • SWE-bench by Jimenez et al. (2023) measures repository-level problem solving: understand an issue, inspect a codebase, modify files, and satisfy tests.

  • Terminal-Bench by Merrill et al. (2026) measures environment interaction through a command line, where an agent must sequence shell commands and manipulate state to complete realistic tasks.

  • τ-bench by Yao et al. (2024) measures interactive tool use under domain policies and evaluates whether the final environment or database state matches the intended goal.

  • These benchmark types stress different error modes:

\[\begin{aligned} \text{SWE-Bench} &\rightarrow \text{issue understanding} \rightarrow \text{repository navigation} \rightarrow \text{code modification} \rightarrow \text{test passing} \\ \text{Terminal-Bench} &\rightarrow \text{environment understanding} \rightarrow \text{command selection} \rightarrow \text{state manipulation} \rightarrow \text{verified terminal outcome} \\ \tau\text{-bench} &\rightarrow \text{user interaction} \rightarrow \text{policy compliance} \rightarrow \text{API/tool sequencing} \rightarrow \text{correct database state} \end{aligned}\]
  • Ultra’s improvement across all three is therefore stronger evidence of general agentic improvement than a gain on any single tool benchmark would be.

Coding capability

  • The Nemotron evaluations distinguish at least three forms of coding capability.

  • LiveCodeBench tests self-contained algorithmic coding. LiveCodeBench by Jain et al. (2024) uses newly released programming-contest problems to reduce contamination and evaluate current code-generation ability.

  • SciCode targets scientific programming, where the model must translate scientific concepts and mathematical specifications into working implementations.

  • SWE-Bench measures repository-level engineering, where the relevant unit of reasoning is no longer a single function but an existing software system. SWE-bench by Jimenez et al. (2023) explicitly requires models to coordinate modifications across real codebases.

  • The family improves along all three axes, but the largest proportional gains are in repository-level software engineering:

    \[\text{SWE-Bench OpenHands: } 38.76_{\text{Nano}} \rightarrow 60.47_{\text{Super}}\]
    • while the final Ultra evaluation reports 70.7 on SWE-Bench Verified under its corresponding harness. Because harnesses and benchmark versions differ, these values should not be interpreted as a strictly controlled scaling curve, but the overall progression is consistent with the increasingly agentic post-training recipe.

Instruction following

  • Precise instruction following is treated as a verifiable capability rather than purely as preference alignment. Generalizing Verifiable Instruction Following by Pyatkin et al. (2025) introduces IFBench to test previously unseen output constraints and shows why performance on memorized constraint types is insufficient.

  • Nemotron progresses from 71.51 on Nano to 72.56 on Super and 81.7 on Ultra’s final BF16 checkpoint. More revealingly, Ultra’s stage-wise evaluation moves from 62.3 after SFT to 78.4 after RLVR, then 81.7 after the second MOPD round.

  • This makes instruction following one of the clearest examples of RLVR acting directly on a capability for which programmatic or structured verification is available.

Robustness and modality coverage

Quantization robustness

  • The evaluation suite is also used to verify that low-precision deployment does not substantially change the model’s behavioral profile.

  • For Ultra, BF16 and NVFP4 results remain close across most reported tasks. SWE-Bench Verified changes from 70.7 to 69.5, GPQA from 87.0 to 87.9, IFBench from 81.7 to 82.3, AA-LCR from 65.4 to 65.5, and RULER 1M from 94.7 to 94.0. Some benchmarks move slightly upward and others downward, suggesting quantization error is not expressed as a uniform scalar degradation.

  • The more meaningful deployment criterion is therefore not

    \[\text{quantized score} = \alpha \times \text{BF16 score}\]
    • for one constant \(\alpha\).

  • Instead, quality must be checked across capability categories because individual operators and recurrent-state errors can influence coding, verbosity, long-context behavior, and reasoning differently.

Multilingual evaluation

  • Multilingual capability is evaluated separately from English reasoning because high English benchmark performance does not imply equivalent reasoning across languages. MMLU-ProX: A Multilingual Benchmark for Advanced Large Language Model Evaluation by Xuan et al. (2025) extends reasoning-oriented MMLU-Pro-style evaluation across 13 typologically diverse languages and documents substantial performance differences between high- and lower-resource languages.

  • Nano reports 59.50 on MMLU-ProX, while Super reaches 79.36. Super also reports 86.67 on WMT24++ English-to-other-language translation. These metrics suggest that the larger model substantially improves multilingual reasoning, although Super still trails Qwen3.5-122B on the reported MMLU-ProX comparison.

Nano Omni multimodal capability

  • Nemotron 3 Nano Omni extends evaluation from text agents to perception-grounded agents. The model uses the Nano text backbone but adds image, video, and audio processing, and is evaluated on document understanding, OCR, GUI perception, speech recognition, audio understanding, spoken interaction, video understanding, and joint audio-video reasoning.

  • Its audio evaluation illustrates the distinction between transcription and reasoning. On OpenASR, where lower word-error rate is better, Nano Omni averages 5.95 versus 6.55 for Qwen3-Omni. On VoiceBench, which evaluates spoken interaction and reasoning, Nano Omni averages 89.4 versus 88.8 for Qwen3-Omni.

  • For joint audio-video reasoning, Nano Omni reports:

Benchmark Reasoning off Reasoning on
DailyOmni 74.5 74.1
WorldSense 55.2 55.4

  • The relatively small reasoning-on versus reasoning-off difference on these two aggregate metrics indicates that enabling explicit reasoning does not uniformly improve every multimodal benchmark. The value of additional inference-time reasoning remains task-dependent.

  • Document and GUI tasks are also emphasized. The reported suite includes ChartQA, DocVQA, TextVQA, InfoVQA, OCR reasoning, ScreenSpot, ScreenSpot-v2, ScreenSpot-Pro, OSWorld, and VideoMME, reflecting an intended use case in agents that must perceive interfaces and documents rather than operate purely through text APIs.

Multimodal efficiency

  • Nano Omni preserves the family’s emphasis on capability per serving cost. On B200, the report measures more than 500 output tokens per second for single-stream inference and approximately 5,000 output tokens per second at maximum concurrency on a multi-document workload. At a fixed target of 50 output tokens per second per user, it reports 9× the aggregate output throughput of Qwen3-Omni on long-video workloads and 7.5× on multi-document workloads.

  • The evaluation philosophy therefore extends the definition of model quality:

    \[\text{useful capability} \neq \text{benchmark accuracy alone}\]

  • For a multimodal agent, time to first token, output rate, context capacity, visual-token count, audio-token rate, and the number of concurrently serviceable users can determine whether a benchmark capability is practical in an interactive system.

Benchmark interpretation

  • The Nemotron reports provide extensive comparison tables, but several methodological details matter.

  • Different external models sometimes use officially reported scores, public aggregators, or NVIDIA-run evaluations when official numbers are unavailable. Some benchmarks use NVIDIA’s NeMo Evaluator and Skills harnesses, while others rely on dedicated benchmark implementations or internal scaffolding. SWE-Bench additionally depends heavily on the agent harness, with Super reporting separate OpenHands, OpenCode, and Codex configurations.

  • For example, Super’s SWE-Bench scores vary from 53.73 under its Codex harness to 60.47 with OpenHands and 59.20 with OpenCode. The model checkpoint is unchanged, but the surrounding agent system changes.

  • This is important for agent evaluation because the measured system is

    \[\text{agent performance} = f( \text{model}, \text{prompt}, \text{tools}, \text{harness}, \text{environment}, \text{sampling}, \text{context management} )\]

  • A benchmark score for an agentic model is therefore less checkpoint-intrinsic than a conventional multiple-choice accuracy number.

Nemotron 3.5 specialization and compression

  • Nemotron 3.5 Super and Lightning: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Models for Agentic AI by NVIDIA et al. (2026) shows that the 3.5 generation is not simply a deployment refresh. The final Super 3.5 model reaches 51.78% on SWE-bench Pro while using approximately 4.6 times fewer active parameters than Nemotron 3 Ultra. Its software-engineering mid-training also improves DeepSWE pass@1 from 6.4% to 10.4% in the controlled experimental-SFT comparison and materially improves robustness when the same model is evaluated under different agent harnesses.

  • Test-time scaling adds a second axis beyond parameter and token scaling. Parallel independent agent attempts followed by self-verification improve the reported Terminal-Bench 2.1 result from 62.7% to 71.9%, DeepSWE from 24.6% to 34.5%, and BrowseComp from 71.5% to 79.8% under the evaluated setup. This demonstrates that agentic evaluation depends on both model policy and inference-time orchestration. NVIDIA Nemotron 3.5 Super and Lightning reports these parallel-agent results together with the single-agent baselines.

  • The following figure (source) shows Nemotron 3.5 Super benchmark performance against six open-model baselines across agentic tasks, code generation, mathematical and scientific reasoning, multimodal understanding, long-context reasoning, and chat preference.

  • Bolt provides a useful compression case study because the report evaluates the model before and after recovery. Immediately after Star Elastic compression, before the later SFT/RL healing and 2-bit quantization stages, the unweighted mean across nine reported metrics falls from 57.58 for the Super 3.5 VL parent to 54.74 for the 74B-A9B compressed student, a decrease of 2.85 points or 4.94% relative.

  • The following figure (source) shows the compression-quality comparison between Super 3.5 VL, the compressed Bolt checkpoint, and reference models, including mean benchmark score against total and active parameter counts plus per-benchmark results.

  • After recovery and quantization, the 24GB Bolt checkpoint reports 28.47 on GDPVal-AA-V2, 50.98% on Terminal-Bench 2.1, 17.01% on Tau3 Banking, 25.30% on HLE without tools, 41.02% on SciCode, 86.43% on GPQA Diamond, and 64.06% on AA-LCR in the report’s evaluation table. The important comparison is not that compression is lossless, because the 2-bit stage is explicitly lossy, but that the larger compressed-and-quantized model remains competitive with smaller local models at a similar memory footprint.

  • The 3.5 results also strengthen an evaluation principle already visible in the earlier family: benchmark numbers for agents are inseparable from harness configuration. Super’s mid-training gains differ materially across OpenHands, Pi, OpenCode, Hermes, Codex, and Claude Code harnesses, so a single SWE-bench number does not uniquely characterize software-engineering capability.

Scaling takeaways

  • Across Nano, Super, and Ultra, capability growth is not uniform across domains.

  • Nano already provides strong self-contained reasoning and million-token retrieval at a small active parameter budget. Super produces large gains in code, software engineering, terminal use, and long-context reasoning through a combination of greater model capacity and more agent-oriented post-training. Ultra then derives its largest additional gains from specialized teachers, environment RL, and MOPD, particularly on long-horizon tool-use tasks.

  • The trajectory can be summarized conceptually as a cumulative progression:

\[\underbrace{ \mathcal{C}_{\mathrm{Nano}} = \text{efficient general reasoning} + \text{tool use} + \text{long context} }_{\text{Nano}} \;\longrightarrow\; \underbrace{ \mathcal{C}_{\mathrm{Super}} = \mathcal{C}_{\mathrm{Nano}} + \text{stronger reasoning} + \text{stronger coding} + \text{software engineering} + \text{terminal / tool agents} }_{\text{Super}} \;\longrightarrow\; \underbrace{ \mathcal{C}_{\mathrm{Ultra}} = \mathcal{C}_{\mathrm{Super}} + \text{frontier-scale reasoning} + \text{specialized agent teachers} + \text{stronger long-horizon execution} + \text{MOPD consolidation} }_{\text{Ultra}}\]
  • The evaluation results therefore reinforce the architectural premise of Nemotron 3. The family is not optimized solely to maximize static benchmark accuracy. It aims to move the combined frontier of capability, context length, agentic execution, and inference efficiency. The most pronounced benefits appear when the model is used in the long-output, tool-using, environment-interacting regimes for which its hybrid architecture and post-training stack were explicitly designed.

Ecosystem, Extensions, Limitations, and Key Takeaways

Open model-systems stack

  • Nemotron 3 is broader than the Nano, Super, and Ultra inference checkpoints. The release strategy exposes multiple layers of the model-development stack: base checkpoints, aligned checkpoints, low-precision deployment variants, reward models, post-training datasets, reinforcement-learning environments, and training recipes. NVIDIA Nemotron 3 Ultra by NVIDIA et al. (2026), for example, releases the 550B-A55B base model, post-trained BF16 model, NVFP4 model, and GenRM together with training recipes, data, and RL environments.

  • The central development hub is the NVIDIA Nemotron Developer Repository, which provides training recipes, deployment guides, and end-to-end examples rather than only model-loading code. The public repository includes separate guidance for model training, data preparation, distributed execution, fine-tuning, deployment, and agentic use.

  • The public NVIDIA Nemotron v3 Hugging Face collection exposes the model side of this ecosystem. In addition to core Nemotron 3 checkpoints, the collection now includes Ultra BF16 and NVFP4 variants, Nemotron 3.5 Lightning, specialized teacher checkpoints, and associated post-training collections.

  • This openness is particularly relevant to Nemotron because many of the family’s main contributions are difficult to reproduce from weights alone. LatentMoE depends on architecture and distributed expert-routing implementation; NVFP4 depends on tensor-specific scaling and numerical safeguards; MTP depends on both training and serving support; multi-environment RL requires executable environments; and MOPD depends on specialized teacher policies and on-policy rollout generation. Releasing only the final checkpoint would therefore omit much of the systems contribution.

  • The value of this release strategy is methodological as well as practical. Modern post-training performance is often determined by details that are absent from a final checkpoint: how demonstrations are generated and filtered, which prompts are admitted into RL, how verifiers are implemented, how rollout difficulty is scheduled, how heterogeneous environments are orchestrated, and how capabilities are recovered after domain-specific optimization. Making recipes, environments, data, and infrastructure visible turns Nemotron into a reproducible model-systems case study rather than only an inference artifact.

Training and post-training assets

  • The open release extends into both pre-training and post-training data. Ultra adds updated code, legal, specialized synthetic, and agentic datasets, while the post-training collection covers SFT and RL data intended to improve reasoning, tool use, instruction following, and general agent behavior. Its released code refresh alone contains 173B new code tokens with a September 30, 2025 cutoff.

  • The current NVIDIA Hugging Face organization organizes these assets into domain-oriented collections such as Nemotron Code & SWE, which includes competitive-programming, software-engineering, CUDA, and RL datasets, and the broader Nemotron post-training collection used across Nano, Super, and Ultra.

  • The practical implication is that Nemotron 3 can be used at several abstraction levels:

    \[\text{pre-training data + recipe} \;\longrightarrow\; \text{base checkpoint} \;\longrightarrow\; \text{SFT / RL environments} \;\longrightarrow\; \text{post-trained checkpoint} \;\longrightarrow\; \\ \text{quantization recipe} \;\longrightarrow\; \text{optimized deployment checkpoint} \;\longrightarrow\; \text{agent / application}\]

  • This makes the family useful not only as a set of models to serve but also as a reference implementation for constructing sparse, long-context, agent-oriented language models.

Multimodal extension

Extending the architecture beyond text

  • Nemotron 3 Nano Omni: Efficient and Open Multimodal Intelligence by NVIDIA et al. (2026) extends the 30B-A3B Nano language backbone to native text, image, video, and audio processing. The model follows an encoder-projector-decoder architecture: C-RADIOv4-H encodes visual inputs, Parakeet-TDT-0.6B-v2 encodes audio, modality-specific projectors map those representations into the language-model embedding space, and the shared Nemotron 3 Nano backbone performs multimodal reasoning.

  • The following figure (source) shows the Nemotron 3 Nano Omni architecture: images and video frames pass through the C-RADIOv4-H vision encoder and visual adaptor, audio passes through the Parakeet encoder and audio adaptor, and visual, audio, and text tokens are combined before processing by the Nemotron 3 Nano 30B-A3B language backbone.

  • The multimodal front end is designed explicitly around token reduction. Instead of converting every image into a fixed tiled representation, images use dynamic resolution with a variable number of visual patches while preserving native aspect ratio. A pixel-shuffle operation performs 4× token downsampling before projection into the LLM. For video, a Conv3D patch embedder combines every two frames, reducing temporal token count by 2×.

  • Audio undergoes even stronger temporal compression. Input is converted to 16 kHz mono and encoded using the Parakeet FastConformer. Log-mel features use a 10 ms hop, followed by three stride-2 convolutional subsampling stages \(2\times2\times2 = 8\), giving approximately an 8× temporal reduction. The resulting stream contains approximately 12.5 audio tokens per second, or about one token every 80 ms. Thirty seconds of audio therefore corresponds to roughly 375 audio tokens.

  • When audio and visual streams occur together, their tokens are interleaved according to temporal order rather than simply appending all audio after all video. This makes temporal correspondence between speech, sound events, and visual frames directly available to the language backbone.

Multimodal context and curriculum

  • Nano Omni uses a 256K context window, shorter than the 1M-token text models but twice the 128K context of its multimodal predecessor. The multimodal context must accommodate not only text but also potentially thousands of visual patches, long video sequences, and audio tokens, making token compression part of the effective context-length design.

  • Multimodal training is staged because introducing all modalities simultaneously risks destabilizing the pretrained text backbone. The training curriculum progressively aligns encoders, adds modalities, and increases context length, explicitly targeting preservation of text reasoning while improving multimodal behavior. The SFT pipeline contains seven stages, beginning with a vision-projector warmup in which the rest of the network remains frozen.

  • This is another example of the broader Nemotron design pattern: capability is introduced incrementally rather than forcing all optimization objectives into one homogeneous training phase.

Omni as a perception sub-agent

  • Nano Omni is particularly suited to a division of labor in which a smaller multimodal model handles perception while larger text models handle difficult planning or orchestration. Its input modalities allow an agent to inspect screenshots, documents, user interfaces, videos, speech, and environmental audio before converting those observations into a text-compatible reasoning state.

  • Conceptually:

\[\underbrace{\text{image / video / audio / GUI}}_{\text{multimodal inputs}} \;\longrightarrow\; \underbrace{\text{Nemotron 3 Nano Omni}}_{\text{perception}} \;\longrightarrow\; \underbrace{\text{structured observation / interpretation}}_{\text{reasoning state}} \;\longrightarrow\; \\ \underbrace{\text{planning model: Nano / Super / Ultra}}_{\text{planning and orchestration}} \;\longrightarrow\; \underbrace{\text{tools and actions}}_{\text{execution}}\]

  • The architecture therefore extends the notion of an efficient Nemotron sub-agent beyond text. Instead of requiring a frontier-scale multimodal model at every perception step, a sparse 30B-A3B model can perform high-volume sensory processing before escalating difficult reasoning tasks.

  • Nano Omni’s release follows the same open-stack approach as the text family, with BF16, FP8, and NVFP4 checkpoints plus a subset of the training data, generation pipelines, Megatron-Bridge training code, and NeMo RL guidance.

Deployment continuum

  • The 3.5 releases extend the open Nemotron stack into three operational tiers. Super 3.5 is the 120B-A12B multimodal and agentic model, Lightning 3.5 is the 30B-A3B text model, and Bolt 74B-A9B is a structurally compressed Super derivative designed around local-memory constraints. The NVIDIA Nemotron 3.5 Super and Lightning and NVIDIA Nemotron 3.5 Bolt 74B-A9B 24GB release pages expose checkpoints, drafters, recipes, datasets, and evaluation material for these complementary operating points.

  • Super and Lightning are released as base BF16, post-trained BF16, and NVFP4 variants, with LiLiCorr for Super and DFlash/DSpark for Lightning as standalone parallel drafters. Bolt adds the unquantized 74B-A9B BF16 checkpoint, the 24GB low-bit checkpoint in Hugging Face and GGUF forms, and LiLiCorr drafters in both ecosystems. This makes the serving artifact itself part of the model family: users can choose native MTP, standalone drafting, datacenter quantization, or local low-bit execution according to latency, concurrency, and memory constraints. Nemotron 3.5 Super and Lightning: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Models for Agentic AI by NVIDIA et al. (2026);

  • The training assets also broaden. Super 3.5 adds Nemotron-Pretraining-Indic-v1 with approximately 2.83T synthetic output tokens across 22 Indian languages and 34 language-script configurations, software-engineering mid-training mixtures, updated multimodal SFT data, specialized RL environments, and reproducibility recipes in NeMo Gym. Bolt demonstrates that those open parent assets can be reused downstream for compression calibration, recovery, QAD, and local-agent specialization rather than requiring a new pre-training run. Nemotron 3.5 Super and Lightning: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Models for Agentic AI by NVIDIA et al. (2026);

  • This continuum makes the model-system perspective more explicit. The same underlying hybrid Mamba-MoE lineage can be scaled up through sparse capacity, distilled downward into a smaller form factor, compressed structurally under a memory budget, quantized from four-bit to approximately two-bit experts, or paired with different speculative drafters. The deployment target therefore influences architecture search, distillation objective, precision recipe, recurrent-state handling, and post-training data rather than being addressed only after a model is finished. Nemotron 3.5 Super and Lightning: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Models for Agentic AI by NVIDIA et al. (2026);

Remaining limitations and open problems

MOPD depends on student-teacher support overlap

  • The clearest algorithmic limitation identified in Ultra concerns Multi-teacher On-Policy Distillation. MOPD operates on student-generated trajectories. Teacher supervision is therefore most informative when those trajectories lie in regions where the specialized teacher itself has meaningful probability mass.

  • When the student already knows roughly how to approach a task but makes suboptimal local decisions, MOPD can be highly effective. The teacher can provide token-level guidance on tool choice, abstention, environment interaction, or sequencing.

  • When the missing capability requires an entirely different reasoning trajectory, the limitation is more fundamental. The student may almost never generate the reasoning path on which the teacher’s advantage depends. Ultra observes this most clearly on HLE, where specialized-teacher gains are recovered much less effectively than on tool-use benchmarks.

  • This can be framed as a support-overlap problem. Let student and teacher trajectory distributions be \(p_S(\tau)\) and \(p_T(\tau)\).

  • On-policy teacher distillation works best when the important regions of the teacher distribution substantially intersect trajectories that the student can generate:

    \[\operatorname{supp}(p_S) \cap \operatorname{supp}(p_T)\]
    • must contain useful trajectories.

  • If the teacher’s strongest reasoning modes are concentrated in

    \[\operatorname{supp}(p_T) \setminus \operatorname{supp}(p_S)\]
    • student-only rollouts cannot expose them efficiently to the distillation objective.

Full-logit distillation is not automatically better

  • A natural extension of MOPD would be to match the teacher’s complete token distribution rather than only score sampled student actions. Ultra tests this idea but finds that top-\(k\) or full-vocabulary logit matching does not improve the current MOPD system and can underperform sampled-token supervision on agentic tasks such as Terminal Bench.

  • One plausible explanation is that teacher probabilities become poorly calibrated when evaluated on prefixes far outside the teacher’s usual trajectory distribution. Matching the entire distribution in such states can propagate more noise than simply evaluating the student action that was actually taken.

  • Thus, richer supervision is not necessarily equivalent to better supervision:

    \[\text{more teacher logits} \not\Rightarrow \text{more useful learning signal}\]

  • Determining when distribution-level teacher matching becomes preferable remains an open question.

Teacher and student initialization remains an open design choice

  • Ultra’s MOPD warmup partly solves the support problem by briefly exposing the student to examples related to a specialized teacher before distillation. However, alternative foundations were not systematically explored.

  • Potential designs include sharing a common SFT stage between student and teachers before specialization, or using specialized teachers to generate an intermediate SFT corpus for the student before starting RL and MOPD. These alternatives could increase support overlap before on-policy distillation begins, but they remain open research directions rather than established Nemotron 3 results.

Long-horizon agentic distillation remains expensive

  • End-to-end agent trajectories can involve dozens or hundreds of tool interactions, while ordinary reasoning tasks may require only a single model response. Mixing both directly into one MOPD batch creates severe rollout-time imbalance.

  • Ultra therefore uses single-turn or pivot-style rollouts for many agentic MOPD tasks rather than always replaying complete environment trajectories. Whether full end-to-end on-policy distillation can deliver additional capability while remaining computationally practical is explicitly left unresolved.

  • This is fundamentally a systems problem as much as an optimization problem. If trajectory durations are \(T_1,T_2,\ldots,T_B\) a synchronous batch can effectively be gated by \(T_{\mathrm{batch}} \approx \max_i T_i\).

  • The broader the trajectory-length distribution, the more compute can sit idle waiting for stragglers. Asynchronous execution reduces this cost but introduces policy staleness and more complex importance correction.

Mamba improves decode-heavy workloads more than prefill-heavy ones

  • The hybrid architecture also has an important performance boundary. Ultra does not dominate every serving regime.

  • For a prefill-heavy 50K-input/2K-output workload, computation is dominated by matrix multiplications over the active parameters. Ultra activates approximately 55B parameters versus 17B for the Qwen3.5 comparison model, producing about a 3.2× active-FLOP disadvantage. Ultra therefore trails Qwen3.5 in that prefill-heavy setting.

  • For decode-heavy 8K-input/64K-output inference, the dominant cost moves toward total weight I/O and token mixing. The total-parameter gap is smaller than the active-FLOP gap, while Mamba-2’s per-step sequence cost remains independent of generated history length. Ultra consequently leads in the decode-heavy regime.

  • The following relation captures the workload dependence:

    \[\text{prefill-heavy} \Rightarrow \text{active FLOPs matter more}\]

    • whereas:

      \[\text{decode-heavy} \Rightarrow \text{weight I/O + sequence-state cost matter more}\]

  • This distinction explains why Nemotron’s architectural advantages are strongest for long-generation reasoning and agent workloads rather than every possible LLM workload.

Multimodal scaling introduces a separate set of constraints

  • Nano Omni exposes different limitations from the text family. Training must align heterogeneous encoders, preserve the text backbone, balance highly unequal modality datasets, and manage rapid context consumption from images, video, and audio. The staged curriculum is explicitly designed to mitigate catastrophic forgetting and cross-modal instability, which indicates that simply attaching encoders to a strong text model is insufficient.

  • The multimodal model also uses a 256K context rather than the text models’ 1M context. Token compression partially compensates for this difference, but very long multimodal trajectories remain more constrained than text-only Ultra or Super workloads.

Key takeaways

  • Nemotron 3 can be understood as a sequence of design choices aimed at maximizing capability under the actual bottlenecks encountered in agentic inference.
Design problem Nemotron 3 mechanism Main effect
Dense parameter scaling is expensive Granular MoE / LatentMoE More total capacity than active compute
Attention decode cost grows with history Mamba-2-heavy backbone Fixed-size recurrent state for most layers
Pure recurrence can limit global interaction Sparse attention anchors Periodic full-context token mixing
Attention KV state remains expensive GQA + FP8 KV cache Smaller cache footprint
Distributed MoE communication is expensive LatentMoE Lower-dimensional expert routing
FP4 can damage sensitive operators Mixed precision Keep critical layers at FP8/BF16
Recurrent-state rounding accumulates Stochastic rounding Reduces systematic state drift
One-token decoding is serial MTP / speculative decoding Multiple tokens per target verification
Long reasoning can become unnecessarily expensive Reasoning-effort and budget control Explicit quality-compute operating point
SFT alone is weak for interactive behavior Multi-environment RLVR Verifiable improvement across task families
One mixed RL policy dilutes domain specialization Specialized teachers + MOPD Consolidates domain-specific agent skills
Multimodal tokens can overwhelm context Dynamic resolution and temporal compression Lower perception-token cost

  • The most important architectural lesson is that Nemotron 3 does not optimize parameter count, FLOPs, context length, or benchmark accuracy independently. A lower-FLOP architecture can still be slow if it moves too many weights; a sparse MoE can still be communication-bound; a million-token context can still be impractical if cache memory limits concurrency; an FP4 checkpoint can still be slower than a larger numerical datatype if its reduced footprint allows much larger batches; and a better reasoning model can still be uneconomical if it generates unnecessarily long traces.

  • The optimization target is therefore closer to:

    \[\max \frac{ \text{useful task capability} }{ \text{latency} \times \text{memory} \times \text{hardware cost} }\]
    • than to maximizing any isolated model metric.

Final perspective

  • Nemotron 3 represents a model architecture and a systems architecture at the same time. At the model level, it combines Mamba-2, sparse experts, selective attention, LatentMoE, and MTP. At the numerical level, it integrates NVFP4 pre-training, mixed-precision inference, low-precision cache storage, and stochastic rounding. At the post-training level, it combines SFT, multi-environment RL, specialized agent environments, generative reward models, reasoning controls, and Multi-teacher On-Policy Distillation. At the serving level, it relies on speculative decoding, context-aware cache design, expert parallelism, tensor parallelism, workload-specific batching, and hardware-specific kernels.

  • The progression from Nano to Super to Ultra is consequently not simply:

    \[\text{small} \rightarrow \text{medium} \rightarrow \text{large}\]

  • It is better characterized as:

    \[\text{efficient sub-agent} \rightarrow \text{general high-throughput agent} \rightarrow \text{frontier-scale orchestrator}\]

  • Nano emphasizes low active compute and high-volume deployment; Super adds substantially stronger coding, tool use, LatentMoE, MTP, and large-scale asynchronous agent training; Ultra scales these ideas while introducing specialized teachers and MOPD for difficult long-horizon workflows. Nano Omni extends the same efficiency philosophy into perception, giving the family a path from multimodal observation through high-level planning and tool execution.

  • This full-stack perspective is the central idea behind Nemotron 3: architecture, data, alignment, quantization, decoding, and serving are co-designed around the economics and systems behavior of long-running AI agents rather than optimized as independent stages.

References

Nemotron 3 family, technical reports, and releases

Hybrid sequence modeling and state-space foundations

Mixture-of-Experts architecture and routing

Multi-token prediction and speculative decoding

Pre-training schedules and optimization

Reinforcement learning, agentic post-training, and distillation

Low-precision training and quantization

Reinforcement-learning environments and training infrastructure

Inference and serving systems

Reasoning, coding, and scientific evaluation

Agentic tool-use, search, and computer-use evaluation

Long-context and multilingual evaluation

Multimodal and speech evaluation

NVIDIA open model and data ecosystem

Credits

Citation

If you found our work useful, please cite it as:

@article{Chadha2020DistilledNemotron3,
  title   = {Nemotron 3},
  author  = {Chadha, Aman and Jain, Vinija},
  journal = {Distilled AI},
  year    = {2020},
  note    = {\url{https://aman.ai}}
}