Overview

  • This chapter introduces the Qwen3 family and the design choices that connect its language and multimodal capabilities. It establishes the model families, training strategy, and inference-time controls before examining their implications.

Foundations

  • This section brings together what Qwen3 is, the central idea: reasoning and ordinary generation in one model, and thinking budgets and inference-time scaling. The following subsections explain how these elements fit into the broader chapter.

What Qwen3 is

  • Qwen3 is a family of open-weight large language models developed by the Qwen team at Alibaba Cloud. The original Qwen3 release spans dense and Mixture-of-Experts (MoE) architectures from 0.6B to 235B total parameters, with the flagship Qwen3-235B-A22B activating only 22B parameters for each token. The family is designed around three objectives: improving reasoning without requiring a separate reasoning model, increasing capability per activated parameter, and extending strong language-model performance across multilingual, coding, mathematical, and agentic tasks. Qwen3 Technical Report by Yang et al. (2025) introduces the architecture, 36T-token pre-training pipeline, four-stage post-training procedure, and evaluation of the original Qwen3 family.

  • The original release contains six dense models and two sparse MoE models:

Model Architecture Total parameters Activated parameters Original context
Qwen3-0.6B Dense 0.6B 0.6B 32K
Qwen3-1.7B Dense 1.7B 1.7B 32K
Qwen3-4B Dense 4B 4B 128K
Qwen3-8B Dense 8B 8B 128K
Qwen3-14B Dense 14B 14B 128K
Qwen3-32B Dense 32B 32B 128K
Qwen3-30B-A3B MoE 30B 3B 128K
Qwen3-235B-A22B MoE 235B 22B 128K

  • Qwen3 Technical Report by Yang et al. (2025) specifies these configurations and shows that both MoE models contain 128 experts while routing each token through 8 experts.

  • This model family subsequently became the language backbone for Qwen3-VL. Qwen3-VL Technical Report by Bai et al. (2025) extends Qwen3 into vision-language modeling with image, video, document, GUI, spatial, and multimodal reasoning capabilities, while expanding native interleaved context to 256K tokens.

The central idea: reasoning and ordinary generation in one model

  • One of Qwen3’s defining changes is its hybrid reasoning design. Earlier model families often separated an instruction-tuned conversational model from a reasoning-specialized model. Within Qwen’s own lineage, for example, Qwen2.5 served as the general-purpose model while QwQ specialized in extended reasoning. Qwen3 instead trains a single model to operate in two regimes. Qwen3 Technical Report by Yang et al. (2025) describes this as the unification of thinking and non-thinking behavior within the same model.

  • In thinking mode, the model can generate an internal extended reasoning trajectory before producing its final response. This mode is intended for problems such as mathematics, coding, scientific reasoning, and other tasks in which additional inference-time computation can improve the answer.

  • Conceptually,

    \[x \rightarrow r_1,r_2,\ldots,r_T \rightarrow y\]
    • where \(x\) is the input, \(r_{1:T}\) is the generated reasoning trajectory, and \(y\) is the final answer.

  • In non-thinking mode, Qwen3 generates the answer directly:

    \[x \rightarrow y\]

  • The practical distinction is therefore not merely two prompting styles. The post-training pipeline explicitly teaches both behaviors and ultimately combines reasoning and non-reasoning examples so that they coexist within the same model. Qwen3 Technical Report by Yang et al. (2025) describes a four-stage post-training pipeline consisting of long-chain-of-thought cold start, reasoning reinforcement learning, thinking-mode fusion, and general reinforcement learning.

  • The original Qwen3 release exposes this behavior through its chat template, allowing applications to choose whether reasoning is appropriate for a request. The Qwen3 repository provides the reference implementation and deployment guidance for controlling these modes.

Thinking budgets and inference-time scaling

  • Qwen3 goes beyond a binary choice between reasoning and direct generation by introducing a thinking budget. The budget bounds how much computation the model can spend producing reasoning tokens before returning an answer. Qwen3 Technical Report by Yang et al. (2025) reports that increasing this budget produces smooth performance improvements across evaluated mathematics, coding, and STEM tasks.

  • A useful abstraction is to regard the model as solving:

    \[y^\star(B) = \operatorname{Qwen3}(x; B)\]
    • where \(B\) is the maximum reasoning budget. Increasing \(B\) permits a longer latent computation: \(T \leq B\).

  • The resulting system exposes a direct quality-compute trade-off:

    \[\text{higher } B \quad\Rightarrow\quad \text{more inference compute} \quad\Rightarrow\quad \text{potentially higher task accuracy}\]

  • This makes Qwen3 an inference-time scalable model rather than one whose effective compute is fixed entirely during training. A simple request can use little or no explicit reasoning, while a difficult mathematical or coding problem can allocate substantially more tokens to deliberation.

  • The following figure (source) shows how Qwen3-235B-A22B performance changes as the thinking-token budget increases across reasoning-intensive benchmarks, illustrating the model’s inference-time scaling behavior.

  • Importantly, additional reasoning is not universally beneficial. On long-context retrieval evaluations, Qwen3’s thinking mode can perform slightly worse because extended reasoning may interfere with a task whose primary requirement is accurate retrieval rather than multi-step deduction. The technical report therefore presents thinking as a computational resource that should be allocated according to task structure rather than enabled indiscriminately.

Model family and training

  • This section connects dense models and sparse Mixture-of-Experts with a training pipeline built around capability density. The following subsections develop these related topics and their implications.

Dense models and sparse Mixture-of-Experts

  • The second major design axis is computational efficiency. Qwen3 provides conventional dense Transformers alongside sparse MoE Transformers.

  • For a dense model, essentially all feed-forward parameters in every layer participate in processing every token:

    \[h_{l+1} = h_l + \operatorname{FFN}(h_l)\]

  • An MoE layer instead contains a collection of expert feed-forward networks, \(E_1,E_2,\ldots,E_N\) and a router selects only a small subset for each token. Abstractly,

    \[\operatorname{MoE}(h) = \sum_{i \in \operatorname{TopK}(g(h))} p_i(h)E_i(h)\]
    • where \(g(h)\) produces routing scores, \(p_i(h)\) is the normalized routing weight, and only the top \(K\) experts execute for that token.

  • Qwen3’s two MoE models use \(N=128, K=8\).

  • Thus Qwen3-235B-A22B contains 235B parameters but activates approximately 22B parameters for a token, while Qwen3-30B-A3B contains 30B total parameters but activates approximately 3B. Qwen3 Technical Report by Yang et al. (2025) further introduces fine-grained expert segmentation, removes the shared experts used in Qwen2.5-MoE, and employs global-batch load balancing to promote useful expert specialization.

  • The important quantity for sparse models is consequently not just \(P_{\text{total}}\), but the relationship between total model capacity and active computation:

    \[P_{\text{active}} \ll P_{\text{total}}\]

  • This is central to Qwen3’s scaling strategy. Large total parameter capacity provides room for specialization, while sparse routing keeps per-token computation substantially below that of a similarly sized dense network.

A training pipeline built around capability density

  • Qwen3’s gains are not attributable to model architecture alone. The training pipeline substantially increases both the scale and composition of the data relative to Qwen2.5. The models are pre-trained on approximately 36 trillion tokens spanning 119 languages and dialects, compared with 29 languages in Qwen2.5. Qwen3 Technical Report by Yang et al. (2025) describes a corpus containing general web data alongside coding, STEM, reasoning, books, multilingual material, and large quantities of synthetic data.

  • The pipeline also illustrates an increasingly important pattern in frontier-model training: previous-generation models become components of the data engine for their successors. Qwen2.5-VL is used to extract text from PDF-like documents; Qwen2.5 subsequently cleans that extracted text; and Qwen2.5, Qwen2.5-Math, and Qwen2.5-Coder generate synthetic textbooks, question-answering examples, instructions, and code. The result is trillions of additional synthetic or model-processed tokens rather than a corpus consisting solely of directly collected text.

  • Pre-training itself is divided into three stages:

    \[\text{General pre-training} \rightarrow \text{Reasoning-intensive pre-training} \rightarrow \text{Long-context adaptation}\]

  • The first stage trains on more than 30T tokens at a 4,096-token sequence length. The second introduces roughly 5T higher-quality tokens with increased concentrations of STEM, code, reasoning, and synthetic material. The final stage uses hundreds of billions of long-context tokens at 32,768-token training sequences and modifies positional handling to enable longer inference contexts. Qwen3 Technical Report by Yang et al. (2025) describes this curriculum together with its RoPE-frequency adjustment, YaRN extrapolation, and Dual Chunk Attention strategy.

  • Post-training then develops the hybrid reasoning behavior. At a high level,

    \[\text{Base model} \rightarrow \text{long-CoT cold start} \rightarrow \text{reasoning RL} \rightarrow \text{thinking-mode fusion} \rightarrow \text{general RL}\]

  • A particularly important engineering result is that smaller Qwen3 models do not simply repeat the full expensive reasoning-RL pipeline. Instead, capabilities from stronger models are transferred downward through strong-to-weak distillation using both off-policy and on-policy mechanisms. The report finds this substantially more computationally efficient than directly applying reinforcement learning to the smaller models.

Extensions and design principles

  • This section connects Qwen3 as a foundation for multimodal models with the broader design philosophy. The following subsections develop these related topics and their implications.

Qwen3 as a foundation for multimodal models

  • Qwen3 also functions as the language-model foundation for Qwen3-VL. Rather than replacing the language architecture, Qwen3-VL surrounds a Qwen3 backbone with a vision encoder and vision-language merger:

    \[\text{visual input} \rightarrow \text{vision encoder} \rightarrow \text{vision-language merger} \rightarrow \text{Qwen3 LLM}\]

  • Qwen3-VL Technical Report by Bai et al. (2025) develops this architecture into dense 2B, 4B, 8B, and 32B models and MoE 30B-A3B and 235B-A22B models, extending the family to native 256K-token interleaved text-image-video contexts.

  • The multimodal extension introduces three especially important mechanisms: Interleaved MRoPE distributes temporal, horizontal, and vertical positional dimensions across frequency bands; DeepStack injects intermediate vision-transformer representations into several early LLM layers; and explicit textual timestamps replace the previous positional-encoding-based representation of video time.

  • The following figure (source) shows the Qwen3-VL architecture, including its native-resolution vision encoder, Qwen3 dense/MoE decoder, DeepStack multi-layer feature injection, Interleaved MRoPE, and timestamp-based representation of video.

  • This relationship is useful for understanding Qwen3 as more than one collection of text checkpoints. The same language-model foundation becomes part of a broader model stack capable of combining perception, reasoning, code, tool use, and action.

The broader design philosophy

  • Taken together, Qwen3’s design can be viewed along four independent scaling axes.

  • First, parameter scaling ranges from sub-billion dense models to a 235B sparse model. Second, sparse activation decouples total representational capacity from per-token computation. Third, thinking budgets allow computation to scale dynamically at inference time. Fourth, strong-to-weak distillation transfers expensive reasoning capabilities into smaller models without requiring each model size to rediscover those capabilities through reinforcement learning.

  • The resulting picture is approximately:

    \[\boxed{ \text{Qwen3 capability} = \\ f( \text{data}, \text{parameters}, \text{sparse activation}, \text{post-training}, \text{distillation}, \text{inference-time compute} ) }\]
    • rather than capability being determined primarily by parameter count.

  • This perspective also explains several of the family’s implementation choices. MoE routing targets lower active compute; thinking budgets target adaptive inference compute; staged pre-training targets higher-quality allocation of training compute; synthetic-data generation increases the leverage of existing models; and distillation amortizes the cost of developing reasoning behavior across an entire model family. Qwen3-VL then carries the same philosophy into multimodal modeling by preserving the Qwen3 language backbone while adding specialized visual representations and progressively extending context length.

  • The official Qwen3 repository provides model checkpoints, inference and deployment guidance, quantization support, evaluation resources, and integration with serving systems such as vLLM and SGLang. The original models are released under Apache 2.0, making the family usable for both research and downstream deployment.

Architecture and Model Design

  • This chapter describes how Qwen3 combines its Transformer backbone with attention, normalization, and sparse expert routing. The architectural components are considered alongside the efficiency trade-offs they introduce.

Transformer backbone

  • This section connects decoder-only Transformer backbone with model configurations. The following subsections develop these related topics and their implications.

Decoder-only Transformer backbone

  • Qwen3 retains the decoder-only Transformer structure established by earlier Qwen generations while making targeted changes to attention stability and sparse scaling. Qwen3 Technical Report by Yang et al. (2025) describes the dense architecture as closely following Qwen2.5, with Grouped Query Attention (GQA), SwiGLU feed-forward networks, Rotary Positional Embeddings (RoPE), and pre-normalization using RMSNorm. The principal architectural changes relative to earlier Qwen models are the removal of QKV biases and the addition of QK-Norm.

  • Given a sequence of token representations \(X = [x_1,x_2,\ldots,x_T] \in \mathbb{R}^{T\times d}\), each Transformer block can be abstracted as:

    \[H_l' = H_l + \operatorname{Attention} \left( \operatorname{RMSNorm}(H_l) \right)\]

    • followed by:

      \[H_{l+1} = H_l' + \operatorname{FFN} \left( \operatorname{RMSNorm}(H_l') \right)\]

  • This is a pre-normalized architecture because normalization occurs before the attention and feed-forward transformations rather than after the residual addition. Qwen2.5 Technical Report by Yang et al. (2024) documents the closely related predecessor architecture from which Qwen3 evolves.

  • The model remains autoregressive. For a token sequence \(x_1,x_2,\ldots,x_T\), the language-model probability factorizes as:

    \[p(x_1,\ldots,x_T) = \prod_{t=1}^{T} p(x_t\mid x_{<t})\]

    • and pre-training minimizes the standard next-token negative log-likelihood:

      \[\mathcal{L}_{\mathrm{LM}} = -\sum_{t=1}^{T} \log p_\theta(x_t\mid x_{<t})\]

  • The architectural innovations therefore do not alter the fundamental training objective. Instead, they change how representations are computed, normalized, routed, and stored during autoregressive processing.

Model configurations

  • Qwen3 scales this common backbone through different depths, attention configurations, embedding choices, and dense versus sparse feed-forward layers. The dense family ranges from 28-layer 0.6B and 1.7B models to a 64-layer 32B model. The two MoE variants increase depth to 48 layers for Qwen3-30B-A3B and 94 layers for Qwen3-235B-A22B. Qwen3 Technical Report by Yang et al. (2025) provides the complete architecture configurations.
Model Layers Query heads KV heads Experts total / active Tied embeddings Context
Qwen3-0.6B 28 16 8 Dense Yes 32K
Qwen3-1.7B 28 16 8 Dense Yes 32K
Qwen3-4B 36 32 8 Dense Yes 128K
Qwen3-8B 36 32 8 Dense No 128K
Qwen3-14B 40 40 8 Dense No 128K
Qwen3-32B 64 64 8 Dense No 128K
Qwen3-30B-A3B 48 32 4 128 / 8 No 128K
Qwen3-235B-A22B 94 64 4 128 / 8 No 128K

  • An important scaling trend is visible in the attention configuration. The number of query heads grows considerably with model size, while the number of key-value heads remains small. Qwen3-235B-A22B, for example, uses 64 query heads but only 4 KV heads. This asymmetry is enabled by GQA and directly reduces the size of the KV cache during autoregressive inference.

  • The smallest dense models also tie their input embedding and output language-model-head parameters. If \(E \in \mathbb{R}^{V\times d}\) is the input embedding matrix and:

    \[W_{\mathrm{LM}}\in\mathbb{R}^{V\times d}\]

    • is the output projection, weight tying imposes:

      \[W_{\mathrm{LM}} = E\]

  • Qwen3-0.6B, Qwen3-1.7B, and Qwen3-4B use this parameter-saving configuration, whereas the larger dense models do not.

Attention and positional encoding

  • This section brings together grouped Query Attention, qK-Norm and attention stability, and rotary Positional Embeddings. The following subsections explain how these elements fit into the broader chapter.

Grouped Query Attention

  • Standard multi-head attention constructs separate query, key, and value projections for every attention head. For head \(i\),

    \[Q_i=XW_i^Q,\qquad K_i=XW_i^K,\qquad V_i=XW_i^V\]

    • and attention is computed as:

      \[\operatorname{Attention}(Q_i,K_i,V_i) = \operatorname{softmax} \left( \frac{Q_iK_i^\top}{\sqrt{d_h}} + M \right)V_i\]
      • where \(M\) is the causal mask.

  • The drawback during autoregressive inference is that previously computed keys and values must be cached for every layer. With conventional multi-head attention, KV-cache storage grows approximately as:

    \[O(2LTn_hd_h)\]
    • where \(L\) is the number of layers, \(T\) is context length, \(n_h\) is the number of heads, and the factor of two represents keys and values.

  • Qwen3 instead uses Grouped Query Attention. GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints by Ainslie et al. (2023) introduces GQA as an intermediate design between multi-head attention and multi-query attention, sharing each key-value head among a group of query heads while retaining quality closer to multi-head attention.

  • For \(n_q\) query heads and \(n_{kv}\) key-value heads, each KV head is shared by approximately \(g=\frac{n_q}{n_{kv}}\) query heads.

  • For Qwen3-235B-A22B, \(n_q=64,\qquad n_{kv}=4\) so \(g=16\).

  • Each key-value head therefore services a group of 16 query heads. Relative to an otherwise equivalent 64-KV-head implementation, this dramatically reduces the KV state that must be retained for long-context generation.

  • The architecture is especially relevant for Qwen3 because most models support 128K contexts. KV-cache memory grows linearly with context length, making reductions in the number of KV heads increasingly valuable as \(T\) becomes large.

QK-Norm and attention stability

  • One of Qwen3’s notable departures from Qwen2.5 is QK-Norm. Before computing attention logits, normalization is applied to the query and key representations. The Qwen3 report explicitly identifies QK-Norm as a mechanism introduced to improve training stability while removing the QKV biases present in earlier Qwen architectures.

  • Conceptually, instead of directly evaluating \(A = \frac{QK^\top}{\sqrt{d_h}}\), the model computes normalized query and key states,

    \[\hat Q=\operatorname{Norm}(Q), \qquad \hat K=\operatorname{Norm}(K)\]

    • followed by:

      \[A = \frac{\hat Q\hat K^\top}{\sqrt{d_h}}\]

  • This constrains the scale of query-key interactions before the softmax. Since softmax attention depends exponentially on its logits,

    \[p_{ij} = \frac{\exp(A_{ij})} {\sum_k \exp(A_{ik})}\]
    • large variations in query or key magnitude can produce extremely peaked attention distributions. Normalizing these representations makes their scale better controlled and helps stabilize optimization at large model and data scales.

  • Qwen3 simultaneously removes the QKV biases used by Qwen2/Qwen2.5. Thus the projections take the simpler form

    \[Q=XW_Q,\qquad K=XW_K,\qquad V=XW_V\]

    • rather than:

      \[Q=XW_Q+b_Q, \qquad K=XW_K+b_K, \qquad V=XW_V+b_V\]

  • The combination of bias-free projections and explicit query-key normalization represents one of the clearest architectural differences between Qwen3 and its immediate predecessors.

Rotary Positional Embeddings

  • Qwen3 continues to use Rotary Positional Embeddings rather than learned absolute positional embeddings. RoPE encodes position by rotating pairs of dimensions in the query and key vectors according to token position.

  • For a two-dimensional component of a representation at position \(m\), the transformation can be expressed as:

    \[R_{\theta,m} = \begin{bmatrix} \cos(m\theta) & -\sin(m\theta)\\ \sin(m\theta) & \cos(m\theta) \end{bmatrix}\]

  • Queries and keys become:

    \[q_m'=R_{\theta,m}q_m, \qquad k_n'=R_{\theta,n}k_n\]

  • Their dot product then depends on relative displacement:

    \[(q_m')^\top k_n' = q_m^\top R_{\theta,n-m} k_n\]

  • This property makes RoPE particularly suitable for causal language models because positional relationships are incorporated directly into attention rather than added as an independent embedding.

  • Qwen3 also modifies RoPE during its long-context pre-training stage. The base frequency is increased from 10,000 to 1,000,000, while YaRN is subsequently used for further length extrapolation. These mechanisms become particularly important for supporting the 128K context lengths of the larger Qwen3 models and will be discussed with the long-context training pipeline in the pre-training section.

Feed-forward and normalization

  • This section connects rMSNorm and pre-normalization with swiGLU feed-forward networks. The following subsections develop these related topics and their implications.

RMSNorm and pre-normalization

  • Qwen3 uses RMSNorm rather than LayerNorm. For hidden representation \(x=(x_1,\ldots,x_d)\), RMSNorm computes \(\operatorname{RMS}(x) = \sqrt{\frac{1}{d}\sum_{i=1}^{d}x_i^2+\epsilon}\) followed by:

    \[\operatorname{RMSNorm}(x) = \gamma \odot \frac{x}{\operatorname{RMS}(x)}\]
    • where \(\gamma\) is a learned scaling vector.

  • Unlike LayerNorm, RMSNorm does not explicitly subtract the mean of the hidden state. Qwen3 applies it in a pre-normalization configuration, so a simplified block is:

    \[x' = x+ \operatorname{Attention}(\operatorname{RMSNorm}(x))\] \[y = x' + \operatorname{FFN}(\operatorname{RMSNorm}(x'))\]

  • Pre-normalization provides a direct residual path across deep Transformer stacks, an especially relevant property for Qwen3-235B-A22B’s 94-layer network.

SwiGLU feed-forward networks

  • The dense Qwen3 models use SwiGLU feed-forward networks. GLU Variants Improve Transformer by Shazeer (2020) studies gated feed-forward variants and shows that replacing conventional Transformer FFNs with gated variants such as SwiGLU can improve model quality.

  • A conventional Transformer FFN has approximately the form:

    \[\operatorname{FFN}(x) = W_2\,\sigma(W_1x)\]

  • SwiGLU instead introduces separate gate and value projections \(u=W_{\mathrm{up}}x\) and \(g=W_{\mathrm{gate}}x\), and applies the SiLU activation:

    \[\operatorname{SiLU}(g) = g\,\sigma(g)\]

  • The gated hidden representation is \(z = \operatorname{SiLU}(g)\odot u\) and the final FFN output is:

    \[\operatorname{SwiGLU}(x) = W_{\mathrm{down}}z\]

  • The multiplicative gate allows the network to control which components of the expanded hidden representation propagate through the feed-forward layer. In dense Qwen3 models, every token traverses this FFN. In MoE models, the corresponding feed-forward computation is instead distributed across sparsely selected expert networks.

Sparse expert design

  • This section brings together mixture-of-Experts architecture, fine-grained expert segmentation, and removal of shared experts. The following subsections explain how these elements fit into the broader chapter.

Mixture-of-Experts architecture

  • The Qwen3 MoE variants preserve the same attention backbone as the dense models but replace ordinary feed-forward computation with sparse expert computation. Qwen3 Technical Report by Yang et al. (2025) states that both Qwen3 MoE models use 128 experts and activate 8 experts per token.

  • Let \(E_1(x),E_2(x),\ldots,E_{128}(x)\) denote the expert FFNs. A router computes logits \(z=W_rx\) from which routing probabilities can be obtained:

    \[p_i = \frac{\exp(z_i)} {\sum_{j=1}^{128}\exp(z_j)}\]

  • The model selects the eight highest-scoring experts \(S(x) = \operatorname{TopK}(p,K=8)\) and computes a sparse mixture:

    \[y = \sum_{i\in S(x)} \tilde p_i E_i(x)\]
    • where \(\tilde p_i\) denotes the routing weights over the selected experts.

  • Consequently, only a fraction of the available expert parameters participates in the forward pass for each token. This is the architectural basis for model names such as Qwen3-30B-A3B and Qwen3-235B-A22B: the first quantity denotes total model parameters, while the \(A\) quantity denotes approximately the parameters activated per token.

Fine-grained expert segmentation

  • Qwen3 inherits fine-grained expert segmentation from the Qwen2.5-MoE lineage. Rather than constructing a small number of very large experts, the expert parameter space is partitioned into a larger number of smaller experts. The Qwen3 models expose 128 such experts.

  • The approach is related to the fine-grained expert strategy used by DeepSeekMoE. DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models by Dai et al. (2024) develops the idea of decomposing experts into finer-grained units so that different combinations can specialize more precisely for different token representations.

  • Conceptually, suppose a conventional MoE selects \(K\) large experts from \(N\) experts. Fine-grained segmentation decomposes the same expert capacity into more, smaller units \(N' > N\), while activating a correspondingly larger number of small experts. This increases the number of possible expert combinations available to the router.

  • For Qwen3, \(N'=128, K'=8\). The combinatorial number of possible eight-expert subsets is \({128 \choose 8}\), although actual routing patterns are learned and highly structured rather than uniformly sampled. The large routing space gives the model substantial flexibility to form different expert combinations for different token types and contexts.

Removal of shared experts

  • Qwen3 makes another important change relative to Qwen2.5-MoE: it removes shared experts. In a shared-expert MoE architecture, some experts execute for essentially every token while routed experts provide additional specialized computation.

  • Abstractly, such a layer can be written as:

    \[y = E_{\mathrm{shared}}(x) + \sum_{i\in S(x)} p_iE_i(x)\]

  • Qwen3 instead relies on routed experts without the Qwen2.5-MoE shared-expert component. Qwen3 Technical Report by Yang et al. (2025) explicitly identifies this as a difference from Qwen2.5-MoE.

  • The resulting architecture places greater responsibility on the routing mechanism and individual expert specialization. Rather than guaranteeing a common shared FFN path for every token, the model must learn effective combinations from its routed expert pool.

Global-batch load balancing

  • Sparse MoE models face a fundamental optimization problem: without appropriate regularization, the router may repeatedly select a small subset of experts. Such expert collapse wastes model capacity and can overload particular experts during distributed training.

  • Qwen3 addresses this with global-batch load balancing. The objective is to encourage routing traffic to remain sufficiently distributed across the expert population while still allowing specialization. Qwen3 Technical Report by Yang et al. (2025) specifically notes the use of a global-batch load-balancing loss for encouraging expert specialization.

  • If \(f_i\) represents the fraction of tokens assigned to expert \(i\) and \(P_i\) represents its average routing probability, a generic auxiliary balancing objective can be expressed schematically as:

    \[\mathcal{L}_{\mathrm{balance}} \propto N \sum_{i=1}^{N} f_iP_i\]

  • The defining Qwen3 detail is that the balancing statistics are computed across the global training batch rather than independently over small local partitions. This gives the optimization objective a broader view of expert utilization and permits specialization patterns that may look locally imbalanced while remaining balanced at the overall batch level.

  • The full optimization objective can therefore be viewed schematically as:

    \[\mathcal{L} = \mathcal{L}_{\mathrm{LM}} + \lambda_{\mathrm{balance}} \mathcal{L}_{\mathrm{balance}}\]
    • where the report establishes the use of the global-batch balancing objective but does not provide all implementation constants required to reconstruct its exact training configuration.

Tokenization and efficiency

  • This section connects tokenization with architectural efficiency as a system-level objective. The following subsections develop these related topics and their implications.

Tokenization

  • All Qwen3 text models use Qwen’s byte-level byte-pair encoding tokenizer with a vocabulary of 151,669 tokens. Qwen3 Technical Report by Yang et al. (2025) specifies the tokenizer and vocabulary size.

  • Byte-level BPE begins from byte representations and repeatedly merges frequently co-occurring units. If \(b_1,b_2,\ldots,b_m\) are the initial byte-level symbols, learned merge operations construct increasingly larger subword units:

    \[(b_i,b_{i+1}) \rightarrow u_j\]

  • The byte-level foundation is particularly useful for Qwen3’s multilingual scope because arbitrary Unicode text can ultimately be represented through bytes rather than requiring a closed vocabulary containing every possible character. The same tokenizer therefore supports the family’s 119 languages and dialects, programming languages, mathematical notation, and mixed-language inputs.

Architectural efficiency as a system-level objective

  • Qwen3’s architecture is best understood as a collection of mutually reinforcing efficiency choices rather than a radical replacement of the Transformer.

  • GQA reduces KV-cache storage and memory bandwidth requirements. QK-Norm improves attention stability as models and training scale increase. RMSNorm and pre-normalization support optimization through deep residual stacks. SwiGLU provides gated feed-forward computation. Fine-grained MoE routing increases total parameter capacity without activating the entire network for every token. Global-batch balancing makes that sparse capacity trainable, while removing shared experts simplifies the sparse execution path.

  • The resulting computation can be summarized schematically as

    \[x \rightarrow \operatorname{RMSNorm} \rightarrow \operatorname{QKNorm\text{-}GQA} \rightarrow \text{residual} \rightarrow \operatorname{RMSNorm} \rightarrow \begin{cases} \operatorname{SwiGLU\ FFN}, & \text{dense}\\ \operatorname{TopK\ MoE}, & \text{sparse} \end{cases} \rightarrow \text{residual}\]

  • This architecture provides the foundation on which Qwen3’s training innovations operate. The more substantial changes in reasoning behavior do not come from replacing the Transformer itself. They emerge from the scale and composition of pre-training data, staged long-context training, the four-stage reasoning post-training pipeline, reinforcement learning, and strong-to-weak distillation.

Pre-training Data and Training Strategy

  • This chapter follows the construction of Qwen3’s training corpus and its progressive pre-training schedule. Data preparation, long-context adaptation, and training efficiency are treated as connected parts of the same process.

Corpus construction

  • This section brings together scaling the pre-training corpus, model-assisted document extraction, and synthetic data generation. The following subsections explain how these elements fit into the broader chapter.

Scaling the pre-training corpus

  • Qwen3 substantially expands the scale and diversity of pre-training relative to Qwen2.5. The full family is trained on approximately 36 trillion tokens covering 119 languages and dialects. The corpus includes general text alongside coding, STEM, reasoning, books, multilingual material, and synthetic data. Qwen3 Technical Report by Qwen Team (2025) describes this as roughly twice the token volume and about three times the language coverage of Qwen2.5.

  • The aggregate corpus can be viewed schematically as:

    \[\mathcal{D} = \mathcal{D}_{\mathrm{general}} \cup \mathcal{D}_{\mathrm{code}} \cup \mathcal{D}_{\mathrm{STEM}} \cup \mathcal{D}_{\mathrm{reasoning}} \cup \mathcal{D}_{\mathrm{books}} \cup \mathcal{D}_{\mathrm{multilingual}} \cup \mathcal{D}_{\mathrm{synthetic}}\]

  • The important change is not simply the increase in \(\|\mathcal{D}\|\).

  • Qwen3 also places substantially more emphasis on data transformation, annotation, synthetic generation, instance-level mixture optimization, and curriculum construction. The pre-training pipeline therefore treats the corpus as an engineered training distribution rather than a static collection of documents.

Model-assisted document extraction

  • A notable part of the Qwen3 data pipeline converts visually structured documents into language-model training data. Rather than relying only on conventional text extraction, Qwen2.5-VL is used to recognize text from a large volume of PDF-like documents. The resulting text is subsequently refined by Qwen2.5. According to the Qwen3 Technical Report by Qwen Team (2025), this process contributes trillions of additional high-quality tokens.

  • The pipeline can be represented as

    \[\text{PDF-like document} \rightarrow \text{Qwen2.5-VL recognition} \rightarrow \text{Qwen2.5 refinement} \rightarrow \text{training text}\]

  • Qwen2.5-VL Technical Report by Bai et al. (2025) describes the predecessor vision-language model’s document parsing, structured-data extraction, visual recognition, and layout-understanding capabilities, which make it suitable for this role in the Qwen3 data pipeline.

Synthetic data generation

  • Qwen3 also relies extensively on synthetic data. Several specialized models from the Qwen2.5 generation are used to generate training examples:

    1. Qwen2.5 \(\rightarrow\) general synthetic data

    2. Qwen2.5-Math \(\rightarrow\) mathematical and reasoning data

    3. Qwen2.5-Coder \(\rightarrow\) code-oriented data

  • The generated material spans textbooks, question-answer pairs, instructions, and code snippets across dozens of domains, collectively contributing trillions of tokens. Qwen3 Technical Report by Qwen Team (2025) identifies these synthetic sources as a major component of the expanded corpus.

  • This strategy allows the training distribution to be deliberately enriched in areas where high-quality naturally occurring data may be scarce. Instead of accepting the natural frequency of difficult mathematical derivations, instructional explanations, programming tasks, or structured question-answer pairs on the web, synthetic generation can increase their representation.

  • Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement by Yang et al. (2024) describes how the Qwen Math lineage uses model-generated mathematical data as part of a self-improvement pipeline, providing the specialized mathematical generator subsequently used for Qwen3 data construction.

  • Similarly, Qwen2.5-Coder Technical Report by Hui et al. (2024) describes a code-specialized Qwen2.5 model trained with extensive code data, synthetic generation, and data mixing, which Qwen3 subsequently uses as one of its synthetic-data generators.

  • Qwen3 therefore illustrates a recursive data-development process:

    \[M_{t-1} \rightarrow \mathcal{D}_{\mathrm{synthetic}} \rightarrow M_t\]
    • where a previous-generation model \(M_{t-1}\) helps construct the training corpus for its successor \(M_t\).

  • This does not mean that all Qwen3 data are synthetic. Rather, synthetic generation supplements a much broader corpus and allows the training mixture to target particular capability domains.

Multilingual expansion

  • The language coverage increases from 29 languages in Qwen2.5 to 119 languages and dialects in Qwen3. This multilingual expansion is incorporated during general pre-training rather than being treated solely as a later fine-tuning step. Qwen3 Technical Report by Qwen Team (2025) states that the first pre-training stage already covers the complete 119-language distribution.

  • From a training-distribution perspective, multilingual modeling introduces a mixture

    \[p_{\mathrm{train}}(x) = \sum_{\ell=1}^{L} w_\ell p(x\mid \ell)\]
    • where \(L=119\) represents supported languages and dialects and \(w_\ell\) is the effective sampling weight assigned to language \(\ell\).

  • The choice of these weights matters. Simply concatenating multilingual corpora can cause high-resource languages to dominate the token distribution. Qwen3’s broader annotation and mixture-optimization pipeline therefore becomes particularly relevant because language is only one dimension along which examples differ.

  • The resulting multilingual capability is intended to coexist with coding, mathematical reasoning, general knowledge, and instruction following rather than exist as an isolated multilingual specialization.

Data quality and sampling

  • This section connects fine-grained data annotation with instance-level data mixture optimization. The following subsections develop these related topics and their implications.

Fine-grained data annotation

  • Before training, Qwen3 applies a multilingual annotation system to more than 30 trillion tokens. The annotations span several dimensions, including educational value, field, domain, and safety. Qwen3 Technical Report by Qwen Team (2025) states that these labels are subsequently used for filtering and data-mixture construction.

  • Conceptually, each training instance \(x_i\) can be associated with metadata:

    \[a_i = ( q_i, f_i, d_i, s_i,\ldots )\]
    • where, for example, \(q_i\) describes educational quality, \(f_i\) its field, \(d_i\) its domain, and \(s_i\) its safety-related annotation.

  • Training-data construction can then operate over \((x_i,a_i)\) rather than over raw documents alone.

  • This distinction matters because two examples originating from the same source can have very different training value. One may contain a rigorous technical explanation, while another may be repetitive, low-information, or otherwise undesirable. Source-level weighting cannot express this distinction as precisely as instance-level labels.

Instance-level data mixture optimization

  • Qwen3 uses the fine-grained annotations to optimize its data mixture at the instance level. The report contrasts this with approaches that primarily optimize mixture proportions at the source or domain level. Small proxy models are trained in extensive ablation experiments to determine how different fine-grained data compositions affect downstream capability. Qwen3 Technical Report by Qwen Team (2025) identifies this proxy-model experimentation as the basis for instance-level mixture optimization.

  • A conventional domain-level mixture might be expressed as

    \[p(x) = \sum_{d=1}^{D} \alpha_d p_d(x)\]
    • where \(\alpha_d\) controls the proportion assigned to an entire domain such as code, books, mathematics, or web text.

  • Qwen3’s finer-grained strategy can instead be conceptualized as assigning value at the example level \(w_i = f(a_i)\), followed by a sampling distribution such as:

    \[p(x_i) = \frac{w_i} {\sum_j w_j}\]

  • The technical report does not publish the exact scoring function \(f\) or the final instance-level sampling algorithm, so those implementation details should not be inferred. What it does establish is the methodology: fine-grained labels are combined with controlled experiments on smaller proxy models to determine more effective training mixtures.

  • This makes data selection itself an empirical scaling problem. Rather than training every full-scale model to test a corpus hypothesis, inexpensive proxy experiments can estimate which mixture changes are worth carrying into the much more costly flagship run.

Progressive pre-training

  • This section brings together three-stage pre-training curriculum, general Stage, and reasoning Stage. The following subsections explain how these elements fit into the broader chapter.

Three-stage pre-training curriculum

  • Qwen3 does not train on a fixed data distribution and context length for the entire 36T-token run. Pre-training is divided into three stages:

    \[\boxed{ \text{General Stage} \rightarrow \text{Reasoning Stage} \rightarrow \text{Long-Context Stage} }\]

  • Each stage changes the effective training distribution or sequence length. Qwen3 Technical Report by Qwen Team (2025) specifies this three-stage curriculum for the complete model family.

Stage Approximate tokens Sequence length Primary emphasis
General Stage More than 30T 4,096 Language, world knowledge, multilingual breadth
Reasoning Stage About 5T 4,096 STEM, coding, reasoning, synthetic high-quality data
Long-Context Stage Hundreds of billions 32,768 Long-document and long-sequence capability
  • The curriculum changes what the model learns as training progresses. The majority of compute establishes broad linguistic and world knowledge. A later high-quality phase shifts the distribution toward reasoning-intensive capabilities, while the final stage adapts the already capable model to substantially longer sequences.

General Stage

  • The General Stage, or S1, accounts for more than 30 trillion tokens. Models train with a sequence length of \(L_{\mathrm{S1}}=4096\).

  • This stage establishes broad language proficiency and general world knowledge while exposing the models to the complete 119-language and dialect distribution. Qwen3 Technical Report by Qwen Team (2025).

  • From a compute perspective, using a moderate context length for the largest portion of training is consequential because standard self-attention has quadratic sequence-length complexity:

    \[C_{\mathrm{attention}} = O(L^2d)\]

  • Moving the entire 30T-token corpus from 4K to 32K sequences would therefore impose a dramatically larger attention-compute and memory burden. Qwen3 instead reserves expensive long-sequence training for the much smaller final stage.

  • The first stage can consequently be viewed as broad capability acquisition under a relatively economical sequence-length regime.

Reasoning Stage

  • The second stage, S2, shifts the training distribution toward higher-quality reasoning-intensive material. Approximately 5 trillion additional tokens are used, still with \(L_{\mathrm{S2}}=4096\).

  • The proportions of STEM, coding, reasoning, and synthetic data are increased, and the learning-rate decay is accelerated. Qwen3 Technical Report by Qwen Team (2025) describes this stage as specifically intended to strengthen reasoning before post-training begins.

  • If the first-stage distribution is \(p_{\mathrm{S1}}(x)\), the second stage can be viewed as changing the mixture to

    \[p_{\mathrm{S2}}(x) \neq p_{\mathrm{S1}}(x)\]
    • with greater probability mass assigned to examples requiring mathematical, scientific, algorithmic, or structured reasoning.

  • This is an important distinction from treating reasoning as exclusively a reinforcement-learning phenomenon. Qwen3 begins shaping reasoning capabilities during pre-training by changing the composition of next-token-prediction data. Post-training subsequently develops the model’s explicit long-chain reasoning behavior.

  • The accelerated learning-rate decay also means that S2 is not simply an additional block of identically configured pre-training. As the model approaches the end of this stage, parameter updates become progressively smaller while the data distribution becomes more capability-focused.

Long-Context Stage

  • The final pre-training stage extends the model’s sequence-processing capability. Models train on hundreds of billions of long-context tokens with a sequence length of:

    \[L_{\mathrm{train}}=32{,}768\]

  • The corpus is intentionally dominated by genuinely long examples. Approximately 75% of the long-context data consist of sequences between:

    \[16{,}384 \leq L \leq 32{,}768\]

    • while the remaining 25% lie between:

      \[4{,}096 \leq L < 16{,}384\]

  • These proportions are reported in the Qwen3 Technical Report by Qwen Team (2025).

  • This distribution matters because merely increasing the maximum sequence length does not guarantee that a model learns to use distant information. A training corpus dominated by short sequences would provide relatively few optimization signals involving long-range dependencies.

  • Qwen3 therefore changes both the allowable context length and the actual length distribution of the training examples.

Long-context adaptation

  • This section brings together roPE base-frequency adaptation, yaRN for context extrapolation, and dual Chunk Attention. The following subsections explain how these elements fit into the broader chapter.

RoPE base-frequency adaptation

  • Long-context adaptation also modifies Qwen3’s positional encoding. During the long-context stage, the RoPE base frequency is increased from \(\theta_{\mathrm{base}}=10{,}000\) to \(\theta_{\mathrm{base}}=1{,}000{,}000\).

  • Qwen3 Technical Report by Qwen Team (2025) states that this change uses the ABF technique and follows the long-context strategy used in Qwen2.5.

  • For RoPE, frequency components can be represented schematically as:

    \[\theta_i = \theta_{\mathrm{base}}^{-2i/d}\]

  • Changing the base modifies the spectrum of positional rotations across dimensions. A larger base causes some dimensions to rotate more slowly as token positions increase, making the representation better suited to longer positional ranges.

  • This modification complements rather than replaces long-context training. Qwen3 both exposes the model to long sequences and adapts the positional mechanism used to represent those sequences.

YaRN for context extrapolation

  • Training reaches 32K-token sequences, but the larger Qwen3 models expose context windows up to 128K. The gap is bridged in part through positional extrapolation.

  • YaRN: Efficient Context Window Extension of Large Language Models by Peng et al. (2023) introduces a RoPE-based context-extension method designed to extrapolate models beyond their original training length with substantially less additional training than straightforward long-context adaptation.

  • Qwen3 uses YaRN together with its long-context training strategy to support longer inference sequences. For long-context evaluation, the report uses a YaRN scaling factor of \(s=4\) corresponding naturally to the expansion:

    \[32\mathrm{K}\times4 = 128\mathrm{K}\]

  • The Qwen3 Technical Report by Qwen Team (2025) reports this configuration for its RULER long-context evaluation.

  • The distinction between training length and inference length is therefore important:

    \[L_{\mathrm{train}}=32\mathrm{K}\]

    • while positional extrapolation allows:

      \[L_{\mathrm{inference}}\approx128\mathrm{K}\]

Dual Chunk Attention

  • Qwen3 additionally incorporates Dual Chunk Attention as part of its context-extension strategy. Training-Free Long-Context Scaling of Large Language Models by An et al. (2024) introduces Dual Chunk Attention (DCA), which decomposes long-sequence attention into mechanisms for positional relationships within a chunk and across different chunks, enabling substantial context extension without requiring full retraining at the target length.

  • The central idea is to divide a long sequence into chunks,

    \[X = [C_1,C_2,\ldots,C_m]\]
    • and distinguish attention interactions according to whether tokens occur within the same chunk or across different chunks.

  • Rather than extrapolating every relative position exactly as ordinary RoPE would at very large distances, DCA constructs attention using separate positional treatment for intra-chunk and inter-chunk relationships.

  • Qwen3 combines DCA with YaRN and its altered RoPE base to achieve a fourfold increase in sequence-length capacity during inference. Qwen3 Technical Report by Qwen Team (2025).

  • The overall long-context path can therefore be summarized as

    \[\boxed{ 32\mathrm{K\ long\text{-}context\ training} + \text{RoPE base adaptation} + \text{YaRN} + \text{DCA} \rightarrow 128\mathrm{K\ inference} }\]
    • for the models configured with 128K context.

Training efficiency

  • This section connects scaling laws for training hyperparameters with what pre-training contributes to Qwen3’s efficiency. The following subsections develop these related topics and their implications.

Scaling laws for training hyperparameters

  • Qwen3 also uses scaling experiments to determine training hyperparameters rather than applying a single configuration across every model size and architecture. The team develops scaling laws for quantities including the learning-rate schedule and batch size, studying their relationships with model architecture, data, and training stage. Separate predicted-optimal strategies are then chosen for dense and MoE models. Qwen3 Technical Report by Qwen Team (2025).

  • Conceptually, instead of treating a hyperparameter such as learning rate \(\eta\) as fixed, the goal is to estimate a relationship such as:

    \[\eta^\star = f(P,D,S,A)\]
    • where \(P\) denotes model scale, \(D\) denotes training-data characteristics, \(S\) denotes training stage, and \(A\) denotes architectural properties such as dense versus MoE computation.

  • Likewise, the preferred batch size can be viewed as:

    \[B^\star = g(P,D,S,A)\]

  • The report does not publish the fitted scaling equations or the complete hyperparameter schedules, so an exact reconstruction of these functions is not supported. The important implementation principle is that Qwen3 uses smaller-scale experiments to predict appropriate optimization settings for each model and each stage rather than assuming that hyperparameters transfer unchanged across scale.

What pre-training contributes to Qwen3’s efficiency

  • The resulting pre-training system combines several distinct forms of scaling:

    1. Data-scale expansion

    2. Quality and mixture optimization

    3. Synthetic-data scaling

    4. Multilingual scaling

    5. Curriculum specialization

    6. Context-length scaling

  • The base-model evaluations reported in the Qwen3 Technical Report by Qwen Team (2025) indicate that, under the same pre-training data, Qwen3 MoE models can reach performance comparable to Qwen3 dense models while using approximately one-fifth as many activated parameters. The report also finds that dense Qwen3 models frequently match larger Qwen2.5 base models, particularly improving on STEM, coding, and reasoning tasks. These results are presented as evidence that the gains come from the combination of architecture, expanded and better-targeted data, and the staged training strategy rather than parameter count alone.

  • The full pre-training pipeline can consequently be summarized as:

    \[\boxed{ \begin{gathered} \text{raw multilingual + domain data}\\ +\,\text{vision-assisted document extraction}\\ +\,\text{specialist-model synthetic generation}\\ \downarrow\\ \text{fine-grained annotation}\\ \downarrow\\ \text{instance-level mixture optimization}\\ \downarrow\\ \text{30T+ general tokens at 4K}\\ \downarrow\\ \text{about 5T reasoning-focused tokens at 4K}\\ \downarrow\\ \text{hundreds of billions of long-context tokens at 32K}\\ \downarrow\\ \text{RoPE adaptation + YaRN + DCA}\\ \downarrow\\ \text{Qwen3 base model} \end{gathered} }\]

  • The key point is that Qwen3’s reasoning capability begins well before reasoning-specific post-training. Its base model has already been deliberately exposed to a higher concentration of STEM, code, reasoning, synthetic instructional material, multilingual data, and long-context examples. The subsequent post-training pipeline can therefore refine and elicit capabilities already established during large-scale next-token prediction rather than attempting to create them entirely from reinforcement learning.

Post-training, Reasoning RL, and Strong-to-Weak Distillation

  • This chapter traces how Qwen3 develops reasoning and general assistant behavior after pre-training. It then examines how those capabilities are transferred to smaller models through distillation.

Four-stage post-training

  • This section develops post-training overview in the context of the chapter. The discussion below connects the relevant implementation details with their implications.

Post-training overview

  • Qwen3’s post-training pipeline is designed around a central requirement: the same model should support both deliberate, long-form reasoning and efficient direct responses. Rather than independently training a reasoning model and an instruction model, Qwen3 first develops reasoning capability, then integrates non-thinking behavior, and finally improves general-purpose behavior through reinforcement learning. Qwen3 Technical Report by Yang et al. (2025) describes four sequential stages for the flagship models: Long-CoT Cold Start, Reasoning RL, Thinking Mode Fusion, and General RL.

  • The following figure (source) shows the complete Qwen3 post-training pipeline, including the four-stage training path for flagship models and the separate strong-to-weak distillation path used to transfer their capabilities into smaller models.

  • The flagship training trajectory can be summarized as:

    \[\text{Base Model} \rightarrow \text{Long-CoT Cold Start} \rightarrow \text{Reasoning RL} \rightarrow \text{Thinking Mode Fusion} \rightarrow \text{General RL}\]

  • The smaller models instead use:

    \[\text{Teacher Models} \rightarrow \text{Off-policy Distillation} \rightarrow \text{On-policy Distillation} \rightarrow \text{Lightweight Qwen3}\]

  • This division is important. Qwen3 does not independently reproduce the expensive reasoning-development process for every model size. Qwen3-32B and Qwen3-235B-A22B serve as strong models from which capabilities can subsequently be transferred to Qwen3-0.6B, 1.7B, 4B, 8B, 14B, and 30B-A3B. Qwen3 Technical Report by Yang et al. (2025) reports that this strong-to-weak strategy is more effective and computationally efficient than directly applying reinforcement learning to the smaller models.

Long-CoT cold start

  • This section brings together long-CoT Cold Start, query filtering for cold-start data, and response generation and rejection sampling. The following subsections explain how these elements fit into the broader chapter.

Long-CoT Cold Start

  • The first post-training stage establishes explicit long-form reasoning patterns before reinforcement learning begins. This is a supervised fine-tuning stage rather than an RL stage.

  • The objective is not simply to maximize immediate benchmark performance. The training set is deliberately kept selective, and the number of training steps is limited so that the model learns useful reasoning patterns without being overly constrained by a narrow supervised distribution. Qwen3 Technical Report by Yang et al. (2025) explicitly describes this stage as establishing foundational reasoning behavior while preserving room for subsequent RL improvement.

  • For a reasoning example consisting of prompt \(x\) and target reasoning-plus-answer sequence \(y=(r,a)\), the cold-start objective is conventional supervised next-token prediction:

    \[\mathcal{L}_{\mathrm{SFT}} = -\sum_{t=1}^{|y|} \log \pi_\theta \left( y_t \mid x,y_{<t} \right)\]

  • The distinction lies primarily in how the training examples are constructed.

Query filtering for cold-start data

  • Qwen3 applies a multi-stage filtering pipeline before accepting reasoning prompts into the cold-start set. Qwen2.5-72B-Instruct is used to identify and remove prompts that are difficult to verify objectively, including prompts containing multiple independent subquestions and open-ended general-generation requests.

  • Queries that Qwen2.5-72B-Instruct can already solve correctly without chain-of-thought reasoning are also removed. The intention is to avoid filling the reasoning dataset with questions for which a model can reach the correct answer through superficial pattern matching or direct recall.

  • The desired subset can be conceptualized as:

    \[\mathcal{D}_{\mathrm{cold}} = \left\{ x\in\mathcal{D} : V(x)=1,\; C(x)=1,\; D(x)=1 \right\}\]
    • where \(V(x)\) denotes that the problem is readily verifiable, \(C(x)\) denotes that it genuinely requires sufficient complexity or reasoning, and \(D(x)\) represents acceptance under the desired domain distribution.

  • Qwen2.5-72B-Instruct additionally assigns domain labels to candidate queries so that the final cold-start corpus can maintain broader domain coverage rather than becoming dominated by a small number of problem types.

Response generation and rejection sampling

  • After query filtering and reservation of a validation set, QwQ-32B generates \(N\) candidate responses for each remaining prompt. QwQ-32B: Embracing the Power of Reinforcement Learning by Qwen Team (2025) describes QwQ as Qwen’s preceding reasoning-focused model, developed through reinforcement learning to strengthen mathematical and coding reasoning.

  • For prompt \(x\), the teacher samples:

    \[y_1,\ldots,y_N \sim \pi_{\mathrm{QwQ}}(\cdot\mid x)\]

  • A useful quantity is:

    \[\operatorname{Pass@N}(x) = \mathbb{1} \left[ \exists i: V(y_i)=1 \right]\]
    • where \(V(y_i)\) denotes whether candidate \(i\) is correct according to the relevant verifier.

  • If QwQ-32B consistently fails on a query, human annotators are brought in to assess response correctness. For prompts with positive Pass@N, additional response filtering removes outputs with incorrect final answers, excessive repetition, unsupported guessing, inconsistencies between reasoning and summary, inappropriate language mixing or style shifts, and suspected overlap with validation material.

  • The resulting cold-start set therefore contains verified reasoning trajectories rather than simply arbitrary teacher-generated chains of thought.

Why a cold start precedes RL

  • Starting reinforcement learning from a raw base model requires the policy to simultaneously discover useful reasoning formats and optimize task success. Qwen3 instead gives the policy an initial reasoning prior:

    \[\pi_{\mathrm{base}} \xrightarrow{\mathrm{SFT}} \pi_{\mathrm{cold}}\]

  • Reinforcement learning then starts from \(\pi_{\mathrm{cold}}\) which already knows how to construct coherent extended solutions.

  • This follows a broader pattern in reasoning-model development in which supervised or curated reasoning traces bootstrap the policy before large-scale outcome-based RL. Qwen’s earlier QwQ-32B: Embracing the Power of Reinforcement Learning describes reinforcement learning as a mechanism for extracting more reasoning capability from a strong pretrained foundation model.

  • The Qwen3 cold-start phase is deliberately restrained, however. The technical report emphasizes minimizing both the number of training examples and the number of SFT steps, because excessive imitation could limit exploration during the next stage.

Reasoning reinforcement learning

  • This section brings together reasoning RL, group Relative Policy Optimization, and large batches, multiple rollouts, and off-policy training. The following subsections explain how these elements fit into the broader chapter.

Reasoning RL

  • The second stage applies reinforcement learning specifically to reasoning. The training set contains 3,995 query-verifier pairs chosen according to four criteria: they must not have appeared in cold-start training, they must be learnable by the cold-start model, they should be as challenging as possible, and they should cover a broad set of subdomains.

  • This creates a deliberately small but high-signal RL environment:

    \[\mathcal{D}_{\mathrm{RL}} = \{(x_i,V_i)\}_{i=1}^{3995}\]
    • where \(V_i\) is a verifier capable of determining whether a generated solution to \(x_i\) is successful.

  • The emphasis on verifiable problems is significant because reasoning RL benefits from rewards that correspond closely to objective task success. Mathematics and coding are particularly suitable because final answers, tests, or other deterministic criteria can often provide reliable feedback.

Group Relative Policy Optimization

  • Qwen3 uses Group Relative Policy Optimization, or GRPO, during reasoning RL. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models by Shao et al. (2024) introduces GRPO as a PPO-style policy-optimization algorithm that estimates relative advantages from groups of sampled responses while eliminating the need for a separate learned critic.

  • For each prompt \(x\), the policy samples a group of outputs \(\{y_1,y_2,\ldots,y_G\}\).

  • Each receives a reward:

    \[r_i=R(x,y_i)\]

  • A simplified group-relative advantage can be written as:

    \[A_i = \frac{ r_i-\operatorname{mean}(r_1,\ldots,r_G) }{ \operatorname{std}(r_1,\ldots,r_G)+\epsilon }\]

  • The policy is then updated to increase the probability of relatively successful trajectories and decrease the probability of relatively unsuccessful ones.

  • A simplified clipped policy objective takes the form

    \[\mathcal{L}_{\mathrm{GRPO}} = - \mathbb{E} \left[ \min \left( \rho_i A_i, \operatorname{clip} (\rho_i,1-\epsilon,1+\epsilon)A_i \right) \right]\]

    • where:

      \[\rho_i = \frac{ \pi_\theta(y_i\mid x) }{ \pi_{\theta_{\mathrm{old}}}(y_i\mid x) }\]

  • The exact Qwen3 training implementation includes additional engineering choices and should not be reconstructed solely from this generic GRPO expression. The important documented point is that Qwen3 uses GRPO to optimize the reasoning policy against its query-verifier training set.

Large batches, multiple rollouts, and off-policy training

  • Qwen3 reports three practical choices as particularly beneficial during reasoning RL: large training batches, many rollouts per query, and off-policy training for improved sample efficiency.

  • Multiple rollouts are especially natural under GRPO. Given \(x\), the model produces \(y_1,\ldots,y_G\).

  • The verifier evaluates each trajectory, allowing learning to depend on relative outcomes within the group rather than on a single sampled answer.

  • Increasing \(G\) provides more information about the policy’s response distribution for a particular problem. If only one rollout is generated, it is difficult to determine whether failure represents a generally weak policy or an unlucky sample. With multiple rollouts, the optimization process observes both successful and unsuccessful reasoning paths for the same prompt.

  • Off-policy training further allows previously generated trajectories to contribute to additional updates rather than requiring every policy update to use only freshly sampled outputs. This increases the amount of learning extracted from expensive reasoning rollouts.

Exploration, exploitation, and entropy

  • Reasoning RL also requires balancing exploitation of already-successful solution strategies against continued exploration. If the policy distribution collapses too quickly, the model may repeatedly produce a narrow set of reasoning patterns and lose the ability to discover alternative solutions.

  • Policy entropy provides one measure of this diversity:

    \[H(\pi_\theta) = - \mathbb{E}_{y\sim\pi_\theta} [ \log\pi_\theta(y\mid x) ]\]

  • Qwen3 explicitly manages exploration and exploitation by controlling model entropy so that it either increases steadily or remains at an appropriate level during reasoning RL. The report states that this training strategy enables a single uninterrupted RL run without manual hyperparameter intervention.

  • For Qwen3-235B-A22B, the reported AIME’24 score rises from 70.1 to 85.1 over 170 reasoning-RL training steps. This experiment is particularly notable because it demonstrates substantial reasoning improvement from a relatively small set of 3,995 verifier-backed problems rather than an enormous supervised corpus.

Thinking and non-thinking fusion

  • This section brings together thinking Mode Fusion, preserving reasoning during mode fusion, and constructing non-thinking data. The following subsections explain how these elements fit into the broader chapter.

Thinking Mode Fusion

  • After the first two stages, the model has been optimized primarily as a reasoning model. Stage 3 introduces the complementary behavior required for ordinary interaction: answering directly without generating an extended reasoning trace.

  • Thinking Mode Fusion performs continual supervised fine-tuning on the Stage 2 reasoning model using a mixture of thinking and non-thinking examples. Qwen3 Technical Report by Yang et al. (2025) describes this stage as the mechanism that transforms the reasoning-specialized checkpoint into a single model capable of both operating modes.

  • The training distribution becomes:

    \[\mathcal{D}_{\mathrm{fusion}} = \mathcal{D}_{\mathrm{think}} \cup \mathcal{D}_{\mathrm{nonthink}}\]

  • The optimization objective remains supervised language modeling:

    \[\mathcal{L}_{\mathrm{fusion}} = - \mathbb{E}_{(x,y)\sim\mathcal{D}_{\mathrm{fusion}}} \left[ \log\pi_\theta(y\mid x,m) \right]\]
    • where \(m \in \{\mathrm{think},\mathrm{nonthink}\}\) represents the requested operating mode.

Preserving reasoning during mode fusion

  • A central risk during Stage 3 is catastrophic degradation of the reasoning capability obtained through Stage 2. Qwen3 addresses this by constructing fresh thinking-mode SFT examples through rejection sampling.

  • Stage 1 queries are presented to the Stage 2 reasoning model, candidate answers are generated, and high-quality responses are selected. Thus the reasoning component of the fusion dataset comes from the already-improved reasoning policy rather than reverting to the earlier cold-start teacher distribution.

  • Conceptually,

    \[x\sim\mathcal{D}_{\mathrm{Stage1}}\] \[y\sim\pi_{\mathrm{Stage2}}(\cdot\mid x)\]

    • followed by:

      \[\mathcal{D}_{\mathrm{think}} = \{(x,y):V(x,y)=1\}\]

  • This effectively distills the Stage 2 policy’s improved reasoning behavior back into supervised examples while non-thinking examples are introduced alongside it.

Constructing non-thinking data

  • The non-thinking portion covers a much broader task distribution than reasoning RL. It includes coding, mathematics, instruction following, multilingual tasks, creative writing, question answering, and role-playing. Automatically generated checklists are used to help assess response quality, and translation examples are deliberately upweighted to strengthen low-resource-language performance.

  • This broader mixture is necessary because a reasoning-specialized policy optimized mainly for verifiable mathematics and coding does not automatically become an effective conversational model.

  • The goal of Stage 3 is therefore not \(\text{reasoning}\rightarrow\text{non-reasoning}\) but:

    \[\text{reasoning} \rightarrow \text{reasoning + direct response}\]

  • The original reasoning capability remains available while a second behavioral regime is added.

Chat-template mode control

  • The two modes are exposed through explicit control signals in the chat template. Qwen3 uses /think and /no_think instructions to select reasoning or direct-response behavior. Thinking mode is the default, so some thinking-mode training examples omit the explicit /think flag.

  • The mapping is conceptually:

    \[(x,\text{/think}) \rightarrow \langle\text{think}\rangle r \langle/\text{think}\rangle y\]

    • while:

      \[(x,\text{/no\_think}) \rightarrow \langle\text{think}\rangle \langle/\text{think}\rangle y\]

  • Even in non-thinking mode, an empty thinking block is retained in the assistant representation. This preserves a consistent response format while allowing the template to suppress explicit reasoning.

  • Multi-turn training further randomizes /think and /no_think flags across user turns. The model is trained to obey the most recently encountered flag, enabling mode changes within an ongoing conversation rather than only at conversation initialization.

Thinking budgets as an emergent consequence of fusion

  • Thinking Mode Fusion also enables Qwen3’s thinking-budget mechanism. Interestingly, the report states that intermediate reasoning budgets are not explicitly trained as a separate capability. They emerge after the model learns both full thinking and non-thinking behavior.

  • Suppose the model generates reasoning tokens:

    \[r_1,r_2,\ldots,r_T\]

  • For a user-defined budget \(B<T\), generation can be interrupted after:

    \[r_1,\ldots,r_B\]

  • A stop-thinking instruction is then inserted, after which the model produces its final answer conditioned on the partial reasoning trajectory:

    \[p \left( y \mid x,r_{1:B} \right)\]

  • The inference procedure can therefore be written as:

    \[r_{1:B} \sim \pi_\theta(\cdot\mid x)\]

    • followed by:

      \[y \sim \pi_\theta \left( \cdot \mid x,r_{1:B},s_{\mathrm{stop}} \right)\]
      • where \(s_{\mathrm{stop}}\) is the injected instruction that terminates deliberation and requests an answer from the reasoning accumulated so far.

  • This mechanism transforms reasoning length into a controllable inference-time resource.

General-purpose reinforcement learning

  • This section brings together general RL, instruction and format following, and preference alignment. The following subsections explain how these elements fit into the broader chapter.

General RL

  • The final flagship stage broadens optimization beyond mathematics and coding. General RL targets instruction following, format following, preference alignment, agent behavior, and specialized application scenarios such as retrieval-augmented generation.

  • The objective is therefore multi-dimensional. Instead of optimizing only \(R_{\mathrm{correctness}}\) the model must respond to a broader reward signal that can conceptually include:

    \[R = f( R_{\mathrm{instruction}}, R_{\mathrm{format}}, R_{\mathrm{preference}}, R_{\mathrm{agent}}, R_{\mathrm{specialized}} )\]

  • The report does not specify a single fixed weighted-sum formula for these signals, so this expression should be interpreted as a conceptual decomposition rather than the exact Qwen3 loss.

Instruction and format following

  • Instruction-following RL trains the model to obey requirements concerning response content, structure, length, and formatting. Format-following additionally reinforces Qwen3-specific conventions such as correctly responding to /think and /no_think and consistently separating reasoning and answer portions using designated thinking tags.

  • This is particularly important because hybrid reasoning introduces behavioral constraints absent from an ordinary instruction model. A model that solves a problem correctly but ignores the requested reasoning mode has still failed part of the Qwen3 interface contract.

  • General RL therefore reinforces both semantic correctness and control behavior.

Preference alignment

  • For open-ended requests without deterministic answers, Qwen3 incorporates preference-oriented optimization aimed at helpfulness, engagement, and response style. Unlike mathematical verification, \(R(x,y)\) cannot always be computed through an exact-match rule or programmatic checker.

  • Qwen3 therefore uses learned reward models for tasks in which human preference is the appropriate supervisory signal. The reward model maps a prompt-response pair to a scalar:

    \[R_\phi(x,y)\in\mathbb{R}\]

  • The policy can then be optimized toward outputs receiving higher predicted preference scores.

  • This allows the final RL stage to cover conversational and creative tasks that would be poorly represented by reasoning verifiers alone.

Agent reinforcement learning

  • General RL also explicitly trains agent behavior. The model learns to invoke tools through designated interfaces and, crucially, is allowed to execute complete multi-turn interactions with real environment feedback during RL rollouts.

  • An agent trajectory can be represented as:

    \[\tau = (s_0,a_0,o_1,s_1,a_1,o_2,\ldots,s_T)\]
    • where \(s_t\) is the current conversational or environmental state, \(a_t\) is a model-generated action or tool invocation, and \(o_{t+1}\) is the resulting observation.

  • The policy objective now concerns the success of a trajectory rather than only the quality of one isolated completion:

    \[J(\theta) = \mathbb{E}_{\tau\sim\pi_\theta} [ R(\tau) ]\]

  • This distinction matters for long-horizon agents because early tool calls alter the environment and therefore change the observations available for subsequent decisions.

Reinforcement learning for RAG behavior

  • Qwen3 also treats retrieval-augmented generation as a specialized RL scenario. Reward signals are introduced to encourage responses that accurately use retrieved context and reduce hallucination.

  • For retrieved evidence \(C\) and response \(y\), the desired behavior is not merely linguistic plausibility \(p(y\mid x,C)\) but factual consistency between \(y\) and the supplied evidence \(C\).

  • The report does not disclose the exact RAG reward function, but its inclusion demonstrates that Qwen3’s final RL stage extends beyond generic preference optimization into task-specific behavioral objectives.

Three reward mechanisms

  • General RL uses three classes of reward.

  • The first is rule-based reward. Deterministic or programmatic rules can evaluate tasks such as answer correctness, instruction constraints, and format adherence. For a simple verifiable task,

    \[R_{\mathrm{rule}}(x,y) = \begin{cases} 1,&\text{if }y\text{ satisfies the verifier},\\ 0,&\text{otherwise} \end{cases}\]

  • The advantage is precision. Where an objective rule is available, it avoids introducing the uncertainty of a learned evaluator.

  • The second is model-based reward with a reference answer. Qwen2.5-72B-Instruct receives the prompt, a reference response, and the generated answer and produces a quality judgment. This provides greater flexibility than strict rule matching and reduces false negatives when multiple semantically correct answers are possible.

Reward application and trade-offs

  • The following material develops the implementation and interpretation of three reward mechanisms. It builds on the preceding discussion and makes the remaining details easier to follow.

  • Conceptually,

    \[R_{\mathrm{ref}} = J_{\mathrm{Qwen2.5-72B}} (x,y,y^\star)\]
    • where \(y^\star\) is the reference answer.

  • The third is model-based reward without a reference answer. A reward model trained on human preference data directly scores the response:

    \[R_{\mathrm{pref}} = R_\phi(x,y)\]

  • This mechanism supports open-ended prompts for which no canonical reference response exists.

  • Together, the three reward classes form a hierarchy:

    \[\boxed{ \text{exact rules} \rightarrow \text{reference-conditioned judge} \rightarrow \text{preference reward model} }\]
    • that expands the range of tasks accessible to RL.

The trade-off introduced by generalization

  • The later post-training stages do not improve every capability simultaneously. Qwen3’s ablations show that Thinking Mode Fusion and General RL substantially improve general behavior, instruction following, agent performance, and mode switching, but provide little improvement on specialized reasoning benchmarks. Some difficult mathematics and coding scores decline slightly.

  • For example, the report observes reductions in thinking-mode performance on AIME’24 and LiveCodeBench after the final stages and attributes this to training across a broader distribution of general tasks. The Qwen team explicitly accepts this trade-off in favor of greater overall versatility.

  • This illustrates a recurring post-training tension:

    \[\text{specialization} \leftrightarrow \text{generalization}\]

  • Optimizing a policy heavily for verifiable mathematical reasoning can produce exceptional performance on that distribution, while later optimization for conversation, instructions, tools, multilingual behavior, and preference alignment can move the policy away from the reasoning-only optimum.

Strong-to-weak capability transfer

  • This section brings together strong-to-Weak Distillation, off-policy distillation, and on-policy distillation. The following subsections explain how these elements fit into the broader chapter.

Strong-to-Weak Distillation

  • Running the complete four-stage pipeline independently for every Qwen3 model would be computationally expensive. Qwen3 therefore uses strong-to-weak distillation for six lightweight models:

    1. Qwen3-0.6B

    2. Qwen3-1.7B

    3. Qwen3-4B

    4. Qwen3-8B

    5. Qwen3-14B

    6. Qwen3-30B-A3B

  • The pipeline contains two stages: off-policy response distillation followed by on-policy logit distillation. Teacher supervision comes from Qwen3-32B or Qwen3-235B-A22B.

  • This design transfers not only answer quality but also the hybrid thinking/non-thinking behavior developed through the expensive flagship post-training pipeline.

Off-policy distillation

  • During off-policy distillation, teacher models generate responses in both reasoning modes:

    \[y_{\mathrm{think}} \sim \pi_T(\cdot\mid x,\text{/think})\]

    • and:

      \[y_{\mathrm{nonthink}} \sim \pi_T(\cdot\mid x,\text{/no\_think})\]

  • These teacher-generated responses form a supervised dataset for the student:

    \[\mathcal{D}_{\mathrm{off}} = \{ (x,m,y_T) \}\]

  • The student is trained through response imitation:

    \[\mathcal{L}_{\mathrm{off}} = - \mathbb{E}_{(x,m,y_T)} \left[ \log \pi_S(y_T\mid x,m) \right]\]

  • Because the sequences originate from the teacher rather than the student, this is off-policy knowledge transfer.

  • The stage gives the lightweight model an initial approximation of both teacher behaviors: extended reasoning and direct generation. It also teaches the control semantics associated with switching between them.

On-policy distillation

  • The second phase changes the source of trajectories. Instead of training only on teacher-generated sequences, prompts are sampled and the student itself generates responses in either /think or /no_think mode.

  • Thus,

    \[y_S \sim \pi_S(\cdot\mid x,m)\]

  • The teacher is then evaluated on the same student-generated prefix, and the student’s token distribution is aligned with the teacher’s distribution. The Qwen3 report describes this as minimizing KL divergence between student and teacher logits.

  • At token position \(t\), let \(p_T(\cdot\mid x,y_{S,<t})\) be the teacher distribution and \(p_S(\cdot\mid x,y_{S,<t})\) the student distribution.

  • A corresponding distillation objective is:

    \[\mathcal{L}_{\mathrm{KD}} = \sum_t D_{\mathrm{KL}} \left( p_T(\cdot\mid x,y_{S,<t}) \;\|\; p_S(\cdot\mid x,y_{S,<t}) \right)\]

  • Expanding the KL term:

    \[D_{\mathrm{KL}}(p_T\|p_S) = \sum_v p_T(v) \log \frac{p_T(v)} {p_S(v)}\]

  • Unlike response-only imitation, this transfers information about the teacher’s entire next-token probability distribution, including relative preferences among tokens that were not actually sampled.

Why on-policy distillation matters

  • Off-policy distillation exposes the student to states visited by the teacher. At inference time, however, the student visits states generated by its own policy.

  • These distributions need not be identical:

    \[d_{\pi_T}(s) \neq d_{\pi_S}(s)\]

  • A small mistake by the student can move generation into a prefix that never appears in teacher-generated demonstrations. Subsequent predictions must then operate in an unfamiliar state.

  • On-policy distillation addresses this mismatch by sampling trajectories from \(\pi_S\) itself and asking the teacher what distribution it would produce at those student-visited states:

    \[s\sim d_{\pi_S} \quad\rightarrow\quad p_T(\cdot\mid s) \quad\rightarrow\quad p_S(\cdot\mid s)\]

  • The student therefore learns corrective teacher behavior specifically in regions of sequence space it is likely to encounter during its own inference.

Distillation versus direct reinforcement learning

  • Qwen3 directly compares on-policy distillation with direct reinforcement learning starting from the same off-policy-distilled Qwen3-8B checkpoint. The comparison measures both downstream performance and GPU-hour cost. Qwen3 Technical Report by Yang et al. (2025) reports that on-policy distillation offers a substantially more favorable performance-efficiency trade-off than applying RL directly to the smaller model.

  • This result motivates a different scaling strategy for model families. Rather than paying the exploration cost of reasoning RL separately for every student,

    \[C_{\mathrm{family}} = \sum_{i=1}^{N} C_{\mathrm{RL}}(M_i)\]

  • Qwen3 concentrates the expensive capability-development process in stronger teachers and amortizes it:

    \[C_{\mathrm{family}} \approx C_{\mathrm{RL}}(M_{\mathrm{teacher}}) + \sum_{i=1}^{N} C_{\mathrm{distill}}(M_i)\]

  • When \(C_{\mathrm{distill}} \ll C_{\mathrm{RL}}\), the savings become increasingly important as the number of released model sizes grows.

The complete capability-transfer pipeline

  • Qwen3’s post-training strategy can ultimately be understood as two connected systems.

  • The first discovers and refines behavior in large models:

    \[\boxed{ \text{Base Model} \rightarrow \text{Curated Long-CoT SFT} \rightarrow \text{GRPO Reasoning RL} \rightarrow \\ \text{Thinking/Non-thinking Fusion} \rightarrow \text{General RL} }\]

  • The second compresses that behavior into smaller models:

    \[\boxed{ \text{Flagship Teacher} \rightarrow \text{Off-policy Response Distillation} \rightarrow \text{On-policy Logit Distillation} \rightarrow \\ \text{Lightweight Student} }\]

  • The resulting models inherit not only reasoning performance but also the control structure developed during flagship post-training:

    \[\text{reason deeply}, \qquad \text{answer directly}, \qquad \text{switch between the two}\]

    • and:

      \[\text{operate under an intermediate reasoning budget}\]

Evaluation, Capabilities, and Inference-Time Scaling

  • This chapter examines Qwen3 across reasoning, instruction following, agents, long context, and multilingual evaluation. It also distinguishes improvements from model training from those attributable to additional inference-time computation.

Evaluation setup

  • This section connects evaluation framework with thinking and non-thinking evaluation. The following subsections develop these related topics and their implications.

Evaluation framework

  • Qwen3 is evaluated as both a conventional language-model family and a reasoning-model family. This distinction is important because the same post-trained checkpoint can operate in thinking and non-thinking modes, and the appropriate evaluation configuration depends on which behavior is being measured. Qwen3 Technical Report by Qwen Team (2025) evaluates the family across knowledge, STEM, mathematics, logical reasoning, coding, instruction following, alignment, agentic tool use, multilingual understanding, and long-context processing.

  • The benchmark suite can be organized around several capability groups:

Capability Representative evaluations
Knowledge and STEM MMLU-Redux, GPQA-Diamond
Mathematics MATH-500, AIME'24, AIME'25
Logical reasoning ZebraLogic, AutoLogi
Coding LiveCodeBench v5, CodeForces
Instruction and alignment IFEval, Arena-Hard, AlignBench, WritingBench
Agents and tool use BFCL v3
Multilingual Multi-IF, INCLUDE, MMMLU, MT-AIME, PolyMath, MLogiQA, Belebele
Long context RULER
  • The breadth of this suite matters because Qwen3’s training objective is not to maximize a single reasoning score. Its post-training pipeline deliberately trades some narrow specialization for a model that can reason, converse, follow instructions, use tools, operate multilingually, and switch between computational modes.

Thinking and non-thinking evaluation

  • A defining property of Qwen3 is that a single checkpoint exposes two inference regimes: \(\pi_\theta(y\mid x,m=\mathrm{think})\) and \(\pi_\theta(y\mid x,m=\mathrm{nonthink})\).

  • Thinking mode allows the model to generate an internal reasoning sequence before producing its final answer. Non-thinking mode suppresses extended reasoning and behaves more like a conventional instruction-tuned language model.

  • The modes therefore correspond to different compute profiles. If \(T_r\) is the number of reasoning tokens and \(T_a\) the number of final-answer tokens, thinking-mode generation requires approximately:

    \[T_{\mathrm{think}} = T_r+T_a\]

    • whereas non-thinking generation is approximately:

      \[T_{\mathrm{nonthink}} = T_a\]

  • The comparison is consequently not only about model quality. It measures the trade-off between inference-time computation and capability.

Core reasoning capabilities

  • This section brings together mathematical reasoning, coding capability, and knowledge and STEM. The following subsections explain how these elements fit into the broader chapter.

Mathematical reasoning

  • Mathematics is one of the areas where Qwen3’s reasoning-oriented training produces its largest reported gains. The technical report evaluates models on MATH-500, AIME’24, AIME’25, and multilingual mathematical benchmarks.

  • AIME is particularly useful for reasoning-model evaluation because its problems generally require multi-step mathematical derivation rather than straightforward factual recall. The final answer is also compact and objectively verifiable, making it well suited to both reinforcement-learning rewards and evaluation.

  • For the flagship Qwen3-235B-A22B in thinking mode, the report gives scores of 85.7 on AIME’24 and 81.5 on AIME’25. Qwen3 Technical Report by Qwen Team (2025) reports these results as part of its evaluation of the final post-trained model.

  • These results should be interpreted together with the post-training procedure. Mathematical reasoning is one of the domains emphasized during Long-CoT Cold Start and Reasoning RL, where objective verifiers provide particularly clean reward signals. The benchmark gains therefore correspond directly to a capability targeted by the training pipeline.

Coding capability

  • Coding is evaluated with benchmarks including LiveCodeBench and CodeForces. LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code by Jain et al. (2024) introduces a continuously updated coding benchmark sourced from competitive-programming platforms, reducing reliance on static coding datasets that may become contaminated during model training.

  • Qwen3-235B-A22B achieves a reported LiveCodeBench v5 score of 70.7 and a CodeForces rating of 2056.

  • These are results reported by the Qwen3 Technical Report by Qwen Team (2025), rather than independent measurements.

  • Coding and mathematics share an important property for post-training: both admit relatively strong verification. Generated programs can be compiled and executed against tests, while mathematical answers can often be checked symbolically or against known solutions.

  • A coding reward can therefore be expressed schematically as:

    \[R_{\mathrm{code}}(y) = \frac{1}{N} \sum_{i=1}^{N} \mathbb{1} [ \operatorname{test}_i(y)=\mathrm{pass} ]\]

  • This makes coding particularly compatible with the verifier-driven RL strategy used by Qwen3.

Knowledge and STEM

  • Qwen3 evaluates broad knowledge and scientific reasoning using benchmarks such as MMLU-Redux and GPQA-Diamond.

  • These benchmarks test a different capability regime from AIME or competitive programming. A model may need both factual knowledge acquired during pre-training and reasoning mechanisms strengthened during post-training.

  • Conceptually, performance depends on both components:

    \[P(\mathrm{correct}) = f( K_{\mathrm{pretrain}}, R_{\mathrm{posttrain}}, C_{\mathrm{context}} )\]
    • where \(K_{\mathrm{pretrain}}\) represents knowledge encoded during large-scale pre-training, \(R_{\mathrm{posttrain}}\) represents reasoning behavior developed during post-training, and \(C_{\mathrm{context}}\) captures information available directly in the prompt.

  • This helps explain why Qwen3’s reasoning development begins during pre-training rather than relying exclusively on RL. The model enters post-training with substantial STEM, mathematical, and coding exposure already embedded in its parameters.

Instruction following and agency

  • This section brings together instruction following, evaluating mode switching, and agent and tool-use evaluation. The following subsections explain how these elements fit into the broader chapter.

Instruction following

  • Qwen3 evaluates instruction-following ability with IFEval and several internal or complementary instruction-oriented benchmarks.

  • Instruction-Following Evaluation for Large Language Models by Zhou et al. (2023) introduces IFEval, which uses objectively verifiable requirements such as response length, required keywords, formatting constraints, and other explicitly checkable instructions.

  • This style of evaluation is particularly relevant to Qwen3 because General RL explicitly optimizes instruction and format adherence.

  • For an instruction containing constraints \(C=\{c_1,c_2,\ldots,c_n\}\), one simple evaluation can be expressed as:

    \[S(y) = \frac{1}{n} \sum_{i=1}^{n} \mathbb{1} [c_i(y)=1]\]

  • A strict prompt-level metric can instead require every condition to hold:

    \[S_{\mathrm{strict}}(y) = \prod_{i=1}^{n} \mathbb{1} [c_i(y)=1]\]

  • This makes instruction following suitable for rule-based evaluation and, correspondingly, rule-based reinforcement-learning rewards.

Evaluating mode switching

  • Ordinary instruction benchmarks do not test whether a hybrid reasoning model correctly obeys requests to enter or exit thinking mode. Qwen3 therefore evaluates this behavior explicitly with ThinkFollow.

  • The benchmark is particularly relevant after Thinking Mode Fusion and General RL because those stages teach the model to interpret /think and /no_think.

  • The report’s post-training ablation shows a ThinkFollow score of 88.7 after Thinking Mode Fusion, increasing to 98.9 after General RL.

  • The result indicates that integrating two behavioral modes through SFT alone produces substantial but imperfect controllability. General RL then substantially reduces remaining mode-switching errors.

  • This distinction is important because a unified reasoning model needs two kinds of capability: (i) reasoning quality and (ii) reasoning controllability.

  • A model that reasons well but ignores requests to avoid extended reasoning does not fully satisfy the intended interface.

Agent and tool-use evaluation

  • Agentic capability is evaluated in part using BFCL v3. The Berkeley Function Calling Leaderboard is designed to test whether models can correctly identify and invoke functions or tools from structured specifications.

  • For Qwen3-235B-A22B, the report gives a BFCL v3 score of 70.8.

  • Qwen3 Technical Report by Qwen Team (2025) highlights agent-related performance as one of the model family’s strongest capability areas.

  • Agent evaluation differs from conventional question answering because correctness can depend on selecting the appropriate function, constructing valid arguments, respecting tool schemas, and reasoning across observations returned by the environment.

  • An agent trajectory can be represented as:

    \[\tau = (s_0,a_0,o_1,a_1,o_2,\ldots,a_T)\]

    • and evaluation may depend on the success of the entire trajectory:

      \[R(\tau) \neq R(a_0)\]

  • This corresponds directly to Qwen3’s General RL setup, in which agents can execute complete multi-turn interactions and receive real environment feedback during training.

Smaller models and distillation

  • This section connects lightweight models and capability density with evidence for strong-to-weak distillation. The following subsections develop these related topics and their implications.

Lightweight models and capability density

  • An important part of Qwen3’s evaluation is not simply whether the largest model performs well, but how much capability can be retained as model size decreases.

  • The technical report finds that Qwen3-30B-A3B and Qwen3-14B in non-thinking mode surpass several larger non-reasoning baselines across much of the evaluation suite. Smaller models such as Qwen3-8B, 4B, 1.7B, and 0.6B are likewise compared against models with similar or larger parameter counts.

  • The relevant efficiency quantity is therefore not only total parameter count \(P_{\mathrm{total}}\), but for MoE models the activated parameter count \(P_{\mathrm{active}}\).

  • For Qwen3-30B-A3B,

    \[P_{\mathrm{total}} \approx30\mathrm{B}\]

    • while:

      \[P_{\mathrm{active}} \approx3\mathrm{B}\]

  • Similarly, Qwen3-235B-A22B has \(P_{\mathrm{total}}\approx235\mathrm{B}\) but approximately \(P_{\mathrm{active}}\approx22\mathrm{B}\) parameters participating in each token’s computation.

  • This is one reason the Qwen3 evaluations repeatedly distinguish total model capacity from inference-time active capacity.

Evidence for strong-to-weak distillation

  • The lightweight-model results also provide empirical evidence for the strong-to-weak distillation pipeline described earlier.

  • Instead of asking whether a small student exactly reproduces a teacher, the relevant question is how much of the teacher’s reasoning and instruction-following behavior survives under a much smaller inference budget.

  • If teacher capability is represented by \(Q_T\) and student capability after distillation by \(Q_S\) one useful conceptual quantity is the retention ratio:

    \[\rho = \frac{Q_S}{Q_T}\]

  • Qwen3 does not define a single aggregate retention metric of this form, because performance varies substantially across benchmarks. Nevertheless, the individual benchmark results show that relatively small Qwen3 models retain substantial reasoning capability after off-policy and on-policy distillation.

  • The evaluation therefore supports a central design decision of the family: capability discovery and capability deployment do not have to occur at the same model scale.

Inference-time scaling

  • This section brings together thinking budget as inference-time scaling, test-time compute as a controllable resource, and thinking is not universally beneficial. The following subsections explain how these elements fit into the broader chapter.

Thinking budget as inference-time scaling

  • One of Qwen3’s most distinctive evaluation experiments studies what happens when the amount of reasoning computation is deliberately varied.

  • For prompt \(x\), let \(B\) denote the maximum number of thinking tokens permitted before the model is instructed to produce its final answer.

  • The resulting model can be viewed as implementing a family of inference policies

    \[\pi_\theta^{(B)}(y\mid x)\]
    • all sharing the same parameters \(\theta\) but receiving different amounts of test-time computation.

  • The following figure (source) shows the performance of Qwen3-235B-A22B as its allocated thinking budget increases across mathematics, coding, and STEM benchmarks.

  • The report observes smooth performance improvements as the thinking budget increases across the four evaluated benchmarks. It further suggests that extending output lengths beyond 32K may yield additional gains, although this is presented as future work rather than a demonstrated result.

Test-time compute as a controllable resource

  • The thinking-budget result changes the usual interpretation of inference.

  • For a conventional non-reasoning language model, inference quality is primarily determined by the fixed model parameters and the input:

    \[Q \approx f(\theta,x)\]

  • For a reasoning model with controllable deliberation, quality additionally becomes a function of inference-time computation:

    \[Q \approx f(\theta,x,B)\]

  • Increasing \(B\) does not change the model weights. Instead, it permits the same model to perform additional sequential computation before committing to an answer.

  • This creates a practical quality-latency trade-off. If token-generation cost is approximately proportional to generated sequence length, then:

    \[C_{\mathrm{inference}} \propto B+T_a\]

  • A simple deployment objective could therefore be expressed as:

    \[B^\star = \arg\max_B \left[ Q(B)-\lambda C(B) \right]\]
    • where \(\lambda\) represents the application’s sensitivity to latency or inference cost.

  • The optimal budget need not be the largest available budget. A straightforward factual query may receive little benefit from additional deliberation, whereas a difficult proof or programming problem may justify substantially more inference-time computation.

Thinking is not universally beneficial

  • The long-context experiments provide an important counterexample to the idea that more reasoning always improves performance.

  • Qwen3 evaluates long-context processing with RULER. RULER: What’s the Real Context Size of Your Long-Context Language Models? by Hsieh et al. (2024) extends needle-in-a-haystack evaluation with multi-hop tracing, aggregation, and increasingly difficult retrieval configurations across long sequences.

  • For these experiments, Qwen3 uses YaRN with a scaling factor of 4 and limits thinking mode to a reasoning budget of 8192 tokens to prevent excessive reasoning over extremely long inputs.

  • The report finds that non-thinking Qwen3 models outperform similarly sized Qwen2.5 models on long-context processing. In thinking mode, however, performance decreases slightly.

Task-dependent effects

  • The following material develops the implementation and interpretation of thinking is not universally beneficial. It builds on the preceding discussion and makes the remaining details easier to follow.

  • The Qwen team hypothesizes that explicit reasoning offers limited benefit on retrieval-oriented tasks and can instead interfere with retrieving the relevant information from the context.

  • This suggests that \(\frac{\partial Q}{\partial B}\) is task-dependent.

  • For reasoning-intensive problems, \(\frac{\partial Q}{\partial B}>0\) over much of the useful budget range.

  • For retrieval-dominated tasks, however, the derivative can become approximately zero or even negative:

    \[\frac{\partial Q}{\partial B}\leq0\]

  • The implication is that adaptive reasoning is more useful than indiscriminately maximizing reasoning length.

Context, language, and ablations

  • This section brings together long-context evaluation, multilingual evaluation, and multilingual reasoning. The following subsections explain how these elements fit into the broader chapter.

Long-context evaluation

  • The RULER results also distinguish nominal context-window size from effective context utilization.

  • A model may technically accept 128K tokens while failing to reliably retrieve or combine information located throughout that window. RULER by Hsieh et al. (2024) was explicitly designed to expose this distinction, showing that claimed maximum context length can substantially exceed the range over which models maintain strong task performance.

  • Qwen3’s long-context evaluation therefore tests the combination of several mechanisms discussed earlier:

\[\begin{gathered} \text{32K long-context training} \\ + \\ \text{RoPE base-frequency adaptation} \\ + \\ \text{YaRN} \\ + \\ \text{Dual Chunk Attention} \end{gathered}\]
  • The reported improvement over comparable Qwen2.5 models in non-thinking mode suggests that the long-context training and positional-extension pipeline improves effective long-context processing, not merely the model’s accepted input length.

Multilingual evaluation

  • Qwen3’s multilingual evaluation reflects the expansion from 29 languages in Qwen2.5 to 119 languages and dialects in Qwen3.

  • The report provides detailed evaluations for languages including Spanish, French, Portuguese, Italian, Arabic, Japanese, Korean, Indonesian, Russian, Vietnamese, German, and Thai. It additionally evaluates broader multilingual understanding with Belebele.

  • The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants by Bandarkar et al. (2023) introduces a parallel multiple-choice reading-comprehension benchmark spanning 122 language variants, allowing language-model comprehension to be compared across high-, medium-, and low-resource languages.

  • Qwen3 evaluates 80 supported Belebele languages while excluding 42 languages that the model family does not optimize for. The report finds substantial improvement over Qwen2.5 and performance comparable to similarly sized Gemma models on this evaluation.

  • This is an important qualification to the headline support for 119 languages and dialects. Language coverage and equal performance are not synonymous. The model is trained across a much broader multilingual distribution, but capability remains dependent on the amount, quality, and difficulty of data available for each language.

Multilingual reasoning

  • Qwen3’s multilingual evaluation extends beyond translation and reading comprehension. MT-AIME2024 and PolyMath test whether mathematical reasoning survives when problems are expressed across languages.

  • This evaluates a more difficult transfer:

    \[\text{reasoning capability in English} \rightarrow \text{reasoning capability across languages}\]

  • Ideally, the model learns a representation in which mathematical structure is not tightly coupled to a single surface language:

    \[R(x_{\ell_1}) \approx R(x_{\ell_2})\]
    • when \(x_{\ell_1}\) and \(x_{\ell_2}\) encode the same underlying problem in different languages.

  • In practice, performance remains language-dependent. Nevertheless, multilingual mathematical benchmarks are important for determining whether the 119-language pre-training expansion transfers into complex reasoning rather than only basic generation.

What the post-training ablations reveal

  • Qwen3’s staged training design makes it possible to examine which capabilities emerge at which point in the pipeline.

  • The reported ablations show that Thinking Mode Fusion substantially improves general and instruction-following behavior while introducing non-thinking mode. For example, ThinkFollow reaches 88.7 after Stage 3.

  • General RL then further improves instruction following, agent behavior, tool use, and mode switching, with ThinkFollow increasing to 98.9.

  • At the same time, some specialized reasoning scores decline. The report gives AIME’24 thinking-mode performance decreasing by 1.9 points and LiveCodeBench v5 decreasing by 1.2 points during the relevant later-stage comparison.

  • These results provide an unusually clear illustration of post-training interference:

    \[\Delta Q_k = Q_k(\theta_{t+1}) - Q_k(\theta_t)\]
    • need not be positive for every capability \(k\).

  • An update that improves instruction following or agent behavior can slightly reduce performance on a narrowly optimized mathematical or coding distribution.

Interpretation

  • This section connects interpreting benchmark results carefully with the central evaluation result. The following subsections develop these related topics and their implications.

Interpreting benchmark results carefully

  • Qwen3’s evaluation results support several architectural and training conclusions, but they should not be reduced to a single notion of “model intelligence.”

  • Different benchmarks probe different combinations of memorization, reasoning, formatting, retrieval, multilingual transfer, code execution, tool selection, and preference alignment.

  • For benchmark \(b\) the measured score is better understood as:

    \[S_b = f_b( K, R, I, M, A, C, D )\]
    • where \(K\) represents knowledge, \(R\) reasoning, \(I\) instruction following, \(M\) multilingual capability, \(A\) agent capability, \(C\) context processing, and \(D\) the decoding and evaluation configuration.

  • The relative contribution of each term varies by benchmark.

  • AIME places unusually high weight on mathematical reasoning. BFCL emphasizes tool-use correctness. IFEval emphasizes adherence to verifiable constraints. RULER emphasizes long-context retrieval and manipulation. Belebele emphasizes multilingual reading comprehension.

  • The broad evaluation suite is therefore more informative than any individual headline score.

The central evaluation result

  • The most consequential result of Qwen3’s evaluation is not a single benchmark number. It is that one model family supports several distinct operating regimes.

  • The same post-trained model can approximately move along the continuum:

    \[\boxed{ \text{fast direct response} \longleftrightarrow \text{bounded reasoning} \longleftrightarrow \text{extended reasoning} }\]
    • without loading separate chat and reasoning checkpoints.

  • For difficult mathematical, coding, and STEM tasks, increasing the thinking budget generally improves the reported performance. For retrieval-heavy long-context tasks, additional reasoning can be unnecessary or mildly harmful. For general interaction, non-thinking mode avoids paying the latency and token cost of extended deliberation.

  • The practical objective is therefore not simply:

    \[\max_B Q(B)\]

    • but something closer to:

      \[\max_B \left[ Q(B\mid x) - \lambda_{\mathrm{latency}}C_{\mathrm{latency}}(B) - \lambda_{\mathrm{tokens}}C_{\mathrm{tokens}}(B) \right]\]

  • This reframes inference-time scaling as a resource-allocation problem. Qwen3’s thinking budget provides the mechanism for controlling that resource, while its evaluation results show both sides of the trade-off: additional computation can materially improve reasoning, but it should be allocated according to the structure of the task rather than applied uniformly.

Qwen3-VL: Multimodal Architecture and Representation

  • This chapter examines the components that extend Qwen3 from text to images and video. It follows visual encoding, feature injection, spatial and temporal positioning, and multimodal context management.

Multimodal foundations

  • This section connects extending Qwen3 beyond text with overall multimodal architecture. The following subsections develop these related topics and their implications.

Extending Qwen3 beyond text

  • Qwen3-VL extends the Qwen3 language-model family into a unified multimodal system for text, images, and video. Rather than replacing the Qwen3 decoder with a separate multimodal architecture, it preserves a Qwen3 language-model backbone and introduces a vision encoder plus a vision-language merger. Qwen3-VL Technical Report by Bai et al. (2025) describes the resulting family as supporting native interleaved multimodal contexts of up to 256K tokens.

  • The model family contains dense variants at 2B, 4B, 8B, and 32B scale, together with MoE variants Qwen3-VL-30B-A3B and Qwen3-VL-235B-A22B. The flagship retains the Qwen3 MoE configuration of approximately 235B total parameters and 22B activated parameters per token.

  • The high-level architecture consists of three modules:

    \[\boxed{ \text{Vision Encoder} \rightarrow \text{Vision-Language Merger} \rightarrow \text{Qwen3 LLM} }\]
    • with text tokens entering the language model directly and visual representations entering after encoding and projection.

  • The framework introduces three particularly important architectural changes over earlier Qwen vision-language models:

    1. Interleaved MRoPE

    2. DeepStack

    3. Text-based temporal alignment

  • Together, these mechanisms target three different multimodal problems: representing spatial and temporal position, preserving visual information across the language model’s depth, and grounding video content in physical time.

Overall multimodal architecture

  • Qwen3-VL follows the three-component vision-language architecture used by Qwen2.5-VL: a visual Transformer extracts visual features, a merger converts those features into representations compatible with the language model, and the Qwen3 decoder jointly processes visual and textual tokens.

  • The processing path for an image \(I\) can be represented as:

    \[I \xrightarrow{E_{\mathrm{vision}}} H_{\mathrm{vision}} \xrightarrow{M_{\mathrm{VL}}} Z_{\mathrm{vision}} \xrightarrow{\mathrm{Qwen3}} Y\]
    • where \(E_{\mathrm{vision}}\) is the vision encoder, \(H_{\mathrm{vision}}\) contains its visual hidden states, \(M_{\mathrm{VL}}\) is the multimodal merger, and \(Z_{\mathrm{vision}}\) contains visual tokens projected into the representation space expected by the language model.

  • Text and visual representations can then participate in one interleaved sequence:

    \[X = [ T_1, V_1, T_2, V_2, \ldots ]\]
    • where \(T_i\) represents a text segment and \(V_i\) a visual segment.

  • This design allows the same autoregressive decoder to reason over heterogeneous evidence rather than requiring a separate reasoning module for each modality.

  • The following figure (source) shows the Qwen3-VL architecture, including dynamic-resolution visual encoding, the vision-language merger, DeepStack connections into multiple LLM layers, Interleaved MRoPE, and textual timestamp representations for video.

Vision encoding and merging

  • This section brings together vision encoder, dynamic resolution, and two-dimensional positional encoding. The following subsections explain how these elements fit into the broader chapter.

Vision encoder

  • Qwen3-VL uses SigLIP 2 as the basis of its vision encoder and continues training it with dynamic input resolutions. SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features by Tschannen et al. (2025) extends SigLIP with stronger semantic understanding, localization and dense prediction, multilingual training, and support for multiple resolutions and native aspect ratios.

  • For most Qwen3-VL variants, the report specifies SigLIP2-SO-400M as the starting vision encoder. The smaller 2B and 4B language models instead use the approximately 300M-parameter SigLIP2-Large variant.

  • The vision encoder converts an image into a sequence of patch-level representations:

    \[I \rightarrow \{h_1^{(v)},h_2^{(v)},\ldots,h_N^{(v)}\}\]

  • Unlike systems that force every image into one fixed square resolution, Qwen3-VL is designed to process dynamic, native-resolution visual inputs. Consequently, \(N\) depends on the input resolution and aspect ratio.

  • This is particularly useful for documents, screenshots, diagrams, charts, and other inputs where aggressive resizing can destroy small text or fine spatial structure.

Dynamic resolution

  • Suppose an input image has dimensions \(H\times W\) and the visual patch size is \(P\times P\). Ignoring implementation-specific padding and preprocessing, the number of patch representations scales approximately as:

    \[N_{\mathrm{patch}} \propto \frac{H}{P} \frac{W}{P}\]

  • Higher-resolution inputs therefore produce more visual tokens and preserve more fine-grained information, but they also increase downstream multimodal computation.

  • This creates a direct visual-compute trade-off:

    \[\text{higher image resolution} \rightarrow \text{more visual tokens} \rightarrow \text{greater visual detail} \rightarrow \text{greater compute}\]

  • Dynamic resolution allows this cost to vary with the input rather than imposing the same token count on every image.

Two-dimensional positional encoding

  • Dynamic image sizes make fixed one-dimensional visual position indices insufficient. Qwen3-VL therefore uses two-dimensional RoPE inside the vision encoder and interpolates absolute position embeddings according to the input size.

  • For a visual patch at coordinates \((h,w)\), its positional representation needs to preserve both axes:

    \[p_{\mathrm{vision}} = (h,w)\]

  • A 2D rotary formulation can conceptually assign transformations according to horizontal and vertical coordinates:

    \[q_{h,w} = R_hR_wq\] \[k_{h,w} = R_hR_wk\]

  • The exact Qwen3-VL implementation should not be inferred from this schematic expression. The important architectural property is that spatial position remains explicitly two-dimensional instead of being represented solely by the patch’s flattened sequence index.

  • Absolute position embeddings are additionally interpolated when the input resolution differs from the encoder’s original positional grid. This enables the pretrained vision encoder to operate over dynamic image dimensions without requiring a separately learned absolute embedding for every possible resolution.

Vision-language merger

  • Raw visual features do not directly match the hidden dimensionality or tokenization expected by the Qwen3 decoder. Qwen3-VL therefore uses an MLP-based vision-language merger.

  • The merger is a two-layer MLP that combines each \(2\times2\) group of visual features into one visual token.

  • If the encoder produces features \(h_{i,j}\), a local group \(G_{i,j} = [h_{2i,2j}, h_{2i+1,2j}, h_{2i,2j+1}, h_{2i+1,2j+1}]\) is combined and projected:

    \[z_{i,j} = \operatorname{MLP} \left( \operatorname{Concat}(G_{i,j}) \right)\]

  • The result lies in the LLM hidden space:

    \[z_{i,j}\in\mathbb{R}^{d_{\mathrm{LLM}}}\]

  • Spatially compressing a \(2\times2\) region reduces the visual sequence length by approximately a factor of four:

    \[N_{\mathrm{LLM}} \approx \frac{N_{\mathrm{vision}}}{4}\]

  • This compression is important because visual tokens participate in Transformer computation once they enter the language model.

The information bottleneck of visual tokens

  • A conventional multimodal architecture commonly sends only the final vision-encoder representation into the LLM:

    \[H_{\mathrm{vision}}^{(L_v)} \rightarrow M_{\mathrm{VL}} \rightarrow H_{\mathrm{LLM}}^{(0)}\]

  • This creates an information bottleneck. Different vision-encoder layers represent different levels of visual abstraction, but only the final representation is explicitly transferred to the language model.

  • Earlier layers may retain useful local texture, shape, character, boundary, and spatial information, while later layers increasingly encode higher-level semantics.

  • For tasks such as OCR, document understanding, charts, diagrams, and high-resolution perception, discarding intermediate representations can therefore remove information useful to the language model.

  • Qwen3-VL addresses this problem with DeepStack.

Multi-level visual features

  • This section connects deepStack with why DeepStack helps. The following subsections develop these related topics and their implications.

DeepStack

  • DeepStack: Deeply Stacking Visual Tokens is Surprisingly Simple and Effective for LMMs by Meng et al. (2024) proposes injecting visual representations into multiple aligned layers of a multimodal language model rather than providing all visual information only at the input layer. The original work shows particularly strong improvements on high-resolution and document-oriented tasks.

  • Qwen3-VL adapts this idea by selecting visual features from three different levels of the vision encoder. Each feature set passes through a dedicated merger and is injected into a corresponding early layer of the language model.

  • Let \(H_v^{(l_1)}, H_v^{(l_2)}, H_v^{(l_3)}\) represent features from three vision-encoder depths.

  • Dedicated mergers transform them into:

    \[Z_1=M_1(H_v^{(l_1)})\] \[Z_2=M_2(H_v^{(l_2)})\] \[Z_3=M_3(H_v^{(l_3)})\]

  • These representations are injected into corresponding hidden states of the first three LLM layers through residual addition.

  • Conceptually,

    \[H_{\mathrm{LLM}}^{(k)} \leftarrow H_{\mathrm{LLM}}^{(k)} + Z_k, \qquad k\in\{1,2,3\}\]

  • The exact layer mapping and tensor operations follow the Qwen3-VL implementation, but the central idea is straightforward: visual information enters the language model at multiple depths.

Why DeepStack helps

  • Without DeepStack, the LLM receives a single compressed representation:

    \[V = M(H_v^{(L_v)})\]

  • All multimodal reasoning must then proceed from this one visual abstraction.

  • With DeepStack, the language model receives:

    \[V^{(1)},V^{(2)},V^{(3)}\]
    • derived from different vision-encoder depths.

  • This creates multiple information pathways:

    \[\boxed{ \text{low/mid/high-level visual features} \rightarrow \text{multiple early LLM layers} }\]

    • rather than:

      \[\boxed{ \text{final visual features} \rightarrow \text{LLM input only} }\]

  • The mechanism is especially relevant to fine-grained perception because semantic compression in the upper vision layers can obscure low-level information needed for tasks such as reading small text or distinguishing subtle visual patterns.

  • The Qwen3-VL report’s DeepStack ablation uses an internal 15B-A2B language model pretrained on 200 billion tokens. Average benchmark performance increases from 74.7 without DeepStack to 76.0 with DeepStack, with notable improvements on InfoVQA and DocVQA.

  • This provides direct evidence that the additional multi-level visual pathways improve downstream visual understanding rather than merely changing the architecture.

Spatial and temporal position

  • This section brings together multimodal positional encoding, from MRoPE to Interleaved MRoPE, and why frequency distribution matters. The following subsections explain how these elements fit into the broader chapter.

Multimodal positional encoding

  • Images and videos introduce positional dimensions that ordinary text does not contain.

  • Text naturally has one sequence coordinate:

    \[p_{\mathrm{text}}=t\]

  • An image requires at least two spatial coordinates:

    \[p_{\mathrm{image}}=(h,w)\]

  • Video adds time:

    \[p_{\mathrm{video}}=(t,h,w)\]

  • A multimodal language model must therefore represent temporal and spatial relationships while remaining compatible with ordinary text tokens.

  • Qwen’s earlier multimodal models address this with Multimodal Rotary Position Embedding, or MRoPE. Qwen3-VL introduces a modified form called Interleaved MRoPE.

From MRoPE to Interleaved MRoPE

  • The original MRoPE formulation partitions embedding dimensions into temporal, horizontal, and vertical subspaces. Conceptually,

    \[d = d_t+d_h+d_w\]

  • The dimensions in \(d_t\) receive temporal positional rotations, those in \(d_h\) receive horizontal rotations, and those in \(d_w\) receive vertical rotations.

  • A schematic representation is

    \[\operatorname{MRoPE}(x,t,h,w) = [ R_t x_t; R_h x_h; R_w x_w ]\]

  • The problem is that RoPE dimensions correspond to different frequency bands. Contiguous partitioning can therefore associate a particular positional axis with a disproportionately narrow portion of the frequency spectrum.

Spatial and temporal encoding details

  • This section develops the implementation and interpretation of from MRoPE to Interleaved MRoPE. It builds on the preceding discussion and makes the remaining details easier to follow.

  • For example, if temporal dimensions occupy one contiguous section, temporal position may primarily receive one band of rotary frequencies, while spatial dimensions receive different bands.

  • Qwen3-VL instead interleaves the temporal, horizontal, and vertical dimensions across the rotary-frequency spectrum.

  • Rather than an arrangement conceptually resembling:

    \[[t,t,t,\ldots,h,h,h,\ldots,w,w,w,\ldots]\]

    • Interleaved MRoPE distributes the axes more like:

      \[[t,h,w,t,h,w,t,h,w,\ldots]\]

  • The exact allocation is implementation-specific, but the principle is that every positional axis receives access to a more balanced range of low- and high-frequency rotary components.

Why frequency distribution matters

  • For RoPE, different dimensions rotate at different rates. A generic rotary frequency can be written as

    \[\omega_i = \theta^{-2i/d}\]

  • Low-frequency components change slowly with position and are useful for representing long-range positional differences. High-frequency components change rapidly and provide finer local discrimination.

  • If one modality axis receives predominantly one part of this spectrum, its positional representation may be biased toward either coarse or fine positional information.

  • Interleaving distributes frequencies more evenly:

    \[\Omega_t \approx \Omega_h \approx \Omega_w\]
    • in terms of coverage across the overall frequency spectrum.

  • The intended result is better balance between temporal and spatial localization, particularly when processing interleaved images and long videos.

Images under Interleaved MRoPE

  • For an image, visual tokens are associated with horizontal and vertical coordinates. The temporal coordinate can remain constant across patches belonging to the same image.

  • For visual token \(v_{i,j}\), its multimodal coordinate can be conceptualized as \(p(v_{i,j}) = (t_0,i,j)\).

  • Thus all patches share \(t_0\) but differ in \(i\) and \(j\).

  • This allows attention to retain the two-dimensional structure of the image even though the visual tokens eventually participate in the same Transformer sequence as text.

Video under Interleaved MRoPE

  • Video adds another dimension.

  • For patch \(v_{f,i,j}\) from frame \(f\) the position becomes \(p(v_{f,i,j}) = (t_f,i,j)\).

  • The temporal component differentiates frames while the spatial components preserve patch location within each frame.

  • The resulting representation therefore provides attention with structured information about both “where” and “when”.

  • This is necessary for questions involving temporal ordering, motion, event localization, or relationships between visually similar frames appearing at different points in a video.

Text-based temporal alignment

  • Qwen3-VL also changes how video time is communicated to the language model.

  • Instead of relying solely on positional embeddings to encode temporal information, the model introduces explicit textual timestamps into the sequence. Qwen3-VL Technical Report by Bai et al. (2025) describes this as a transition from implicit temporal position toward text-based time alignment.

  • For example, a video sequence can conceptually appear as:

      <3.0 seconds>
  [visual tokens for frame or clip]

  <6.0 seconds>
  [visual tokens for frame or clip]

  <9.0 seconds>
  [visual tokens for frame or clip]

  • Time therefore becomes part of the linguistic context itself.

  • If frame \(f_i\) corresponds to physical time \(\tau_i\) the multimodal sequence explicitly represents \((\tau_i,V_i)\).

  • The language model can consequently reason about \(\tau_i\) using mechanisms already learned for processing numbers and text.

Why textual timestamps are useful

  • Pure positional embeddings encode ordering well: \(f_1<f_2<f_3\)

  • But ordering does not necessarily communicate physical duration.

  • Suppose three sampled frames correspond to: \(1\text{ s}, \quad 2\text{ s}, \quad 120\text{ s}\)

  • A simple sequential representation establishes their order but does not directly tell the language model that the third event occurs almost two minutes after the second. Explicit timestamps preserve this information: \(\tau_3-\tau_2 = 118\text{ s}\)

  • This makes questions such as “What happens around 2 minutes into the video?” or “How much time passes between the first and second events?” more naturally compatible with language-model reasoning.

  • The report supports both second-based timestamps and hour-minute-second representations for longer videos.

Context and representation trade-offs

  • This section brings together native interleaved multimodal context, visual token length and context consumption, and preserving the text capabilities of Qwen3. The following subsections explain how these elements fit into the broader chapter.

Native interleaved multimodal context

  • Qwen3-VL supports a native context window of up to 256K tokens for interleaved text, images, and video.

  • This is materially different from merely allowing a large text prompt alongside one image. A long multimodal sequence can contain multiple visual and textual segments:

    \[X = [ T_1, I_1, T_2, I_2, T_3, V_1, T_4,\ldots ]\]

  • The model must maintain references across these segments.

  • For example, a document-analysis task might contain dozens or hundreds of page images together with textual instructions, while a video task may contain many temporally sampled visual segments followed by questions referring to events far apart in time.

  • The context window therefore needs to support both long-range sequence modeling and cross-modal alignment.

Visual token length and context consumption

  • Dynamic-resolution vision creates an important systems consequence: visual inputs consume variable portions of the context window.

  • If image \(i\) produces \(N_i\) post-merger visual tokens, a multimodal context containing \(K\) images and \(T\) text tokens has approximate total sequence length:

    \[L = T + \sum_{i=1}^{K}N_i\]

  • For video containing \(F\) sampled frames,

    \[L = T + \sum_{f=1}^{F}N_f + T_{\mathrm{timestamp}}\]
    • where \(T_{\mathrm{timestamp}}\) accounts for explicit temporal labels and related formatting.

  • High-resolution images preserve more visual detail but produce more visual tokens. Dense video sampling improves temporal coverage but likewise increases sequence length.

  • Multimodal inference therefore involves a resource allocation problem across three dimensions:

    \[\boxed{ \text{spatial resolution} \times \text{temporal sampling density} \times \text{text context} }\]
    • under a finite context budget.

Preserving the text capabilities of Qwen3

  • A recurring problem in multimodal model development is that adding vision can degrade the language model’s original text capabilities.

  • Qwen3-VL is explicitly designed to avoid treating multimodal capability as a replacement for language capability. Its language component remains based on Qwen3, while multimodal pre-training jointly updates the system so that visual representations become compatible with the decoder’s existing language and reasoning machinery.

  • The Qwen3-VL Technical Report by Bai et al. (2025) reports that the flagship model surpasses its text-only counterpart on a majority of the language benchmarks used in its evaluation. The report presents this as evidence that multimodal training can strengthen rather than necessarily dilute general language capability.

  • This makes the intended relationship

    \[\text{Qwen3-VL} \neq \text{Qwen3}+\text{isolated image module}\]

  • Instead, multimodal training is intended to produce a model in which visual and textual evidence participate in a shared reasoning process.

Architecture as an information-preservation problem

  • Many of Qwen3-VL’s architectural changes can be understood through a common principle: preserving information as raw multimodal inputs are compressed into a representation that an autoregressive language model can manipulate.

  • Dynamic-resolution vision attempts to preserve spatial detail:

    \[\text{native image} \rightarrow \text{resolution-adaptive visual tokens}\]

  • The vision-language merger reduces token count while mapping visual features into the LLM representation space:

    \[2\times2\text{ visual features} \rightarrow 1\text{ LLM visual token}\]

  • DeepStack preserves information from multiple vision-encoder depths:

    \[\text{multi-level visual hierarchy} \rightarrow \text{multiple LLM layers}\]
Representation and efficiency trade-offs

  • The following material develops the implementation and interpretation of architecture as an information-preservation problem. It builds on the preceding discussion and makes the remaining details easier to follow.

  • Interleaved MRoPE preserves structured position:

    \[(t,h,w) \rightarrow \text{balanced rotary-frequency representation}\]

  • Textual timestamps preserve physical video time:

    \[\text{frame position} \rightarrow \text{explicit temporal semantics}\]

  • Finally, the 256K native context window preserves relationships across long interleaved sequences:

    \[[T_1,I_1,T_2,I_2,\ldots,V_k] \rightarrow \text{joint long-context reasoning}\]

  • The architecture can therefore be summarized as

    \[\boxed{ \begin{aligned} \text{Image / Video} &\rightarrow \text{dynamic-resolution vision encoder}\\ &\rightarrow \text{multi-level visual representations}\\ &\rightarrow \text{2}\times\text{2 MLP token compression}\\ &\rightarrow \text{DeepStack injection}\\ &\rightarrow \text{Interleaved MRoPE}\\ &\rightarrow \text{Qwen3 decoder}\\ &\rightarrow \text{multimodal autoregressive output} \end{aligned} }\]

  • The result is not simply a Qwen3 model capable of accepting pictures. Qwen3-VL modifies how visual features are encoded, compressed, injected, positioned, and temporally grounded so that image and video information can participate in the same long-context reasoning process as language.

Qwen3-VL Pre-training and Multimodal Data

  • This chapter describes the staged Qwen3-VL training curriculum and the data used to develop its visual capabilities. It connects image-text alignment with document understanding, grounding, video, and multimodal reasoning.

Training curriculum

  • This section brings together pre-training overview, continuous training of the vision encoder, and stage 0: Vision-Language Alignment. The following subsections explain how these elements fit into the broader chapter.

Pre-training overview

  • Qwen3-VL’s pre-training pipeline is designed to solve two related problems: establishing alignment between an independently initialized visual representation space and the Qwen3 language model, and progressively extending that aligned system toward increasingly long multimodal contexts. Qwen3-VL Technical Report by Bai et al. (2025) organizes this process into four stages, progressing from an 8K alignment stage to native 256K multimodal context training.

  • The overall curriculum is:

    \[\boxed{ \text{Vision-Language Alignment} \rightarrow \text{Multimodal Pre-training} \rightarrow \text{Long-Context Pre-training} \rightarrow \\ \text{Ultra-Long-Context Adaptation} }\]
    • with the following training configuration:
Stage Objective Trainable parameters Token budget Sequence length
S0 Vision-Language Alignment Merger 67B 8,192
S1 Multimodal Pre-training All ~1T 8,192
S2 Long-Context Pre-training All ~1T 32,768
S3 Ultra-Long-Context Adaptation All 100B 262,144

Continuous training of the vision encoder

  • Before the four-stage multimodal curriculum, the SigLIP-2-based vision encoder undergoes continuous training with dynamic resolutions. SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features by Tschannen et al. (2025) provides the pretrained visual foundation used by Qwen3-VL, while Qwen3-VL further adapts it to variable-resolution inputs.

  • The resulting starting point can be represented as

    \[E_{\mathrm{SigLIP2}} \xrightarrow{\text{dynamic-resolution continuation}} E_{\mathrm{Qwen3\text{-}VL}}\]

  • This visual encoder is then combined with the MLP merger and Qwen3 backbone:

    \[E_{\mathrm{vision}} + M_{\mathrm{VL}} + L_{\mathrm{Qwen3}}\]

  • The staged pre-training process determines when each of these components becomes trainable.

Stage 0: Vision-Language Alignment

  • Stage 0 addresses the initial modality gap between visual features and the language model. Only the vision-language merger is updated:

    \[\theta_{\mathrm{trainable}}=\theta_{\mathrm{merger}}\]
    • while \(\theta_{\mathrm{vision}}\) and \(\theta_{\mathrm{LLM}}\) remain frozen.

  • The model trains on approximately \(67\mathrm{B}\) tokens with a maximum sequence length of 8192.

  • The training mixture consists primarily of high-quality image-caption pairs, visual knowledge data, and OCR data.

  • This stage can be understood as learning a transformation:

    \[M_{\mathrm{VL}}:\mathbb{R}^{d_v}\rightarrow\mathbb{R}^{d_l}\]
    • where \(d_v\) is the representation dimension of the vision encoder and \(d_l\) is the language-model hidden dimension.

  • Because the visual and language backbones are frozen, the merger must learn to map visual features into a representation the pretrained language model can interpret without simultaneously allowing the two underlying representation spaces to drift.

Why alignment comes first

  • Suppose the vision encoder produces \(H_v=E_{\mathrm{vision}}(I)\) and the language model expects embeddings distributed according to some learned representation space:

    \[\mathcal{H}_{\mathrm{LLM}}\]

  • Initially,

    \[H_v\not\sim\mathcal{H}_{\mathrm{LLM}}\]

  • The merger learns \(Z_v=M_{\mathrm{VL}}(H_v)\) such that \(Z_v\) becomes usable by the existing language decoder. Freezing both backbones reduces the degrees of freedom during this initial alignment problem:

    \[\min_{\theta_M}\mathcal{L}\left(L_{\mathrm{Qwen3}}(M_{\theta_M}(E_{\mathrm{vision}}(I)))\right)\]

  • Once this mapping is established, the full system can be jointly optimized.

Stage 1: Multimodal Pre-training

  • Stage 1 unfreezes the entire model:

    \[\theta_{\mathrm{trainable}}=\{\theta_{\mathrm{vision}},\theta_{\mathrm{merger}},\theta_{\mathrm{LLM}}\}\]

  • Training expands to approximately \(1\mathrm{T}\) tokens while retaining an \(8192\) token sequence length.

  • The data mixture now becomes substantially broader. It contains both vision-language and text-only data, with multimodal data covering interleaved image-text documents, visual grounding, visual question answering, STEM tasks, and an initial amount of video data.

  • The inclusion of text-only data is deliberate. Multimodal pre-training should add visual capability without eroding the language capability inherited from Qwen3.

  • The mixture can be represented as:

    \[\mathcal{D}_{S1}=\mathcal{D}_{\mathrm{text}}\cup\mathcal{D}_{\mathrm{image-text}}\cup\mathcal{D}_{\mathrm{interleaved}}\cup\mathcal{D}_{\mathrm{grounding}}\cup\mathcal{D}_{\mathrm{VQA}}\cup\mathcal{D}_{\mathrm{STEM}}\cup\mathcal{D}_{\mathrm{video}}\]

  • This is the stage in which the model transitions from an aligned language model with visual inputs into a jointly trained multimodal foundation model.

Stage 2: Long-Context Pre-training

  • Stage 2 keeps all parameters trainable but quadruples the maximum sequence length: \(8192\rightarrow32768\).

  • Training again uses approximately \(1\mathrm{T}\) tokens. The data mixture also changes. Qwen3-VL increases the proportion of text-only data to reinforce long-form textual understanding while substantially increasing video and agent-oriented instruction-following data in the multimodal portion.

  • Thus Stage 2 changes both \(L_{\mathrm{context}}\) and \(p(d)\), where \(p(d)\) is the sampling distribution over data categories. The goal is not simply to expose the model to longer token sequences. The training data must contain tasks that actually require information to persist and interact over those longer ranges.

  • For video, this means longer temporal sequences. For documents, it means multiple pages or interleaved sections. For agents, it means longer interaction histories and multi-step instructions.

Stage 3: Ultra-Long-Context Adaptation

  • The final pre-training stage increases sequence length from \(32768\rightarrow262144\).

  • Rather than repeating another trillion-token stage, Qwen3-VL uses a more focused \(100\mathrm{B}\) token corpus emphasizing long videos and long documents.

  • The curriculum therefore follows:

    \[8\mathrm{K}\rightarrow32\mathrm{K}\rightarrow256\mathrm{K}\]

  • This staged expansion avoids paying the computational cost of 256K-context training throughout the entire pre-training process. For standard self-attention, attention computation scales approximately as \(O(L^2)\) so increasing sequence length from \(8\mathrm{K}\) to \(256\mathrm{K}\) without changing any other assumptions represents a \(32\times\) increase in sequence length and potentially a much larger increase in raw attention work. Qwen3-VL therefore reserves ultra-long training for a relatively focused adaptation stage.

Progressive context expansion as curriculum learning

  • The four stages can be interpreted as a curriculum over two dimensions: (i) modality complexity and (ii) context length.

  • Early training solves the comparatively constrained problem:

    \[\text{visual feature}\rightarrow\text{language representation}\]

  • Later stages solve:

    \[\text{images + text + video + tools + long-range dependencies}\]

  • The context curriculum simultaneously progresses as:

    \[8192\rightarrow8192\rightarrow32768\rightarrow262144\]

  • The system therefore does not attempt to learn visual alignment, multimodal reasoning, video understanding, and 256K long-context behavior simultaneously from initialization.

Square-root-normalized token loss

  • Qwen3-VL also changes how training examples contribute to the optimization objective. Earlier approaches can assign losses at the sample level, but multimodal training produces unusually large variation in sample length. A short text instruction, a high-resolution image example, and a long video sequence can contain very different numbers of tokens.

  • Qwen3-VL therefore moves to a square-root-normalized per-token loss. The Qwen3-VL Technical Report by Bai et al. (2025) states that this modification better balances text-only and multimodal learning and improves multimodal performance without compromising text capabilities.

  • The report establishes the square-root reweighting principle but does not provide enough detail to justify reconstructing a more specific implementation equation beyond its stated normalization behavior. The central objective is to prevent sample-length differences from causing either very long multimodal examples or short text examples to contribute disproportionately to training.

  • Conceptually, if a sample contains \(n\) prediction tokens, its contribution is reweighted as a function of \(\sqrt{n}\) rather than treating every sample identically regardless of length. This is especially important because Qwen3-VL’s training distribution contains samples ranging from short image-caption pairs to 256K-token interleaved contexts.

Image-text and visual knowledge

  • This section connects caption and interleaved image-text data with visual knowledge data. The following subsections develop these related topics and their implications.

Caption and interleaved image-text data

  • A major component of multimodal pre-training is image-text alignment data. Qwen3-VL applies aggressive quality filtering to web-scale image-text corpora and replaces weak textual supervision, such as sparse alt-text, with richer model-generated descriptions.

  • Rather than merely learning:

    \[I\leftrightarrow\text{object label}\]

    • the desired supervision includes:

      \[I\leftrightarrow\{\text{objects},\text{attributes},\text{relationships},\text{layout},\text{context},\text{interactions}\}\]

  • For book-scale interleaved data, a fine-tuned Qwen2.5-VL-7B parses text and embedded visual elements while maintaining their alignment. Consecutive pages can then be merged into sequences reaching \(256\mathrm{K}\) tokens.

  • Quality controls remove pure-text or weakly aligned segments and require ultra-long book sequences to contain sufficient pages and a meaningful image-to-text ratio. This is important because a nominally multimodal 256K sequence dominated almost entirely by text would provide little training signal for long-range visual-text interaction.

Visual knowledge data

  • Qwen3-VL also constructs a large entity-centered visual knowledge corpus spanning more than a dozen semantic categories, including animals, plants, landmarks, food, vehicles, electronics, clothing, and other real-world or fictional concepts.

  • The underlying entity distribution is long-tailed. Uniform sampling would overrepresent the rare tail relative to annotation quality, while pure frequency sampling would underrepresent it. Qwen3-VL instead uses importance-based sampling, assigning more training weight to prominent entities while retaining less frequent entities for coverage.

  • The objective can be conceptualized as balancing:

    \[\text{quality}+\text{utility}+\text{coverage}\]

  • Captions are enriched to describe not only identity but also visual attributes, surrounding context, spatial arrangement, and interactions.

OCR and document understanding

  • This section brings together oCR training, multilingual OCR, and document parsing. The following subsections explain how these elements fit into the broader chapter.

OCR training

  • OCR receives substantial dedicated data investment. Qwen3-VL uses approximately \(30\mathrm{M}\) internally collected OCR samples processed through a coarse-to-fine pseudo-labeling pipeline. OCR-specialized models provide initial labels, which are subsequently refined using Qwen2.5-VL. No human annotation is required for this particular pipeline.

  • Multilingual OCR is expanded substantially beyond Qwen2.5-VL. Qwen3-VL adds \(29\) languages and synthesizes approximately \(30\mathrm{M}\) high-quality multilingual OCR examples, together with more than \(1\mathrm{M}\) internally collected real-world multilingual images.

  • The model’s OCR training distribution can therefore be viewed as:

    \[\mathcal{D}_{\mathrm{OCR}}=\mathcal{D}_{\mathrm{real}}\cup\mathcal{D}_{\mathrm{pseudo}}\cup\mathcal{D}_{\mathrm{synthetic}}\cup\mathcal{D}_{\mathrm{multilingual}}\]

  • This mixture is intended to provide both scale and robustness to real-world typography, layouts, scripts, and imaging conditions.

Multilingual OCR

  • Qwen3-VL expands visual text recognition to 39 languages in the evaluation described by the report.

  • The following figure (source) shows multilingual OCR accuracy on the report’s internally constructed evaluation set. The model exceeds 70% accuracy on 32 of the 39 tested languages.

  • This multilingual expansion is important because visual-language multilinguality differs from ordinary text multilinguality. The system must recognize the script visually before linguistic interpretation can begin:

    \[\text{pixels}\rightarrow\text{characters}\rightarrow\text{words}\rightarrow\text{semantics}\]

  • Errors in visual transcription propagate into downstream language reasoning.

Document parsing

  • Document understanding requires more than recognizing characters. The model must recover reading order, layout, tables, figures, and relationships between spatial regions.

  • Qwen3-VL collects approximately \(3\mathrm{M}\) PDFs from Common Crawl, balanced across ten document categories at approximately \(300\mathrm{K}\) samples per category, together with approximately \(4\mathrm{M}\) internal documents.

  • An internal layout model first predicts reading order and bounding boxes for textual and non-textual regions. Qwen2.5-VL-72B then performs region-level recognition.

  • The pipeline is approximately:

    \[\text{PDF}\rightarrow\text{layout segmentation}\rightarrow\text{reading order}\rightarrow\text{region recognition}\rightarrow\text{structured reconstruction}\]

  • This produces position-aware training data rather than plain OCR transcripts.

QwenVL-HTML and QwenVL-Markdown

  • Qwen3-VL represents parsed documents in two formats. QwenVL-HTML preserves fine-grained element-level bounding boxes, while QwenVL-Markdown localizes images and tables and represents tables using LaTeX.

  • These formats support different levels of structural fidelity:

    \[\text{QwenVL-HTML}\rightarrow\text{fine-grained layout preservation}\] \[\text{QwenVL-Markdown}\rightarrow\text{lighter structured document representation}\]

  • Large synthetic HTML corpora provide precise structural supervision. Real-document pseudo-labels are then generated and filtered for quality. The final corpus combines:

    \[\text{synthetic precision}+\text{real-world diversity}\]

Long-document understanding

  • Long-document training extends beyond parsing individual pages. Qwen3-VL constructs synthetic long-document sequences by combining multiple single-page samples. Page images are placed at the beginning of the sequence and followed by corresponding OCR- or HTML-derived text.

  • The model also receives long-document VQA examples generated from high-quality multi-page PDFs. These questions require reasoning across pages and across heterogeneous content including text, tables, charts, and figures.

  • A long-document question can therefore require evidence:

    \[E=\{e_{p_1},e_{p_2},\ldots,e_{p_k}\}\]

    • distributed across multiple pages. The model must retrieve those pieces and combine them:

      \[y=f(x,e_{p_1},e_{p_2},\ldots,e_{p_k})\]

  • This is substantially different from simply parsing a 100-page document into text. It explicitly trains cross-page multimodal reasoning.

Spatial supervision

  • This section brings together grounding, point-based grounding and counting, and normalized coordinates. The following subsections explain how these elements fit into the broader chapter.

Grounding

  • Visual grounding trains the model to associate language with precise locations in an image. Qwen3-VL supports both bounding-box and point-based grounding.

  • Its box-grounding corpus combines established datasets such as COCO, Objects365, OpenImages, and RefCOCO-family datasets with a large synthetic annotation pipeline.

  • Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection by Liu et al. (2023) provides the open-vocabulary detection component used together with Qwen2.5-VL during Qwen3-VL’s synthetic grounding pipeline.

  • The pipeline follows:

    \[\text{unlabeled image}\rightarrow\text{candidate objects}\rightarrow\text{localization}\rightarrow\text{quality filtering}\]

  • Qwen2.5-VL first identifies candidate objects. Grounding DINO and Qwen2.5-VL then provide localization and annotation, followed by confidence and accuracy filtering.

Point-based grounding and counting

  • Point grounding combines public pointing annotations, detection and segmentation datasets converted into point supervision, and synthetically generated annotations targeting fine-grained image details.

  • A target can be represented as:

    \[p=(x,y)\]

    • rather than a box:

      \[b=(x_1,y_1,x_2,y_2)\]

  • Qwen3-VL additionally derives counting tasks from this grounding data. Three task formulations are used: direct counting, box-based counting, and point-based counting. This couples quantitative visual reasoning with explicit localization.

Normalized coordinates

  • Qwen3-VL changes the grounding coordinate representation from its predecessor by normalizing coordinates to \([0,1000]\).

  • For an image of width \(W\) and height \(H\), a pixel coordinate \((x,y)\) can conceptually be mapped to

    \[x'=1000\frac{x}{W}\] \[y'=1000\frac{y}{H}\]

  • This normalization makes coordinate outputs less dependent on absolute input resolution and aspect ratio. The model therefore learns a resolution-independent spatial language:

    \[\text{pixels}\rightarrow\text{normalized spatial coordinates}\]

Spatial understanding

  • Qwen3-VL’s spatial training extends beyond object localization. Its specialized spatial-understanding data includes relational descriptions such as one object being to the left of another, affordance labels such as whether an object is graspable or pressable, and action-conditioned queries requiring planning.

  • This moves the task from \(\text{Where is object }A?\) toward questions such as \(\text{How is }A\text{ related to }B?\) and \(\text{What action involving }A\text{ is feasible?}\).

  • Spatial references are deliberately expressed relative to other objects or scene frames rather than relying only on absolute coordinates. This encourages relational representations that can transfer across different camera viewpoints and scene layouts.

3D grounding

  • Qwen3-VL also introduces dedicated pre-training for monocular 3D grounding. Each example contains a single camera image, a natural-language referring expression, and a corresponding \(9\text{-DoF}\) 3D bounding-box annotation represented in structured JSON.

  • Because the underlying datasets come from different sensors and camera systems, the raw annotations contain inconsistent camera intrinsics and noise. Qwen3-VL filters heavily occluded or inaccurate examples and follows the coordinate-unification strategy of Omni3D: A Large Benchmark and Model for 3D Object Detection in the Wild by Brazil et al. (2023), which provides a framework for combining diverse 3D detection datasets.

  • The result maps heterogeneous data into a common virtual-camera coordinate system. Rich referring expressions additionally describe object attributes, scene arrangement, location, affordances, and interactions with nearby objects.

Code and video

  • This section brings together multimodal coding, video pre-training, and spatio-temporal grounding. The following subsections explain how these elements fit into the broader chapter.

Multimodal coding

  • Qwen3-VL preserves the text-only coding corpus used by Qwen3 and Qwen3-Coder while adding tasks in which visual understanding must lead to executable or structured output.

  • The multimodal coding mixture includes UI screenshot to HTML/CSS, image to editable SVG, visual programming problems, image-bearing coding questions, and diagram or equation transcription into code or markup.

  • For screenshot-to-code, for example,

    \[I_{\mathrm{UI}}\rightarrow c_{\mathrm{HTML/CSS}}\rightarrow\hat{I}_{\mathrm{UI}}\]

  • The generated program can potentially be evaluated through the visual similarity between the original interface and the rendered reconstruction.

Video pre-training

  • Video introduces both spatial and temporal complexity, and Qwen3-VL’s video data pipeline is designed around temporal density rather than treating a video as an unordered collection of images.

  • Long-video supervision uses a short-to-long caption synthesis strategy. Local observations are combined into holistic, timestamp-interleaved descriptions, while additional fine-grained annotations capture event-level summaries and segment-specific visual details.

  • The representation can be conceptualized as

    \[V=\{(\tau_1,c_1),(\tau_2,c_2),\ldots,(\tau_n,c_n)\}\]
    • where \(\tau_i\) denotes time and \(c_i\) describes the corresponding event or visual segment. This directly complements Qwen3-VL’s explicit textual timestamp architecture.

Spatio-temporal grounding

  • The video corpus also contains object-, action-, and person-level temporal grounding. Instead of answering only what occurs in a video, the model can be trained to determine when an event occurs or where an object appears during a specified interval.

  • This couples spatial localization with temporal localization. The intended representation is therefore

    \[\text{entity}+\text{space}+\text{time}\]

  • This data supports downstream capabilities such as event localization, dense video captioning, and long-video question answering.

Video-source balancing

  • The video corpus combines multiple source categories, including instructional content, cinematic material, and egocentric recordings. Sampling is controlled using metadata such as title, duration, and category.

  • The desired training mixture can be expressed as:

    \[p(V)=\sum_k\alpha_kp(V\mid c_k)\]
    • where \(c_k\) denotes a video category and \(\alpha_k\) controls its contribution.

Length-adaptive video sampling

  • Video sampling also changes according to the sequence-length stage. The model dynamically adjusts quantities such as frames per second and maximum frame count according to the available token budget.

  • Let \(F\) be the number of sampled frames and \(N_v\) the average number of visual tokens per frame. The visual contribution to sequence length is approximately \(L_v\propto FN_v\).

  • Given total context budget \(L_{\max}\), sampling must satisfy approximately:

    \[L_{\mathrm{text}}+L_v+L_{\mathrm{timestamp}}\leq L_{\max}\]

  • Sparse sampling risks missing short events. Excessive frame sampling may force spatial resolution downward or exceed the context budget. Qwen3-VL therefore adapts temporal sampling to the context length available during each training stage.

STEM and data integration

  • This section brings together sTEM perception data, multimodal STEM reasoning, and why multimodal data diversity matters. The following subsections explain how these elements fit into the broader chapter.

STEM perception data

  • STEM reasoning receives dedicated visual supervision. Qwen3-VL programmatically renders geometric diagrams and uses them to create approximately \(1\mathrm{M}\) point-grounding examples involving features such as intersections, corners, and centers of gravity.

  • It additionally creates approximately \(2\mathrm{M}\) perception-oriented visual question-answer pairs. A separate two-stage captioning and verification pipeline produces approximately \(6\mathrm{M}\) richly annotated STEM diagram captions.

  • Programmatic generation is particularly useful for geometry because the underlying scene parameters are known exactly. If a renderer creates a point \(P=(x,y)\), its location is available directly from the generation process rather than inferred through noisy human annotation.

Multimodal STEM reasoning

  • Perception alone is insufficient for visual mathematics. Qwen3-VL therefore combines fine-grained diagram understanding with large-scale reasoning data.

  • The report describes more than \(60\mathrm{M}\) K-12 and undergraduate-level multimodal exercises. These examples undergo cleaning to remove corrupted images, irrelevant material, incomplete problems, and incorrect solutions. Chinese and English versions are produced where appropriate, and answer representations are standardized.

  • Qwen3-VL additionally synthesizes more than \(12\mathrm{M}\) long-chain-of-thought multimodal reasoning examples paired with images.

  • The intended training progression is

    \[\text{see the diagram}\rightarrow\text{identify relevant visual structure}\rightarrow\text{reason over that structure}\rightarrow\text{derive the answer}\]

  • This separates the perceptual and reasoning components during data construction while training the final model to integrate them.

Why multimodal data diversity matters

  • The Qwen3-VL corpus is not organized around a single generic image-text objective. Each major capability receives data whose output structure reflects the target behavior.

  • Captioning learns:

    \[I\rightarrow T\]

  • OCR learns:

    \[I_{\mathrm{text}}\rightarrow\text{transcription}\]

  • Grounding learns:

    \[(I,T)\rightarrow(x,y)\]

    • or:

      \[(I,T)\rightarrow(x_1,y_1,x_2,y_2)\]

  • Document parsing learns:

    \[I_{\mathrm{doc}}\rightarrow\text{structured document}\]
Interactions across data sources

  • The following material develops the implementation and interpretation of why multimodal data diversity matters. It builds on the preceding discussion and makes the remaining details easier to follow.

  • 3D grounding learns:

    \[(I,T)\rightarrow B_{\mathrm{3D}}\]

  • Multimodal coding learns:

    \[I\rightarrow\text{executable representation}\]

  • Video grounding learns:

    \[(V,T)\rightarrow(\tau,\text{location})\]

  • Long-document VQA learns:

    \[(D_{1:n},q)\rightarrow a\]

  • Multimodal STEM learns:

    \[(I,q)\rightarrow r\rightarrow a\]

  • The resulting foundation model therefore receives supervision over a much richer output space than ordinary image captioning.

The role of synthetic data

  • Synthetic data appears repeatedly throughout Qwen3-VL’s pre-training pipeline. It is used for multilingual OCR, grounding, HTML documents, visual knowledge descriptions, geometric diagrams, multimodal reasoning, and other tasks.

  • The general pipeline can be written as:

    \[x\xrightarrow{\text{generator}}\tilde{y}\xrightarrow{\text{verifier/filter}}y^\star\]

  • This creates a recurring pattern:

    \[\boxed{\text{large-scale synthesis}+\text{task-specific verification}}\]
    • which allows capability-specific datasets to grow beyond the scale practical for manual annotation.

Preserving text capability during multimodal training

  • A central constraint throughout the four-stage curriculum is that multimodal training should not destroy the Qwen3 backbone’s language capability.

  • First, text-only data remains in the mixture during full-parameter multimodal training:

    \[\mathcal{D}=\alpha\mathcal{D}_{\mathrm{text}}+(1-\alpha)\mathcal{D}_{\mathrm{VL}}\]

  • Second, the square-root-normalized loss attempts to balance examples with very different sequence characteristics.

  • The training objective is therefore not \(\max Q_{\mathrm{vision}}\) in isolation. It is closer to:

    \[\max f(Q_{\mathrm{text}},Q_{\mathrm{vision}},Q_{\mathrm{video}},Q_{\mathrm{reasoning}},Q_{\mathrm{long-context}})\]

  • This objective explains why Qwen3-VL’s data mixture continues to contain substantial text-only data even after visual alignment has been established.

Pre-training as progressive capability integration

  • The complete pre-training pipeline can be summarized as a sequence of increasingly difficult integration problems:

    \[\boxed{ \begin{aligned} \text{S0: }&\text{vision}\leftrightarrow\text{language alignment}\\[4pt] \text{S1: }&\text{text + images + grounding + STEM + initial video}\\[4pt] \text{S2: }&\text{longer text + more video + agentic data}\\[4pt] \text{S3: }&\text{256K long documents + long video}. \end{aligned} }\]

Qwen3-VL Post-training, Multimodal Reasoning, and Visual Agents

  • This chapter traces Qwen3-VL from supervised multimodal adaptation through distillation and reinforcement learning. It then examines how visual reasoning is extended into tool-mediated interaction.

Training stages

  • This section brings together post-training overview, supervised fine-tuning, and instruct and thinking variants. The following subsections explain how these elements fit into the broader chapter.

Post-training overview

  • Qwen3-VL’s post-training pipeline transforms the multimodal base model into instruction-following and reasoning-oriented variants while preserving the broad visual capabilities acquired during pre-training. Qwen3-VL Technical Report by Bai et al. (2025) organizes the main pipeline into three stages:

    \[\boxed{\text{Supervised Fine-Tuning}\rightarrow\text{Strong-to-Weak Distillation}\rightarrow\text{Reinforcement Learning}}\]

  • The reinforcement-learning stage is itself divided into Reasoning RL and General RL. In parallel, Qwen3-VL introduces a specialized visual-agent pipeline called “Thinking with Images,” which teaches the model to interact with visual tools through multi-turn trajectories rather than treating the original image as immutable context.

Supervised fine-tuning

  • The first post-training stage establishes instruction-following behavior and activates reasoning capabilities latent in the pretrained model. Qwen3-VL performs SFT first at a 32K context length and then at 256K, with the latter phase emphasizing long documents and long videos.

    \[\mathrm{SFT}_{32K}\rightarrow\mathrm{SFT}_{256K}\]

Instruct and thinking variants

  • Qwen3-VL separates its SFT data into standard and Chain-of-Thought formats to produce non-thinking and thinking models. The non-thinking target is approximately \(x\rightarrow y\), whereas the thinking target explicitly models an intermediate reasoning trajectory \(x\rightarrow r\rightarrow y\).

  • Here, the input may contain text, images, video, or interleaved multimodal context, while the reasoning trajectory precedes the final answer.

SFT data composition

  • The primary SFT corpus contains approximately 1.2 million examples. Roughly one third are text-only and two thirds contain image-text or video-text inputs. The corpus includes multilingual, single-turn, multi-turn, single-image, multi-image, video, and interleaved image-text examples.

    \[\mathcal{D}_{\mathrm{SFT}}= \mathcal{D}_{\mathrm{text}}\cup \mathcal{D}_{\mathrm{image}}\cup \mathcal{D}_{\mathrm{video}}\cup \mathcal{D}_{\mathrm{multi-image}}\cup \mathcal{D}_{\mathrm{agent}}\]

  • The multimodal tasks include spatial reasoning, fine-grained image-grounded reasoning, spatio-temporal video grounding, long-document comprehension, and agentic tool use.

Long-context SFT

  • The 256K phase contains examples whose semantics require long-context processing, including hundreds of pages of technical documents, complete textbooks, and videos reaching approximately two hours. The second SFT epoch interleaves these ultra-long examples with 32K samples rather than specializing exclusively to extremely long inputs.

SFT data filtering

  • Qwen3-VL applies both query and response filtering. Qwen2.5-VL identifies prompts that cannot be reliably verified, ambiguous instructions may be minimally clarified, and low-substance web-derived queries are removed. Responses are filtered using deterministic rules and learned evaluators for correctness, completeness, clarity, helpfulness, visual grounding, repetition, formatting, and language consistency.

Long-CoT cold start

  • Thinking models receive a curated Long Chain-of-Thought cold-start dataset with an approximately 1:1 ratio between text-only and vision-language examples. The multimodal portion includes VQA, OCR, 2D and 3D grounding, video analysis, STEM reasoning, and agentic workflows. The text component contains mathematics, coding, logical reasoning, and general STEM.

    \[\pi_{\mathrm{base}} \xrightarrow{\mathrm{Long\text{-}CoT\ SFT}} \pi_{\mathrm{reason}}\]

  • Examples are selected to emphasize genuine difficulty. For vision-language mathematics, Qwen3-VL also removes examples that a text-only Qwen3-30B-nothink model can solve correctly without seeing the image. This filters out nominally multimodal tasks whose visual input is unnecessary.

Distillation and reasoning RL

  • This section brings together strong-to-Weak Distillation, reasoning reinforcement learning, and mixed-task RL and rewards. The following subsections explain how these elements fit into the broader chapter.

Strong-to-Weak Distillation

  • After SFT, Qwen3-VL applies the Strong-to-Weak Distillation strategy introduced by Qwen3. Notably, this stage uses text-only data to fine-tune the LLM backbone, yet the Qwen3-VL report states that it improves reasoning on both text-centric and multimodal tasks.

  • Off-policy distillation trains students on teacher-generated responses:

    \[y_T\sim\pi_T(\cdot\mid x)\]

  • On-policy distillation instead samples from the student:

    \[y_S\sim\pi_S(\cdot\mid x)\]

    • and aligns teacher and student logits through KL divergence:

      \[\mathcal{L}_{\mathrm{KD}} = \sum_tD_{\mathrm{KL}} \left( \pi_T(\cdot\mid x,y_{S,<t}) \| \pi_S(\cdot\mid x,y_{S,<t}) \right)\]

  • Because the latter operates on contexts visited by the student itself, it reduces the train-inference mismatch of purely off-policy imitation.

Reasoning reinforcement learning

  • Reasoning RL spans mathematics, coding, logical reasoning, visual grounding, and visual puzzles. Tasks are selected so that answers can be verified deterministically using rules or code executors.

  • For multimodal queries, a preliminary Qwen3-VL-235B-A22B checkpoint generates 16 responses per query. Queries for which all responses fail are discarded. After additional source filtering, the RL corpus contains approximately 30K queries.

  • During training, 16 responses are again sampled for each query, and examples with pass rates above 90% are removed as too easy. The resulting curriculum concentrates optimization on problems that are neither effectively solved nor effectively impossible.

Mixed-task RL and rewards

  • Task-specific datasets are shuffled into mixed batches using predetermined task ratios. A unified reward infrastructure handles data preprocessing and reward aggregation while individual tasks retain task-specific verification logic.

  • Qwen3-VL uses task-specific format prompts instead of explicit format rewards and applies a penalty when the response language differs from the prompt language.

Soft Adaptive Policy Optimization

  • Qwen3-VL uses Soft Adaptive Policy Optimization, or SAPO, for Reasoning RL. Soft Adaptive Policy Optimization by Gao et al. (2025) introduces a policy-gradient approach using smooth adaptive gating rather than brittle hard clipping of off-policy updates.

  • For token-level importance ratio:

    \[r_t(\theta)= \frac{\pi_\theta(a_t\mid s_t)} {\pi_{\mathrm{old}}(a_t\mid s_t)}\]

  • SAPO conceptually applies a smooth transformation:

    \[r_t\rightarrow g_\tau(r_t)\]
    • where the temperature-controlled gate attenuates strongly off-policy updates while retaining useful gradient signal. The method combines sequence-level coherence with token-adaptive weighting.

General RL and visual agents

  • This section brings together general reinforcement learning, from multimodal reasoning to visual agency, and first-stage visual-agent bootstrapping. The following subsections explain how these elements fit into the broader chapter.

General reinforcement learning

  • General RL broadens the objective from verifiable reasoning to practical multimodal interaction. Its task mixture includes VQA, image captioning, OCR, document parsing, grounding, and clock recognition. The principal objectives are instruction following and preference alignment.

  • Feedback combines rule-based and model-based rewards:

    \[R(y)=R_{\mathrm{rule}}(y)+R_{\mathrm{model}}(y)\]

  • Rule-based rewards provide high-precision feedback when correctness is deterministically verifiable. Qwen2.5-VL-72B-Instruct or Qwen3 models serve as judges for more nuanced outputs where multiple responses may be valid.

  • General RL also acts as a corrective mechanism for flawed priors acquired during SFT. Specialized verifiable examples target errors such as counter-intuitive counting and difficult clock recognition. Dedicated datasets similarly increase the training frequency of rare undesirable behaviors such as language mixing, repetition, and formatting failures.

From multimodal reasoning to visual agency

  • Ordinary multimodal reasoning assumes fixed visual evidence:

    \[(I,q)\rightarrow r\rightarrow y\]

  • Qwen3-VL extends this paradigm through “Thinking with Images,” where the model can interact with visual tools during reasoning:

    \[\boxed{ \text{think}\rightarrow \text{act}\rightarrow \text{observe}\rightarrow \text{think}\rightarrow\cdots\rightarrow \text{answer} }\]

  • The broader research direction is discussed in Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontiers by Su et al. (2025).

First-stage visual-agent bootstrapping

  • Visual-agent training begins with approximately 10K synthetic grounding examples, primarily simple two-turn visual question-answering tasks. Qwen2.5-VL-32B is first fine-tuned to emulate the pattern

    \[\text{think}\rightarrow \text{act}\rightarrow \text{analyze feedback}\rightarrow \text{answer}\]

  • It is then trained with multi-turn, tool-integrated RL.

Distilling visual-agent trajectories

  • The trained Qwen2.5-VL-32B visual agent generates a larger dataset of approximately 120K multi-turn agentic interactions spanning a broader range of visual tasks. Qwen3-VL then undergoes its own cold-start SFT and tool-integrated RL using both distilled and synthesized trajectories.

  • This transfers not merely answers but policies for acquiring additional visual information.

Three rewards for visual agents

  • Tool-integrated RL uses three complementary reward signals: an answer-accuracy reward evaluated with Qwen3-32B, a multi-turn reasoning reward evaluated with Qwen2.5-VL-72B, and a tool-calling reward comparing actual tool usage with an expert-estimated target.

  • Conceptually,

    \[R(\tau)= \alpha R_{\mathrm{answer}}+ \beta R_{\mathrm{reason}}+ \gamma R_{\mathrm{tool}}\]

  • The report does not specify these weighting coefficients, so this equation is only a schematic representation of the multi-objective reward.

Reward hacking and adaptive visual exploration

  • Early experiments found that optimizing answer and reasoning rewards alone could lead agents to make only one tool call regardless of task complexity. Qwen3-VL therefore adds an explicit tool-calling reward to encourage an amount of visual exploration appropriate to the task.

  • If the actual number of calls is \(N\), and the expert-estimated target is \(N^\star\), the reward encourages the policy to keep its tool-use behavior aligned with task complexity rather than minimizing interaction indiscriminately.

  • This changes the objective from simply learning to use a visual tool to learning when and how extensively to use one.

Tool use and capability integration

  • This section brings together tools as inference-time visual compute, why tool use can outperform model scaling, and the complete post-training pipeline. The following subsections explain how these elements fit into the broader chapter.

Tools as inference-time visual compute

  • Thinking with Images introduces a multimodal analogue of the text reasoning budget.

  • Text reasoning compute grows with the number of reasoning tokens:

    \[C_{\mathrm{text}}\propto T_{\mathrm{reason}}\]

  • Visual-agent compute can additionally grow with tool interactions:

    \[C_{\mathrm{visual}}\propto N_{\mathrm{tool\ calls}}\]

  • The effective inference budget can therefore be viewed as

    \[C_{\mathrm{total}} = C_{\mathrm{language}} + C_{\mathrm{vision}} + C_{\mathrm{tools}}\]

  • A model can spend additional compute either by reasoning longer or by acquiring better visual evidence.

Why tool use can outperform model scaling

  • The Qwen3-VL evaluation indicates that active visual inspection can provide substantial gains on fine-grained perception. With tools enabled, Qwen3-VL-235B-A22B reports 93.7 on V, 85.3 on HRBench-4K, and 82.3 on HRBench-8K, while adding tools yields an approximately five-point absolute improvement on V across Qwen3-VL model sizes.

  • This suggests that some visual failures arise from insufficient access to fine-grained evidence rather than model capacity alone. A tool can alter the information available to the policy itself.

The complete post-training pipeline

  • Qwen3-VL’s post-training design can be summarized as:

    \[\boxed{ \begin{aligned} \text{Multimodal Base Model} &\rightarrow\text{SFT}\\ &\rightarrow\text{Long-CoT Reasoning}\\ &\rightarrow\text{Strong-to-Weak Distillation}\\ &\rightarrow\text{Reasoning RL with SAPO}\\ &\rightarrow\text{General RL}\\ &\rightarrow\text{tool-integrated visual agency}. \end{aligned} }\]

  • Each stage addresses a different limitation. SFT establishes instruction behavior. Long-CoT data bootstraps explicit reasoning. Strong-to-weak distillation transfers reasoning capability into smaller models. Reasoning RL improves verifiable text and multimodal problem solving. General RL broadens instruction following and preference alignment while correcting persistent failure modes. Thinking with Images teaches the model to alter its own visual information state through tool use.

  • The resulting model is trained not merely to map multimodal input to text \((x,I)\rightarrow y\) but, when appropriate, to execute a closed-loop process:

    \[\boxed{ (x,I)\rightarrow r_0\rightarrow a_0\rightarrow o_1\rightarrow r_1\rightarrow a_1\rightarrow \cdots\rightarrow y }\]

  • This closed-loop formulation connects Qwen3’s emphasis on reasoning and agents with Qwen3-VL’s visual perception stack, making multimodal post-training as much about learning when and how to acquire information as it is about learning how to reason over information already present.

Pre-training, mid-training, and post-training objectives and loss functions

  • Qwen3 Technical Report by Qwen Team (2025) and Qwen3-VL Technical Report by Qwen Team (2025) describe optimization across pre-training and post-training at different levels of specificity. The reports do not formally use “mid-training” as a separate top-level phase; here, mid-training is a useful conceptual label for the intermediate pre-training stages that deliberately shift the data distribution toward reasoning, multimodal capability, or longer context before supervised post-training begins. Across these phases, the core generative objective remains token prediction, while the trainable parameter set, data mixture, sequence length, normalization, and later reward or distillation signals change. The reports do not publish every implementation-level coefficient or complete loss formula, so the equations below distinguish direct report specifications from faithful mathematical abstractions of the stated training objectives.

Qwen3: General pre-training

  • Qwen3 begins with a General Stage that trains all models on more than 30 trillion tokens at a sequence length of 4,096, establishing broad language proficiency and world knowledge across 119 languages and dialects. Qwen3 Technical Report by Qwen Team (2025) describes the stage and its data composition but does not introduce a stage-specific alternative to causal language modeling. The objective can therefore be represented as standard next-token negative log-likelihood:

    \[\mathcal{L}_{\mathrm{Qwen3\text{-}S1}} = -\mathbb{E}_{x\sim\mathcal{D}_{\mathrm{general}}} \left[ \sum_{t=1}^{|x|} \underbrace{\log p_\theta(x_t\mid x_{<t})}_{\text{next-token log-probability}} \right]\]
    • where \(\mathcal{D}_{\mathrm{general}}\) is the broad multilingual pre-training mixture and \(\theta\) contains the trainable Qwen3 parameters. At this stage, the optimization target is not a separate reasoning loss or multilingual loss; those capabilities are induced through the composition and scale of the token-prediction data.

Qwen3: Reasoning-focused mid-training

  • Qwen3’s second pre-training stage can be viewed as reasoning-focused mid-training. The report calls it the Reasoning Stage and continues training on roughly 5 trillion higher-quality tokens at the same 4,096-token sequence length, while increasing the proportions of STEM, coding, reasoning, and synthetic data and accelerating learning-rate decay. The mathematical token-prediction objective remains the same, but the expectation is taken over a deliberately shifted distribution:

    \[\mathcal{L}_{\mathrm{Qwen3\text{-}S2}} = -\mathbb{E}_{x\sim \underbrace{\mathcal{D}_{\mathrm{reason}}}_{\text{STEM, coding, reasoning, and synthetic-heavy mixture}}} \left[ \sum_{t=1}^{|x|} \underbrace{\log p_\theta(x_t\mid x_{<t})}_{\text{causal language-model objective}} \right]\]

  • The important optimization change is thus distributional rather than a new explicit reasoning criterion:

    \[\underbrace{\mathcal{D}_{\mathrm{general}}}_{\text{broad capability acquisition}} \quad\longrightarrow\quad \underbrace{\mathcal{D}_{\mathrm{reason}}}_{\text{higher concentration of reasoning-relevant tokens}}\]

Qwen3: Long-context adaptation

  • In the final Qwen3 pre-training stage, the context length increases to 32,768 and training continues on hundreds of billions of long-context tokens. The corpus is dominated by long sequences, while RoPE configuration is adapted for the longer operating regime. The report again describes continued pre-training rather than a new task loss, so the objective is the same causal likelihood evaluated on long sequences:

    \[\mathcal{L}_{\mathrm{Qwen3\text{-}long}} = -\mathbb{E}_{x\sim\mathcal{D}_{\mathrm{long}}} \left[ \sum_{t=1}^{|x|} \underbrace{\log p_\theta(x_t\mid x_{<t})}_{\text{token prediction across extended context}} \right]\]
    • where \(\mathcal{D}_{\mathrm{long}}\) contains the long-context corpus. The objective itself does not explicitly reward retrieval over distant tokens; long-range behavior emerges from optimizing token prediction over sequences whose length distribution and positional treatment expose the model to those dependencies.

Qwen3-VL: Vision-Language Alignment

  • Qwen3-VL pre-training begins with Stage S0, Vision-Language Alignment. Only the MLP vision-language merger is updated, while the vision encoder and Qwen3 LLM backbone remain frozen. The stage uses approximately 67 billion tokens of image-caption, visual-knowledge, and OCR data at an 8,192-token sequence length. The conditional token-prediction objective can be written as:

    \[\mathcal{L}_{\mathrm{VL\text{-}S0}} = -\mathbb{E}_{(m,x)\sim\mathcal{D}_{\mathrm{align}}} \left[ \sum_{t=1}^{|x|} \underbrace{\log p_{\theta_M}(x_t\mid m,x_{<t})}_{\text{text-token likelihood conditioned on visual tokens}} \right], \qquad \underbrace{\theta_M\ \text{trainable}}_{\text{MLP merger}} \quad \underbrace{\theta_V,\theta_L\ \text{frozen}}_{\text{vision encoder and LLM}}\]
    • where \(m\) denotes the visual input, \(\theta_M\) the merger parameters, \(\theta_V\) the vision encoder, and \(\theta_L\) the language-model backbone. The defining feature of S0 is therefore parameter restriction: gradients update the modality bridge without modifying the two large pretrained endpoints.

Qwen3-VL: Multimodal mid-training

  • Stage S1 transitions to full-parameter multimodal pre-training on approximately 1 trillion tokens at an 8,192-token sequence length. All components are unfrozen, and the data mixture combines text-only examples with image-text documents, grounding, VQA, STEM, and some video. Conceptually, the same conditional language-model objective now propagates through the complete vision-language stack:

    \[\mathcal{L}_{\mathrm{VL\text{-}S1}} = -\mathbb{E}_{z\sim \underbrace{\mathcal{D}_{\mathrm{text}}\cup\mathcal{D}_{\mathrm{VL}}}_{\text{text-only and multimodal training mixture}}} \left[ \sum_{t} \underbrace{\log p_{\theta_V,\theta_M,\theta_L}(z_t\mid z_{<t})}_{\text{end-to-end token-prediction objective}} \right]\]

  • Qwen3-VL specifically changes loss aggregation relative to its predecessor by moving from a per-sample loss to a square-root-normalized per-token loss to better balance text-only and multimodal examples. The report states this normalization principle but does not publish a complete closed-form implementation. A faithful schematic representation is:

    \[\mathcal{L}_{\mathrm{sqrt}} \propto -\sum_{i=1}^{B} \underbrace{\frac{1}{\sqrt{n_i}}}_{\text{square-root length normalization}} \sum_{t=1}^{n_i} \underbrace{\log p_\theta(z_{i,t}\mid z_{i,<t})}_{\text{per-token negative log-likelihood contribution}}\]
    • where \(n_i\) is the number of predicted tokens associated with sample \(i\). The proportional form is intentional: Qwen3-VL Technical Report by Qwen Team (2025) specifies square-root-normalized per-token loss but does not provide the exact normalization denominator or implementation coefficients, so this equation captures the stated weighting principle rather than claiming an unpublished exact formula.

Qwen3-VL: Long-context and ultra-long-context mid-training

  • Stages S2 and S3 preserve full-parameter training while progressively extending context. S2 uses approximately 1 trillion tokens at 32,768 tokens per sequence, increases the text-only proportion, and adds substantially more video and agent-oriented instruction-following data. S3 then adapts the model to 262,144-token sequences using a focused 100-billion-token mixture emphasizing long video and long documents. The objective family remains square-root-normalized token prediction, but the training distribution and context regime change:

    \[\mathcal{L}_{\mathrm{VL\text{-}long}} \propto -\mathbb{E}_{z\sim \underbrace{\mathcal{D}_{\mathrm{long\text{-}VL}}}_{\text{long text, video, documents, and agent-oriented sequences}}} \left[ \sum_{i} \underbrace{\frac{1}{\sqrt{n_i}}}_{\text{square-root normalization}} \sum_{t=1}^{n_i} \underbrace{\log p_\theta(z_{i,t}\mid z_{i,<t})}_{\text{token prediction over long multimodal context}} \right]\]

  • The progression through Qwen3-VL pre-training can therefore be interpreted as changing which parameters and data regimes are optimized under a largely shared generative objective:

    \[\underbrace{\text{S0: merger-only alignment}}_{\text{67B tokens, 8K}} \rightarrow \underbrace{\text{S1: full multimodal training}}_{\sim\text{1T tokens, 8K}} \rightarrow \underbrace{\text{S2: long-context training}}_{\sim\text{1T tokens, 32K}} \rightarrow \underbrace{\text{S3: ultra-long adaptation}}_{\text{100B tokens, 256K}}\]

Transition from pre-/mid-training to post-training

  • Across Qwen3 and Qwen3-VL, pre-training and the intermediate reasoning, multimodal, or long-context stages primarily alter the data distribution, context length, parameter scope, and loss normalization while retaining autoregressive token prediction as the core learning signal. Post-training changes the optimization regime more fundamentally: supervised stages condition on curated target responses, distillation introduces teacher supervision and KL matching, and reinforcement learning replaces fixed target imitation with verifier-, judge-, preference-, or trajectory-derived rewards. The following subsections make those post-training objectives explicit.

Qwen3: Long-CoT cold start and Thinking Mode Fusion

  • Qwen3’s Long-CoT Cold Start is supervised fine-tuning on a deliberately small, heavily filtered set of long reasoning trajectories. The later Thinking Mode Fusion stage is also continual SFT, but mixes thinking and non-thinking responses so that a single checkpoint learns both behaviors. Qwen3 Technical Report by Qwen Team (2025) describes both stages as supervised training, although it does not state a separate closed-form loss for them. Their token-level objective can therefore be represented by the standard conditional autoregressive negative log-likelihood:

    \[\mathcal{L}_{\mathrm{SFT}} = -\mathbb{E}_{(x,y)\sim\mathcal{D}} \left[ \sum_{t=1}^{|y|} \underbrace{\log \pi_\theta(y_t\mid x,y_{<t})}_{\text{log-probability of the curated target token}} \right]\]
    • where \(x\) is the prompt, \(y\) is the curated target response, and \(\pi_\theta\) is the model being optimized. In Long-CoT Cold Start, \(y\) contains the selected reasoning trajectory and answer. In Thinking Mode Fusion, the dataset instead contains both thinking and non-thinking examples, including the mode-control conventions used by the chat template.

Qwen3: Reasoning RL

  • Qwen3 next applies Group Relative Policy Optimization (GRPO) to 3,995 query-verifier pairs. The report emphasizes verifiable problems, many rollouts per query, large batches, off-policy training for sample efficiency, and entropy control. Qwen3 Technical Report by Qwen Team (2025) identifies GRPO as the optimization algorithm but does not reproduce its complete mathematical objective. At the level needed to understand this stage:

    \[\mathcal{J}_{\mathrm{reason}} = \mathbb{E}_{x,\{y_i\}_{i=1}^{G}} \left[ \frac{1}{G}\sum_{i=1}^{G} \underbrace{A_i}_{\text{group-relative advantage}} \underbrace{\log \pi_\theta(y_i\mid x)}_{\text{policy log-probability of reasoning trajectory}} \right]\] \[A_i = \underbrace{\frac{r_i-\mu_{\mathbf r}}{\sigma_{\mathbf r}+\epsilon}}_{\text{relative quality of rollout }i\text{ within the sampled group}}\]

  • This abstraction captures the central signal: multiple candidate reasoning trajectories receive verifiable outcome feedback, and relatively successful trajectories are reinforced.

Qwen3: General RL

  • Qwen3’s final flagship stage uses more than 20 task-specific reward settings spanning instruction following, format following, preference alignment, agent behavior, and specialized scenarios such as RAG. The report specifies rule-based rewards, model-based rewards with reference answers, and model-based rewards without references. A useful decomposition is:

    \[R(x,y) = \underbrace{\lambda_{\mathrm{rule}}R_{\mathrm{rule}}(x,y)}_{\text{verifiable correctness and format}} + \underbrace{\lambda_{\mathrm{ref}}R_{\mathrm{judge}}(x,y,y^\star)}_{\text{model-based score with reference}} + \underbrace{\lambda_{\mathrm{pref}}R_{\mathrm{pref}}(x,y)}_{\text{reference-free preference score}}\]

  • The \(\lambda\) terms are conceptual task-dependent weights, not published Qwen3 coefficients. The corresponding objective is to maximize expected task reward:

    \[\mathcal{J}_{\mathrm{general}} = \mathbb{E}_{x\sim\mathcal{D},\,y\sim\pi_\theta(\cdot\mid x)} \left[ \underbrace{R(x,y)}_{\text{task-specific composite feedback}} \right]\]

Qwen3: Strong-to-Weak Distillation

  • Off-policy distillation uses teacher-generated /think and /no_think responses and can be represented as response-level supervised imitation:

    \[\mathcal{L}_{\mathrm{off}} = -\mathbb{E}_{x,\,y_T\sim\pi_T} \left[ \sum_t \underbrace{\log\pi_S(y_{T,t}\mid x,y_{T,<t})}_{\text{student likelihood of teacher-generated token}} \right]\]

  • In on-policy distillation, the student generates trajectories and the teacher supplies token distributions on those student states. The report explicitly describes teacher-student logit alignment:

    \[\mathcal{L}_{\mathrm{on}} = \mathbb{E}_{x,\,y_S\sim\pi_S} \left[ \sum_t \underbrace{ D_{\mathrm{KL}}\left(\pi_T(\cdot\mid x,y_{S,<t})\,\Vert\,\pi_S(\cdot\mid x,y_{S,<t})\right) }_{\text{teacher-student logit alignment on student-generated states}} \right]\]

Qwen3-VL: SFT and Long-CoT cold start

  • Qwen3-VL begins post-training with SFT, first at 32K context and then at 256K, using standard response formats for non-thinking models and CoT-formatted targets for thinking models. Its Long-CoT cold-start data combines text-only and multimodal reasoning examples at approximately a 1:1 ratio. Qwen3-VL Technical Report by Qwen Team (2025) does not provide a separate explicit SFT equation, so the objective can be represented as:

    \[\mathcal{L}_{\mathrm{VL\text{-}SFT}} = -\mathbb{E}_{(m,x,y)\sim\mathcal{D}_{\mathrm{VL}}} \left[ \sum_t \underbrace{\log\pi_\theta(y_t\mid m,x,y_{<t})}_{\text{supervised response-token likelihood conditioned on multimodal context}} \right]\]
    • where \(m\) denotes images, video, or other visual context and is absent for text-only examples.

Qwen3-VL: Strong-to-Weak Distillation

  • Qwen3-VL adopts Qwen3’s two-phase strategy: teacher-response distillation followed by on-policy student sequences with KL logit matching:

    \[\mathcal{L}_{\mathrm{VL\text{-}distill}} = \underbrace{\mathcal{L}_{\mathrm{off}}}_{\text{teacher-response imitation}} + \underbrace{ \lambda_{\mathrm{KL}} \mathbb{E}_{x,\,y_S\sim\pi_S} \left[\sum_t D_{\mathrm{KL}}\left(\pi_T(\cdot\mid x,y_{S,<t})\,\Vert\,\pi_S(\cdot\mid x,y_{S,<t})\right)\right] }_{\text{on-policy teacher-student logit alignment}}\]

  • The coefficient is conceptual: the report presents these as sequential phases rather than publishing a joint weighted objective.

Qwen3-VL: Reasoning RL with SAPO

  • Qwen3-VL Reasoning RL covers mathematics, coding, logical reasoning, visual grounding, and visual puzzles. Approximately 30K RL queries remain after curation; 16 responses are sampled per query; easy queries above 90% pass rate are filtered; and a language-mismatch penalty discourages code-switching. The report uses Soft Adaptive Policy Optimization (SAPO).

    \[R_{\mathrm{reason}}(x,y) = \underbrace{R_{\mathrm{verify}}(x,y)}_{\text{task-specific deterministic correctness}} - \underbrace{\lambda_{\mathrm{lang}}\mathbf{1}[\operatorname{lang}(y)\neq\operatorname{lang}(x)]}_{\text{response-language mismatch penalty}}\]

  • The Qwen3-VL report names SAPO but does not reproduce its full equation or hyperparameters. A deliberately abstract representation is therefore:

    \[\mathcal{J}_{\mathrm{SAPO}} = \mathbb{E} \left[ \underbrace{w_{\mathrm{SAPO}}(\rho_t,A_t)}_{\text{smooth adaptive policy weighting}} \underbrace{A_t}_{\text{reward-derived advantage}} \right], \qquad \underbrace{\rho_t=\frac{\pi_\theta(y_t\mid s_t)}{\pi_{\mathrm{old}}(y_t\mid s_t)}}_{\text{policy probability ratio}}\]

  • This notation captures the role of SAPO without inventing coefficients not reported by Qwen3-VL.

Qwen3-VL: General RL

  • General RL broadens the reward landscape to VQA, captioning, OCR, document parsing, grounding, clock recognition, instruction following, and preference alignment. It combines rule-based feedback with model-based judging and targeted penalties:

    \[R_{\mathrm{VL\text{-}general}}(x,m,y) = \underbrace{\lambda_{\mathrm{rule}}R_{\mathrm{rule}}(x,m,y)}_{\text{verifiable task and instruction feedback}} + \underbrace{\lambda_{\mathrm{judge}}R_{\mathrm{judge}}(x,m,y,y^\star)}_{\text{model-based quality judgment}} - \underbrace{\lambda_{\mathrm{bad}}P_{\mathrm{bad}}(y)}_{\text{language mixing, repetition, and formatting penalties}}\]

  • The coefficients are conceptual because the report does not publish their numerical values.

Qwen3-VL: Thinking with Images

  • Thinking with Images adds a trajectory-level agentic objective. Tool-integrated RL combines final-answer accuracy judged by Qwen3-32B, multi-turn reasoning quality judged by Qwen2.5-VL-72B, and a tool-calling reward based on actual versus expert-estimated tool usage:

    \[R_{\mathrm{agent}}(\tau) = \underbrace{\lambda_{\mathrm{ans}}R_{\mathrm{ans}}(\tau)}_{\text{final-answer accuracy}} + \underbrace{\lambda_{\mathrm{reason}}R_{\mathrm{reason}}(\tau)}_{\text{multi-turn reasoning quality}} + \underbrace{\lambda_{\mathrm{tool}}R_{\mathrm{tool}}(\tau)}_{\text{appropriate tool-call behavior}}\]
    • where \(\tau\) is the complete multi-turn trajectory. The report specifies the three signals but not their scalar combination weights.

  • Across both families, the progression is:

    \[\boxed{ \underbrace{\text{token likelihood}}_{\text{SFT / cold start}} \rightarrow \underbrace{\text{relative or adaptive policy optimization}}_{\text{reasoning RL}} \rightarrow \underbrace{\text{multi-source reward maximization}}_{\text{general RL}} }\]
    • with strong-to-weak distillation adding a parallel path based on teacher-response imitation followed by on-policy logit alignment.

Qwen3-VL Evaluation, Capabilities, and Closing Perspective

  • This chapter evaluates Qwen3-VL across perception, reasoning, grounding, video, and agent tasks. The closing sections connect those results to the architectural and training decisions discussed earlier.

Evaluation scope

  • Qwen3-VL is evaluated as a general multimodal foundation model rather than primarily as an image question-answering system. The evaluation suite spans general visual understanding, multimodal STEM reasoning, alignment, OCR and documents, 2D and 3D grounding, fine-grained perception, multi-image reasoning, embodied and spatial understanding, video, GUI agents, and text-only capabilities.

  • This breadth reflects the model’s intended capability stack:

    \[\boxed{ \text{perception} + \text{reasoning} + \text{long context} + \text{grounding} + \text{action} }\]

  • The following figure (source) summarizes Qwen3-VL’s performance across representative multimodal, reasoning, coding, document, video, and agentic benchmarks, comparing the flagship Instruct and Thinking variants with contemporary frontier models.

Perception and reasoning

  • This section brings together general visual understanding, multimodal reasoning, and alignment and multimodal hallucination. The following subsections explain how these elements fit into the broader chapter.

General visual understanding

  • General visual understanding is evaluated using benchmarks including MMBench, MMStar, and RealWorldQA. These tasks test combinations of recognition, commonsense reasoning, image-text alignment, and real-world visual understanding.

  • The flagship Qwen3-VL-235B-A22B-Thinking reaches 78.7 on MMStar, while Qwen3-VL-235B-A22B-Instruct reports 89.3/88.9 on the English and Chinese MMBench evaluations and 79.2 on RealWorldQA.

  • An important result is that thinking does not universally improve visual tasks. For example, the report notes cases where an Instruct variant matches or exceeds its Thinking counterpart on comparatively perception-heavy benchmarks.

  • This parallels the text-only Qwen3 finding that additional reasoning computation is task dependent:

    \[\Delta Q_{\mathrm{thinking}} = Q_{\mathrm{thinking}} - Q_{\mathrm{instruct}}\]
    • is not guaranteed to be positive for every task.

Multimodal reasoning

  • Reasoning evaluation includes MMMU, MMMU-Pro, MathVista, MathVision, MathVerse, We-Math, DynaMath, LogicVista, VisualPuzzles, VLMs Are Blind, ZeroBench, and VisuLogic.

  • MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI by Yue et al. (2023) evaluates expert-level multimodal reasoning across academic disciplines, while MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts by Lu et al. (2023) focuses specifically on mathematical reasoning grounded in visual information.

  • These tasks often require a pipeline such as:

    \[I \rightarrow z_{\mathrm{visual}} \rightarrow r_1 \rightarrow r_2 \rightarrow \cdots \rightarrow y\]

  • The visual representation is therefore only the starting point. Errors can originate from either perception or reasoning:

    \[P(\mathrm{error}) = f( P_{\mathrm{perception}}, P_{\mathrm{reasoning}}, P_{\mathrm{integration}} )\]

  • The report shows that Thinking variants are particularly useful on benchmarks requiring multi-step reasoning, although the magnitude of the gain depends strongly on the task. Qwen3-VL-235B-A22B-Thinking is reported to perform especially strongly on MathVista, MathVision, MathVerse, ZeroBench, LogicVista, and VisuLogic.

Alignment and multimodal hallucination

  • Qwen3-VL evaluates multimodal alignment using MM-MT-Bench, HallusionBench, and MIA-Bench.

  • HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models by Guan et al. (2023) is particularly important because it tests whether a model’s linguistic priors override the visual evidence.

  • A desirable multimodal policy should approximate \(P(y\mid I,q)\) rather than silently collapsing toward \(P(y\mid q)\) when the image contradicts an expected textual prior.

  • This connects directly to Qwen3-VL’s training-time filtering of examples that can be solved without their images. Both mechanisms target the same fundamental problem: ensuring that vision is causally relevant to the answer rather than decorative context.

Text-rich and spatial vision

  • This section brings together oCR and document understanding, multilingual visual text, and 2D grounding. The following subsections explain how these elements fit into the broader chapter.

OCR and document understanding

  • OCR and document understanding are among Qwen3-VL’s most heavily developed capabilities.

  • Evaluation includes OCRBench, DocVQA, InfoVQA, AI2D, ChartQA, CharXiv, and MMLongBench-Doc. These benchmarks collectively test text recognition, document layout, charts and diagrams, structured visual reasoning, and long-document retrieval.

  • On MMLongBench-Doc, the flagship reports 57.0% for the Instruct model and 56.2% for the Thinking model.

  • The slight advantage of the Instruct model is again instructive. Document understanding often contains a substantial retrieval component:

    \[\text{document} \rightarrow \text{locate evidence} \rightarrow \text{extract} \rightarrow \text{answer}\]

  • Longer reasoning does not automatically improve the first two operations.

  • By contrast, the report observes a stronger benefit from Thinking on the reasoning subset of CharXiv, where chart understanding must be followed by multi-step inference.

  • This suggests a useful distinction:

    \[\text{visual retrieval} \neq \text{visual reasoning}\]

Multilingual visual text

  • Qwen3-VL expands OCR support substantially beyond Qwen2.5-VL. The report evaluates 39 languages on an internal multilingual OCR benchmark and reports accuracy above 70% for 32 of them.

  • The multilingual OCR figure introduced in the previous section provides the corresponding per-language breakdown and is not repeated here.

  • Multilingual visual understanding is harder than text-only multilinguality because it composes two uncertain transformations:

    \[\text{pixels} \rightarrow \text{script recognition} \rightarrow \text{language understanding}\]

  • Consequently, improved multilingual language modeling alone cannot compensate for weak visual recognition of a script.

2D grounding

  • Grounding asks the model to convert linguistic references into explicit spatial predictions.

  • For 2D grounding, \((I,q) \rightarrow (x_1,y_1,x_2,y_2)\) or \((I,q) \rightarrow (x,y)\).

  • Qwen3-VL evaluates this capability across open-vocabulary grounding and counting tasks. On ODinW-13, the flagship reports 48.6 mAP when confidence scores are fixed to 1.0 and all dataset categories are supplied in the prompt.

3D grounding

  • Qwen3-VL also evaluates monocular 3D localization using Omni3D.

  • Omni3D: A Large Benchmark and Model for 3D Object Detection in the Wild by Brazil et al. (2023) combines diverse indoor and outdoor 3D datasets into a unified benchmark.

  • Qwen3-VL receives an image plus a textual object category and predicts a structured 3D bounding box. Evaluation uses \(\mathrm{mAP}@0.15\) with detection confidence fixed at 1.0.

  • The task requires the model to infer:

    \[\text{2D appearance} \rightarrow \text{3D position + extent + orientation}\]

  • Unlike ordinary 2D grounding, the correct output cannot be recovered directly from image-plane coordinates. It requires an implicit model of scene geometry.

Fine-grained perception and visual tools

  • Fine-grained perception is evaluated using V* and HRBench.

  • The flagship Qwen3-VL-235B-A22B with tools reports 93.7 on V*, 85.3 on HRBench-4K, and 82.3 on HRBench-8K.

  • More importantly, the report observes an approximately five-point improvement on V* from enabling tools across Qwen3-VL model sizes.

  • This provides an empirical counterpart to the visual-agent formulation developed during post-training:

    \[\text{static perception} \rightarrow \text{active perception}\]

  • A model facing insufficient visual evidence has at least two possible scaling axes: (i) larger model or (ii) better observation.

  • The reported experiments show that improving the observation through tool use can provide larger gains than increasing model size on some fine-grained tasks.

Multi-image and temporal understanding

  • This section brings together multi-image reasoning, spatial and embodied understanding, and video understanding. The following subsections explain how these elements fit into the broader chapter.

Multi-image reasoning

  • Multi-image tasks require the model to maintain entities and relationships across separate visual inputs.

  • Qwen3-VL is evaluated on BLINK and MuirBench, covering visual correspondence and reasoning across images. The flagship Thinking model reports 80.1 on MuirBench.

  • The underlying problem differs from ordinary image understanding because evidence is distributed:

    \[E = \{I_1,I_2,\ldots,I_n\}\]

  • The answer may depend on a relation R(I_i,I_j) rather than any individual image.

  • This capability connects naturally to Qwen3-VL’s interleaved multimodal context, where visual inputs are treated as components of a shared sequence rather than independent queries.

Spatial and embodied understanding

  • Spatial evaluation includes ERQA, VSIBench, EmbSpatial, RefSpatial, and RoboSpatialHome.

  • The flagship reports 84.3 on EmbSpatial, 69.9 on RefSpatial, 73.9 on RoboSpatialHome, 52.5 on ERQA, and 60.0 on VSIBench.

  • These tasks test a transition from visual semantics toward physical reasoning:

    \[\text{What is visible?} \rightarrow \text{Where is it?} \rightarrow \text{How are objects related?} \rightarrow \text{What action is possible?}\]

  • The training mixture of pointing, grounding, relative-position annotations, affordances, and spatio-temporal perception is designed specifically to support this transition.

Video understanding

  • Video adds temporal state to multimodal reasoning: \(I \rightarrow I(t)\)

  • Qwen3-VL’s video stack combines Interleaved MRoPE, explicit textual timestamps, temporally dense captions, long-context training, and adaptive frame sampling.

  • The report notes that the 8B Qwen3-VL model reaches performance competitive with the much larger Qwen2.5-VL-72B on its video evaluations, illustrating the effect of architectural and data improvements independently of parameter scaling.

  • For the flagship evaluation, videos are capped at 2048 frames and 224K video tokens.

  • VideoMMMU and MMVU permit up to 768 tokens per frame, while the other reported video benchmarks use up to 640.

  • Sampling is generally performed at 2 fps with Charades-STA evaluated at 4 fps.

  • These details matter because video benchmark performance depends not only on model quality but also on how much of the video the model is permitted to observe.

Long-video context

  • Long-video evaluation tests whether the model can preserve and retrieve information across much larger temporal spans.

  • Qwen3-VL’s native training context reaches 256K but the report additionally studies extrapolation beyond this regime using a video Needle-in-a-Haystack experiment.

  • The following figure (source) shows Needle-in-a-Haystack accuracy for Qwen3-VL-235B-A22B-Instruct as a salient frame is inserted at different depths in videos ranging from short within-training-context sequences to videos extending to approximately two hours.

  • The experiment tests whether a model can recover one relevant visual observation from a long temporal context:

    \[V = [f_1,\ldots,f_{i-1},f_{\mathrm{needle}},f_{i+1},\ldots,f_n]\]

  • Success requires retaining enough information about

    \[f_{\mathrm{needle}}\]
    • to answer a later query even when large quantities of irrelevant video surround it.

  • The distinction between trained and extrapolated context is important. Strong results beyond 256K indicate useful extrapolation behavior, but should not be interpreted as equivalent to native training at every longer context length.

Agents and text preservation

  • This section connects GUI agents with retaining text capabilities. The following subsections develop these related topics and their implications.

GUI agents

  • Qwen3-VL evaluates GUI perception on ScreenSpot, ScreenSpot Pro, and OSWorld-G, and interactive decision making on AndroidWorld and OSWorld.

  • A GUI agent must convert language and pixels into actions:

    \[(\text{instruction},I_t) \rightarrow a_t\]

  • After executing the action, the environment changes:

    \[s_{t+1} = T(s_t,a_t)\]
    • and the agent must perceive the resulting interface before choosing another action.

  • This produces a closed loop:

    \[\boxed{ \text{observe} \rightarrow \text{reason} \rightarrow \text{act} \rightarrow \text{observe} }\]
    • which closely matches the visual-agent post-training framework.

  • The report gives Qwen3-VL-32B scores of 41 on OSWorld and 63.7 on AndroidWorld.

  • These evaluations are especially significant because success depends jointly on perception, grounding, planning, action selection, and recovery from environmental feedback.

Retaining text capabilities

  • A multimodal model should ideally gain vision without sacrificing the language model on which it is built.

  • Qwen3-VL therefore evaluates text-only capabilities across knowledge, mathematics, reasoning, alignment, coding, agents, and multilingual benchmarks.

  • The report compares the multimodal models against corresponding Qwen3 language models and other text-focused systems. Its results indicate that Qwen3-VL retains strong text capabilities and, in several reasoning-oriented evaluations, can improve over its text-only predecessor.

  • This is an important systems result because multimodal training optimizes a much broader objective:

    \[\mathcal{L} = \mathcal{L}_{\mathrm{text}} + \mathcal{L}_{\mathrm{vision}} + \mathcal{L}_{\mathrm{video}} + \mathcal{L}_{\mathrm{agent}} +\cdots\]

  • Without careful data balancing and post-training, gradients from the new modalities could interfere with capabilities already present in the language backbone.

  • The continued inclusion of text data, square-root loss reweighting, text-only reasoning distillation, and mixed post-training all contribute to mitigating this problem.

Ablations and synthesis

  • This section brings together what the DeepStack ablation shows, interpreting the benchmark results carefully, and what Qwen3-VL adds to Qwen3. The following subsections explain how these elements fit into the broader chapter.

What the DeepStack ablation shows

  • Qwen3-VL provides a controlled ablation of DeepStack using an internal 15B-A2B model pretrained on 200B tokens.

  • Without DeepStack, average performance across the reported benchmark suite is 74.7.

  • Adding DeepStack increases this to 76.0.

  • The gains are particularly visible on visually detailed tasks. InfoVQA rises from \(71.9\rightarrow74.2\).

Interpreting the ablation results

  • The following material develops the implementation and interpretation of what the DeepStack ablation shows. It builds on the preceding discussion and makes the remaining details easier to follow.

  • DocVQA from \(89.5\rightarrow91.1\) OCRBench from \(81.0\rightarrow83.6\) and ChartQA from \(81.5\rightarrow83.3\).

  • This supports the architectural hypothesis developed earlier: injecting representations from multiple vision-encoder depths preserves visual information that would otherwise be compressed into only the final vision representation.

Interpreting the benchmark results carefully

  • The Qwen3-VL report compares its models against a mixture of open and proprietary systems, but these comparisons are not always perfectly controlled.

  • This is particularly explicit for video. API and resource constraints led to different maximum input-frame counts for some proprietary models. The report notes limits of 512 frames for Gemini 2.5 Pro, 256 for GPT-5, and 100 for Claude Opus 4.1, while Qwen3-VL’s own evaluation allows substantially more frames.

  • Consequently,

    \[Q_{\mathrm{observed}} = f( Q_{\mathrm{model}}, N_{\mathrm{frames}}, R_{\mathrm{image}}, L_{\mathrm{context}}, B_{\mathrm{thinking}}, D_{\mathrm{decoding}} )\]

  • Benchmark numbers should therefore be interpreted together with inference configuration rather than as architecture-independent constants.

What Qwen3-VL adds to Qwen3

  • The relationship between the two model families is best understood as an expansion of the state over which the model can reason.

  • Qwen3 operates primarily over linguistic state:

    \[s_t = \text{textual context}\]

  • Qwen3-VL expands this to:

    \[s_t = \{ \text{text}, \text{images}, \text{video}, \text{space}, \text{time}, \text{tool observations} \}\]

  • Dynamic-resolution vision preserves visual detail. DeepStack exposes multiple levels of that representation to the language model. Interleaved MRoPE represents spatial and temporal coordinates. Textual timestamps make physical video time explicit. Long-context training allows evidence to persist across documents and videos. Multimodal SFT and RL teach the decoder to reason over that evidence. Thinking with Images allows the model to acquire new evidence during inference.

  • The resulting progression is

    \[\boxed{ \text{language model} \rightarrow \text{multimodal reasoner} \rightarrow \text{visual agent} }\]

Closing perspective

  • Across Qwen3 and Qwen3-VL, the central architectural and training direction is not simply parameter scaling.

  • Qwen3 combines dense and sparse MoE architectures, a 36-trillion-token pre-training corpus, long-context adaptation, reasoning RL, thinking/non-thinking behavior, controllable inference-time compute, and strong-to-weak distillation.

  • Qwen3-VL extends the same philosophy into multimodal space. It adds dynamic-resolution visual encoding, multi-level DeepStack injection, Interleaved MRoPE, explicit video timestamps, 256K native multimodal context, large-scale OCR and grounding supervision, multimodal reasoning RL, and visual tool use.

  • The complete progression can be summarized as

    \[\boxed{ \begin{aligned} \text{large-scale pre-training} &\rightarrow \text{general representations} \\ &\rightarrow \text{reasoning post-training} \\ &\rightarrow \text{controllable test-time compute} \\ &\rightarrow \text{multimodal perception} \\ &\rightarrow \text{spatial and temporal reasoning} \\ &\rightarrow \text{tool-mediated agency}. \end{aligned} }\]

  • Viewed this way, Qwen3’s broader design principle is:

    \[\boxed{ \text{allocate computation where the task requires it} }\]

  • That principle connects the model architecture, training curriculum, reasoning system, multimodal extension, and agentic tool-use behavior into a single technical trajectory.

References

Qwen3 and Qwen3-VL

Reasoning and reinforcement learning

Long-context mechanisms

Vision architecture

Grounding and spatial perception

Visual reasoning and agents

Multimodal reasoning evaluation

OCR, documents, and long multimodal context

Credits

Citation

@article{Chadha2020DistilledQwen3,
  title   = {Qwen3},
  author  = {Chadha, Aman and Jain, Vinija and Ghosh, Gargi},
  journal = {Distilled AI},
  year    = {2020},
  note    = {\url{https://aman.ai}}
}