Overview

  • NVIDIA Cosmos 3 is a family of open, omnimodal world models for Physical AI. Its central architectural idea is to place world understanding, multimodal generation, physical prediction, and action generation inside one model family rather than treating them as separate systems. In practical terms, the same foundation can be configured as a vision-language model, image or video generator, audio-visual generator, forward-dynamics model, inverse-dynamics model, or robot policy/world-action model. Cosmos 3: Omnimodal World Models for Physical AI by NVIDIA et al. (2026) introduces this unified formulation across language, image, video, audio, and action.

  • The defining distinction is therefore not simply that Cosmos 3 is multimodal. It is intended to be both omnimodal and omni-functional: modalities that traditionally serve only as observations, such as video, can also be generated; actions can serve as conditioning signals or generated outputs; and representations learned for understanding can condition generation. This turns the model into a common backbone for the perceive-reason-simulate-act loop needed by robots, autonomous vehicles, and other embodied systems.

  • The following figure (source) shows how a single Cosmos 3 omnimodal world model spans vision-language understanding, image generation, audio-visual generation, policy/world-action modeling, forward dynamics, and inverse dynamics.

  • The following figure (source) shows how the pre-trained and mid-trained Cosmos 3 foundation serves as a common starting point for downstream Physical AI specialization, including text-to-image and image-to-video synthetic data generation, robot policy learning, and interactive training environments.

From Fragmented Physical AI Stacks to a Unified World Model

From Fragmented Physical AI Stacks to a Unified World Model

  • A conventional Physical AI stack often decomposes intelligence into several specialized systems. A vision-language model interprets the current scene and instruction; a policy or vision-language-action model selects actions; a forward-dynamics or world model predicts their consequences; and separate generative models may synthesize future observations or training data. Cosmos 3 starts from the premise that these capabilities are intrinsically coupled rather than independent. NVIDIA Launches Cosmos 3, the Open Frontier Foundation Model for Physical AI describes the architecture accordingly as combining vision reasoning, world generation, and action prediction within one system.

  • Consider an embodied agent operating from observation \(o_t\) under an instruction \(c\). A modular system might separately approximate:

    \[z_t = f_{\text{perception}}(o_t,c)\] \[a_t \sim \pi(a_t \mid z_t,c)\] \[o_{t+1} \sim p_{\text{world}}(o_{t+1}\mid o_{\leq t},a_t)\]

  • Here, perception, policy, and world dynamics may have different parameters and representations. Cosmos 3 instead seeks a shared multimodal representation in which closely related conditional distributions can be learned by the same backbone:

    \[p(\text{text},\text{image},\text{video},\text{audio},\text{action})\]
    • with the particular conditioning and output arrangement determining the task. The result is not literally one undifferentiated computation path. Cosmos 3 uses specialized encoders and transformer pathways internally. The important point is that these components participate in a common architecture and representation space, allowing one model to express many different physical-AI conditional distributions.

  • This creates several important operating modes:

  • Forward dynamics conditions on observations and actions to predict how the world evolves:

    \[p(o_{t+1:t+H}\mid o_{\leq t},a_{t:t+H-1})\]

  • Inverse dynamics conditions on observed state transitions and infers actions:

    \[p(a_{t:t+H-1}\mid o_{\leq t+H})\]

  • Policy generation conditions on observations and task context and predicts actions:

    \[p(a_{t:t+H-1}\mid o_{\leq t},c)\]

  • World-action generation can jointly model future observations and actions:

    \[p(o_{t+1:t+H},a_{t:t+H-1}\mid o_{\leq t},c)\]

  • This bidirectionality is especially important. Action is elevated to a first-class modality rather than being merely an external control interface attached to a visual world model. Consequently, the model can learn both “given this action, what happens?” and “given this transition or desired outcome, what action explains or produces it?” The latter connects world modeling directly to policy learning. Introducing Cosmos 3 Edge illustrates this explicitly for robotics, where Cosmos 3 Edge can predict the visual consequence of an action, infer the action associated with a change, or directly generate actions toward a desired outcome.

The Five Core Modalities

  • Cosmos 3 jointly models five broad modalities:
Modality Representative role
Language Instructions, descriptions, reasoning, planning, and textual outputs
Image Static world observations and generated visual states
Video Temporal observations, future states, physical dynamics, and simulated rollouts
Audio Ambient sound and synchronized audio-visual world generation
Action Robot controls, ego/camera motion, poses, manipulation states, and physical trajectories

  • The inclusion of action is particularly consequential. Whereas a conventional multimodal model might learn relationships among language, images, audio, and video, Cosmos 3 can connect perceptual changes directly to the controls that cause them. This is why NVIDIA characterizes the model as a foundation for Physical AI rather than simply a general media model. The NVIDIA Cosmos repository exposes this distinction directly: its Reasoner accepts text and vision and produces text, while its Generator accepts text, vision, sound, and action and can generate vision, sound, and action.

  • Internally, raw modalities are not simply concatenated in their native forms. Modality-specific encoders map them into a shared transformer representation. Visual understanding uses a ViT-based encoder, whereas visual generation uses latent representations from a video VAE. Audio has its own latent encoder, and physical actions are converted into dedicated action tokens. Learnable modality embeddings further tell the shared backbone what type of information each token represents.

  • This separation is important because “unified” does not mean forcing every modality into an identical tokenization scheme. Cosmos 3 instead separates modality-specific representation from shared multimodal computation.

Two Computational Regimes: Reasoning and Generation

  • The core architectural mechanism enabling this unification is a Mixture-of-Transformers, or MoT, backbone. Cosmos 3 couples two different computational regimes:

    • an autoregressive transformer tower for understanding and reasoning; and

    • a diffusion transformer tower for multimodal generation.

  • The autoregressive tower operates naturally over language and visual-understanding representations. Given a sequence \(x_1,\ldots,x_T\), its language-generation objective takes the conventional causal form:

    \[\mathcal{L}_{\mathrm{AR}} = -\sum_{t=1}^{T} \log p_{\theta}(x_t\mid x_{<t})\]

  • The diffusion tower instead learns to recover clean continuous latent representations from noisy ones. Abstractly, if \(x_0\) is a clean image, video, audio, or action latent and \(x_t\) its noisy version at diffusion time \(t\), training learns an appropriate denoising or flow-prediction target:

    \[\mathcal{L}_{\mathrm{gen}} = \mathbb{E}_{x_0,t,\epsilon} \left[ \left\| v_{\theta}(x_t,t,c)-v_t \right\|_2^2 \right]\]
    • where \(c\) denotes conditioning information and \(v_t\) is the corresponding training target. The precise Cosmos 3 generator formulation and its training stages are discussed later in the primer; the key conceptual point here is that discrete autoregressive reasoning and continuous multimodal generation coexist within the same MoT framework rather than requiring separate foundation models.

  • This design builds on the broader MoT principle developed in Mixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Models by Liang et al. (2025), which showed that modality-specific transformer parameters can coexist with global cross-modal attention, allowing heterogeneous modalities to retain specialized computation while exchanging information through a shared sequence representation.

  • Cosmos 3 extends that general idea in a particularly consequential direction. The specialization is not merely between text and image modalities. Its two towers correspond to different generation objectives and computational behaviors: causal autoregression for reasoning and diffusion-style denoising for world generation. The towers maintain separate normalization and MLP components while participating in shared multimodal attention, creating an explicit interface between “understand the world” and “generate or simulate the world.”

  • The attention structure is central to the design. Language reasoning uses causal self-attention, so token \(i\) cannot access future language tokens. Diffusion tokens use full attention over their permitted context because denoising does not have the same autoregressive causality constraint. The shared multimodal attention mechanism nevertheless provides the interface through which reasoning representations can inform generated visual, audio, or action trajectories. What Is Mixture-of-Transformers (MoT)? describes this broader MoT principle as conditional transformer computation in which different inputs or objectives activate specialized processing while remaining within a shared representation space.

One Model, Many Conditional Tasks

  • The practical consequence of the architecture is that changing the arrangement of inputs, conditioning signals, and generated modalities changes what type of model Cosmos 3 behaves like. Conceptually:

    \[\text{Cosmos 3} + \text{conditioning configuration} + \text{generation mode} \rightarrow \text{task-specific behavior}\]

  • For example, text and visual observations followed by textual generation instantiate a VLM-like reasoner. Text followed by generated visual latents yields text-to-image or text-to-video behavior. A starting image plus generated future visual latents produces image-to-video generation. Past video followed by future video generation implements visual forward prediction. Video together with action conditioning produces action-conditioned forward dynamics, while observations conditioned toward generated action tokens produce a policy or inverse-dynamics model. The architecture therefore treats model categories that are normally separate products as different factorizations of a common multimodal modeling problem.

  • This is the conceptual center of Cosmos 3. It is less useful to think of it as “a video generator that also understands language” than as a shared world-modeling substrate capable of mapping among descriptions, observations, trajectories, sounds, and actions.

Cosmos 3 Model Family

  • Cosmos 3 is released at three principal scales intended for different points on the quality, latency, memory, and deployment spectrum:
Model Scale Primary deployment regime Representative role
Cosmos3-Super \(64\text{B}\) Data center Maximum generation quality, synthetic-data generation, post-training, teacher model
Cosmos3-Nano \(16\text{B}\) Data center / workstation Balanced quality and speed, adaptable base model, robotics and world-action post-training
Cosmos3-Edge \(4\text{B}\) Edge / on-device Real-time visual reasoning and robot control

  • The model sizes and intended hardware regimes are documented in the NVIDIA Cosmos repository, which positions Super for H200/B200/GB200-class deployment, Nano across workstation and data-center accelerators, and Edge for constrained real-time environments.

  • Cosmos3-Super is the high-capacity member and is particularly important for high-quality world generation and as a teacher for smaller models. Cosmos3-Nano trades capacity for substantially more flexible deployment and forms the basis of the released DROID robot-policy specialization. Cosmos3-Edge compresses the same overall physical-AI philosophy into a \(4\text{B}\)-parameter model intended for on-device reasoning and control. Introducing Cosmos 3 Edge reports operation at \(640\times360\) robot-control resolution, generation of \(32\) actions per inference, and real-time control at \(15\text{ Hz}\) on Jetson Thor.

  • This scaling strategy makes the family itself part of the Physical AI stack: Super can serve high-compute generation and teacher workloads, Nano can support adaptable production workloads, and Edge targets low-latency embodied execution.

Cosmos 3 as a Physical AI Foundation

  • The broader Cosmos project predates Cosmos 3. Cosmos World Foundation Model Platform for Physical AI by NVIDIA et al. (2025) established the earlier premise that Physical AI should be trained digitally before large-scale real-world deployment, using world foundation models, video tokenizers, curation infrastructure, and post-training recipes to construct application-specific simulated worlds.

  • Cosmos 3 advances this premise from a platform centered primarily on world generation toward a foundation that jointly handles perception, reasoning, simulation, and action. That shift matters because simulated experience becomes substantially more useful when the same representational system can relate:

    \[\text{observation} \leftrightarrow \text{world state} \leftrightarrow \text{future} \leftrightarrow \text{action}\]

  • A generated video can consequently represent more than plausible pixels. When conditioned on actions, it can represent an expected physical consequence of those actions. Conversely, observed state changes can provide evidence about the action that caused them. In policy mode, actions and predicted visual consequences can be generated together, connecting policy learning with an internal predictive model of the world.

  • This makes Cosmos 3 relevant to three tightly connected Physical AI workflows. First, it can provide world understanding and physical reasoning from real observations. Second, it can generate synthetic experience and counterfactual future trajectories that expand the effective training distribution without executing every scenario in the physical world. Third, it can be post-trained into action-producing policies that operate directly on embodied tasks.

Open Physical AI Stack

  • Cosmos 3 is also intended as an open development substrate rather than only a hosted model. The Cosmos 3 technical report by NVIDIA et al. (2026) accompanies the models with source code, open-weight checkpoints, synthetic Physical AI datasets, and the Cosmos-HUE evaluation benchmark under the OpenMDW-1.1 licensing framework.

  • The released ecosystem includes specialized synthetic datasets covering physical interaction scenes, embodied robotics, autonomous driving, digital humans, and warehouse operations. The model implementation supports development and serving paths spanning Hugging Face Diffusers and Transformers, vLLM/vLLM-Omni, SGLang, TensorRT-LLM, and NVIDIA NIM, while the broader Cosmos stack includes training, data-curation, and evaluation infrastructure. NVIDIA Cosmos on GitHub provides the central implementation and integration entry point.

  • The architecture and ecosystem therefore reinforce the same design goal: Physical AI should not require independently constructing a VLM, video simulator, dynamics model, policy model, data engine, and evaluation stack for every embodied application. Cosmos 3 attempts to provide a common pretrained substrate from which those capabilities can be invoked directly or obtained through task-specific post-training.

Model Architecture

  • Cosmos 3 is built around a single architectural abstraction for heterogeneous Physical AI tasks: encode each modality into a transformer-compatible representation, arrange those representations into autoregressive and diffusion subsequences, and process them through a dual-tower Mixture-of-Transformers backbone. The autoregressive tower provides multimodal understanding and language reasoning, while the diffusion tower generates continuous visual, audio, and action representations. Cosmos 3: Omnimodal World Models for Physical AI by NVIDIA et al. (2026) describes this architecture as the mechanism that allows a single model family to perform perception, reasoning, world generation, dynamics modeling, and policy prediction without changing the core network.

  • At a high level, the computation can be written as

    \[x^{(m)} \xrightarrow{E_m} z^{(m)} \xrightarrow{\text{modality embedding}} \hat z^{(m)} \xrightarrow{\text{token packing}} S \xrightarrow{\text{MoT}} Y\]
    • where \(m\) denotes a modality, \(E_m\) is its modality-specific encoder, and \(S\) is the unified sequence consumed by the Mixture-of-Transformers backbone. What changes between tasks is primarily the composition of \(S\), which tokens are clean conditioning variables, which tokens are noisy generation targets, and which tower processes them.

Modality-Specific Encoders

  • A central design decision in Cosmos 3 is that multimodal unification occurs after modality-specific encoding. Text, images, video, audio, and actions have fundamentally different statistical and geometric structure, so they are not forced through a common raw tokenizer.

  • For each non-language modality \(m\), Cosmos 3 obtains an encoded representation and adds a learned modality-specific embedding:

    \[\hat z_i^{(m)} = P_m\left(E_m(x^{(m)})_i\right) + e_m\]
    • where \(P_m\) projects the encoder representation into the transformer hidden dimension and \(e_m\) identifies its modality. This allows the MoT backbone to operate in a shared hidden space while retaining information about whether a token originated from vision, audio, or action.

Visual Understanding Encoder

  • Cosmos 3 deliberately maintains separate visual representations for understanding and generation.

  • For understanding, images and video frames are encoded using a vision transformer pretrained with vision-language alignment. The ViT operates with \(16\times16\) image patches. A subsequent two-layer MLP merges each \(2\times2\) group of ViT tokens and projects the result into the transformer latent dimension. Thus, before considering any internal ViT compression, the post-ViT merge reduces the spatial token grid by a factor of four in token count.

  • The understanding path also incorporates two techniques associated with Qwen3-VL Technical Report by Bai et al. (2025): DeepStack-style aggregation exposes features from multiple levels of the ViT to the language model, while text-based timestamps are interleaved with video frames to provide explicit temporal grounding rather than requiring the model to infer physical time entirely from visual token positions.

  • Unlike the generative visual tokenizer, the understanding ViT is jointly trained with the Cosmos 3 backbone. The resulting representation is therefore optimized together with the Reasoner for tasks such as recognition, grounding, temporal understanding, planning, and physical reasoning.

Visual Generation Encoder

  • Generation uses a separate video VAE representation. Cosmos 3 adopts the video VAE from Wan2.2-TI2V-5B rather than attempting to diffuse directly over RGB pixels. The VAE compresses video temporally by \(4\times\) and spatially by an effective \(32\times32\) factor, with the latter obtained from \(16\times16\) VAE spatial compression followed by an additional \(2\times2\) patch merge.

  • For an input video

    \[X\in \mathbb{R}^{T\times H\times W\times C}\]

    • the resulting latent grid is approximately organized as

      \[Z_{\mathrm{video}} \in \mathbb{R}^{ \frac{T}{4} \times \frac{H}{32} \times \frac{W}{32} \times d_{\mathrm{VAE}} }\]

  • A learned linear projection subsequently maps each VAE latent token into the hidden dimension \(d_{\mathrm{model}}\) expected by the MoT:

    \[z_i^{\mathrm{DM}} = W_{\mathrm{vision}}z_i^{\mathrm{VAE}} + b_{\mathrm{vision}}\]

  • The VAE itself remains frozen during Cosmos 3 training. This division allows the expensive world model to operate on a dramatically compressed spatiotemporal representation while preserving a decoder capable of reconstructing generated latents back into video.

  • The distinction between the two visual pathways is consequently fundamental:

    \[\text{pixels} \rightarrow \begin{cases} \text{ViT tokens} & \text{visual understanding / AR}\\ \text{VAE latents} & \text{visual generation / DM} \end{cases}\]

  • The same physical scene can therefore have one representation optimized for semantic interpretation and another optimized for reconstruction and generation.

Audio Encoder

  • Audio is also represented continuously through a VAE. Cosmos 3 operates on stereo audio sampled at \(48\,\text{kHz}\) and uses a hop size of \(1{,}920\) samples. The resulting token rate is \(\frac{48{,}000}{1{,}920} = 25\) audio tokens per second.

  • The audio VAE is frozen during Cosmos 3 training, and its outputs are linearly projected into the transformer hidden dimension. Audio tokens then enter the diffusion subsequence alongside visual and action tokens. This makes synchronized audio-video generation a native sequence configuration rather than a post-processing pipeline in which sound is generated by an independent model.

Actions as a Native Modality

  • The most distinctive encoder in Cosmos 3 is the action pathway. Physical systems expose highly heterogeneous control spaces: autonomous vehicles may provide ego trajectories, cameras may provide extrinsic motion, robot arms expose end-effector or joint trajectories, and egocentric human datasets may contain head, wrist, and finger poses. Cosmos 3 maps these representations into a common conceptual action interface based primarily on geometry.

  • The following figure (source) shows the unified action representation across autonomous vehicles, camera motion, egocentric motion, single-arm robots, dual-arm robots, and humanoid robots. Motions are decomposed into shared ego-pose, effector-pose, and grasp-state components.

Relative Pose as a Pseudo-Action

  • Cosmos 3 defines an action as a causal variable associated with a transition between world states. If \(v_{t-1}\) and \(v_t\) denote consecutive visual states, then \(a_t\) represents the transition

    \[v_{t-1} \xrightarrow{a_t} v_t\]

  • Rather than attempting to normalize low-level controller interfaces such as motor currents, PID parameters, or actuator-specific commands across embodiments, Cosmos 3 represents motion using changes in physical pose. Given consecutive poses,

    \[T_{t-1},T_t\in SE(3)\]

    • the relative transformation is:

      \[\Delta T_t = T_{t-1}^{-1}T_t\]

  • This relative transformation acts as a pseudo-action. It captures what physical displacement occurred while abstracting away much of the embodiment-specific controller that produced it.

  • Translation contributes three dimensions. Rotation uses a continuous six-dimensional representation, producing a \(9\)-dimensional pose representation:

    \[a_t^{\mathrm{pose}} = [ \Delta x, \Delta y, \Delta z, r_1,\ldots,r_6 ] \in \mathbb{R}^{9}\]

  • On the Continuity of Rotation Representations in Neural Networks by Zhou et al. (2019) motivates the \(6\text{D}\) rotation parameterization used here by showing why common low-dimensional rotation parameterizations introduce discontinuities that can make learning more difficult.

Embodiment-Specific Composition

  • The common geometric primitives are composed differently depending on the embodiment. Cameras and autonomous vehicles require only a \(9\text{D}\) ego pose. A single-arm robot combines a \(9\text{D}\) ego pose, \(9\text{D}\) end-effector pose, and a scalar gripper state, while dual-arm and humanoid configurations incorporate multiple effectors and grasp states. Egocentric human motion additionally represents fingertip positions relative to each wrist.

  • This produces different native dimensionalities across domains. Cosmos 3 does not pad all of these representations into one arbitrary universal vector. Instead, it uses domain-specific projections around a shared transformer representation.

  • For domain \(k\) with normalized native action \(x\in\mathbb{R}^{d_{\mathrm{in}}^{(k)}}\) the input projection is:

    \[z = W_{\mathrm{in}}^{(k)}x + b_{\mathrm{in}}^{(k)}\]

    • with:

      \[W_{\mathrm{in}}^{(k)} \in \mathbb{R}^{ d_{\mathrm{model}}\times d_{\mathrm{in}}^{(k)} }\]

  • The latent action token therefore always satisfies \(z\in\mathbb{R}^{d_{\mathrm{model}}}\), irrespective of the dimensionality of the original embodiment.

  • Generation reverses the projection:

    \[\hat x = W_{\mathrm{out}}^{(k)}z + b_{\mathrm{out}}^{(k)}\]

  • The domain-specific input and output matrices are initialized from scratch and trained jointly with the MoT backbone. Predicted \(6\text{D}\) rotations are finally projected back into valid \(3\times3\) rotation matrices in \(SO(3)\) using singular value decomposition.

  • The architectural principle is therefore:

    \[\text{heterogeneous controls} \rightarrow \text{domain projection} \rightarrow \text{shared action latent space} \rightarrow \text{shared world model}\]

  • This allows the backbone to learn general relationships between physical motion and visual change while leaving embodiment-specific coordinate systems at the model boundary.

Unified Token Arrangement

  • After encoding, Cosmos 3 converts every task into one packed sequence containing two regions:

    \[S = [ S_{\mathrm{AR}}; S_{\mathrm{DM}} ]\]

  • The autoregressive subsequence \(S_{\mathrm{AR}}\) contains language tokens and, when applicable, ViT-derived image or video tokens. The diffusion subsequence \(S_{\mathrm{DM}}\) contains VAE-derived visual tokens, audio tokens, and action tokens.

  • The packing rules are deliberately consistent across tasks:

  • autoregressive tokens precede diffusion tokens;

  • clean conditioning tokens precede noisy target tokens inside the diffusion region; and

  • diffusion modalities are ordered vision, audio, then action.

  • This turns task specification largely into sequence construction rather than architecture selection.

Autoregressive Prefix

  • For generative tasks, Cosmos 3 defines a common autoregressive prefix

    \[S_{\mathrm{AR}} = [ l_1,\ldots,l_n, \langle EOS\rangle, \langle BOG\rangle ]\]
    • where \(\langle BOG\rangle\) marks the transition from autoregressive conditioning into generation.

  • A text-to-image request can consequently be represented as

    \[S_{\mathrm{T2I}} = [ S_{\mathrm{AR}}, \tilde v_1 ]\]
    • where \(\tilde v_1\) denotes noisy image latents to be denoised by the generator.

  • For text-to-video with optional audio,

    \[S_{\mathrm{T2V+Audio}} = [ S_{\mathrm{AR}}, \tilde v_{1:N}, \tilde s ]\]
    • where \(N\) is the number of latent video frames and \(\tilde s\) denotes noisy audio tokens.

Conditioning Through Clean Tokens

  • Image-to-video and video-to-video illustrate an important feature of the formulation: conditioning observations are represented as clean diffusion tokens, while unknown future observations are noisy tokens.

  • For \(P\) observed latent frames followed by an unknown continuation,

    \[S_{\mathrm{V2V}} = [ S_{\mathrm{AR}}, v_{1:P}, \tilde v_{P+1:N} ]\]

  • When \(P=1\), the task reduces to image-to-video. For \(P>1\) the same architecture performs video continuation.

  • Video transfer follows exactly the same abstraction. A depth map, edge sequence, or other control video is encoded as clean visual conditioning tokens, while the desired RGB video is represented by noisy visual tokens:

    \[S_{\mathrm{Transfer}} = [ S_{\mathrm{AR}}, v^{\mathrm{ctrl}}_{1:N}, \tilde v_{1:N} ]\]

  • The architecture therefore does not require a separate ControlNet-like network to distinguish a control stream from an output stream. Their roles are communicated through token arrangement and clean/noisy status.

Forward Dynamics, Inverse Dynamics, and Policy as Denoising Configurations

  • Action modeling makes the consequences of the unified token representation particularly clear. Cosmos 3 derives three different physical-AI objectives simply by varying which video and action tokens are corrupted.

  • The following figure (source) shows the three action sequence configurations. Forward dynamics denoises future visual states given clean actions, inverse dynamics denoises actions given clean visual states, and policy mode jointly denoises future video and actions.

  • For forward dynamics,

    \[[ v_{t-1}, a_t, a_{t+1}, \tilde v_t, \tilde v_{t+1} ]\]
    • asks the model to infer the future world trajectory from known actions.

  • For inverse dynamics,

    \[[ v_{t-1}, v_t, v_{t+1}, \tilde a_t, \tilde a_{t+1} ]\]
    • asks the model to infer the interventions that explain an observed trajectory.

  • Policy mode makes both future modalities unknown:

    \[[ v_{t-1}, \tilde v_t, \tilde v_{t+1}, \tilde a_t, \tilde a_{t+1} ]\]

  • The model must therefore jointly generate an action trajectory and the visual trajectory associated with it.

  • This formulation is significant because forward dynamics, inverse dynamics, and policy learning are not implemented as three architectural heads. They become three corruption-and-conditioning patterns over a common sequence model.

Mixture-of-Transformers Backbone

  • The packed sequence enters Cosmos 3’s dual-tower Mixture-of-Transformers backbone. The architecture is related to Mixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Models by Liang et al. (2024), which separates modality-specific transformer parameters while preserving global information exchange through attention.

  • Cosmos 3 specializes this principle around computational function rather than simply modality:

    \[S_{\mathrm{AR}} \rightarrow \text{Reasoner tower}\] \[S_{\mathrm{DM}} \rightarrow \text{Generator tower}\]

  • Each transformer layer contains independent parameters for the two pathways, including separate normalization modules, attention projections, and feed-forward networks. Both pathways are initialized from pretrained VLM weights, allowing the generator to begin from representations already carrying language and visual semantics rather than learning the entire multimodal feature hierarchy from scratch.

  • The following figure (source) shows the full Mixture-of-Transformers architecture: modality-specific encoders construct the AR and DM subsequences, separate Reasoner and Generator parameters process them, and dual-stream joint attention connects the two pathways while preserving autoregressive causality.

Dual-Stream Joint Attention

  • The interaction between the towers is asymmetric. Let

    \[Q_{\mathrm{AR}},K_{\mathrm{AR}},V_{\mathrm{AR}}\]

    • denote Reasoner queries, keys, and values and

      \[Q_{\mathrm{DM}},K_{\mathrm{DM}},V_{\mathrm{DM}}\]
      • their Generator counterparts.

  • The Reasoner performs ordinary causal attention only over its own sequence:

    \[O_{\mathrm{AR}} = \operatorname{Attn}_{\mathrm{causal}} ( Q_{\mathrm{AR}}, K_{\mathrm{AR}}, V_{\mathrm{AR}} )\]

  • Thus, an AR token at position \(i\) cannot inspect a later AR token and cannot inspect diffusion tokens at all. The pretrained language model’s causal semantics remain intact.

  • The Generator behaves differently:

    \[O_{\mathrm{DM}} = \operatorname{Attn}_{\mathrm{full}} \left( Q_{\mathrm{DM}}, [K_{\mathrm{AR}};K_{\mathrm{DM}}], [V_{\mathrm{AR}};V_{\mathrm{DM}}] \right)\]

  • Every diffusion query can therefore attend to the entire available AR context and the diffusion sequence. This provides direct access to textual instructions, Reasoner visual representations, clean conditioning observations, and other generation tokens.

  • The information flow is intentionally directional:

    \[\boxed{ \text{Reasoner} \longrightarrow \text{Generator} }\]

    • but not:

      \[\text{Reasoner} \longleftarrow \text{Generator}\]

  • In other words, generation can be conditioned on reasoning, but noisy generative state cannot contaminate the causal reasoning context. This is one of the architectural mechanisms that makes it possible to preserve strong VLM behavior while adding diffusion generation.

Multimodal Positional Encoding

  • A unified attention space creates a subtle problem: position has different physical meanings across modalities.

  • For text, position primarily denotes sequence order. For an image, it denotes two-dimensional spatial location. For video, it denotes space and time. For audio, it denotes time at an audio sampling rate. For robot actions, it denotes time at a control frequency.

  • Cosmos 3 therefore extends three-dimensional multimodal rotary position embeddings, or MRoPE, to represent each token using coordinates

    \[(t,h,w)\]

  • The approach builds on the multimodal positional formulation used by Qwen3-VL Technical Report by Bai et al. (2025), but modifies temporal positioning so heterogeneous physical streams can share a common real-time axis.

Coordinate Assignment

  • Language uses \(t=h=w\) so 3D MRoPE collapses to ordinary one-dimensional sequence positioning.

  • Visual tokens instead vary spatially and temporally:

    \[(t,h,w), \qquad h\in[0,H-1], \qquad w\in[0,W-1]\]

  • All tokens belonging to a given visual frame share \(t\) while receiving different \(h\) and \(w\) coordinates.

  • Audio and action have no native spatial grid, so \(h=w=0\) and only \(t\) changes.

  • The following figure (source) shows the 3D MRoPE coordinate system across language, video, audio, and action tokens, together with FPS modulation that makes temporal spacing correspond to physical elapsed time rather than raw token count.

AR-to-Diffusion Positional Margin

  • Cosmos 3 introduces an unusually large fixed positional gap between the autoregressive and diffusion subsequences. Without this separation, the last language token and first generated visual tokens occupy nearby temporal positions and therefore receive highly similar temporal positional encodings. In experiments, this produced oversaturation and checkerboard artifacts in early generated video frames, particularly for Cosmos3-Super.

  • Cosmos 3 therefore shifts the diffusion temporal coordinates by a fixed margin:

    \[k=15{,}000\]

  • Conceptually:

    \[t_{\mathrm{DM}} = t_{\mathrm{local}} + t_{\mathrm{AR,end}} + 15{,}000\]

  • The gap acts as a positional boundary between reasoning context and generated media without introducing an additional learned boundary embedding.

Absolute Temporal Modulation

  • Raw frame indices cannot provide a common physical clock because different modalities operate at different rates. For example, one frame at \(24\text{ FPS}\) represents \(\frac{1}{24}\text{ s}\) whereas one frame at \(60\text{ FPS}\) represents \(\frac{1}{60}\text{ s}\).

  • Treating both as a positional increment of \(1\) would cause identical positional distances to correspond to different amounts of physical time. The same issue occurs between video, audio, and robot actions.

  • Cosmos 3 addresses this through temporal steps per second, or TPS. For video,

    \[\mathrm{TPS}_{\mathrm{video}} = \frac{\mathrm{FPS}} {r_{\mathrm{temporal}}}\]

    • where the video VAE has:

      \[r_{\mathrm{temporal}}=4\]

  • For \(24\text{ FPS}\) video,

    \[\mathrm{TPS}_{\mathrm{video}} = \frac{24}{4} = 6\]

  • Audio produces:

    \[\mathrm{TPS}_{\mathrm{audio}} = \frac{48{,}000}{1{,}920} = 25\]
    • while action TPS equals the control sampling frequency.

  • Cosmos 3 chooses \(\mathrm{TPS}_{\mathrm{base}}=6\) and scales each temporal coordinate increment according to:

    \[\delta t = \frac{\mathrm{TPS}_{\mathrm{base}}} {\mathrm{TPS}}\]

  • Consequently, a stream sampled more densely in physical time advances more slowly through positional space. At \(24\text{ FPS}\) the latent-video increment is \(1\); for a denser stream, the increment becomes smaller; for a sparser stream, it becomes larger. Equal durations in the real world therefore span approximately equal positional distances despite differences in token rate.

  • This mechanism is particularly important for Physical AI because synchronization is semantic. An action at time \(t\) should align with the visual transition and audio event occurring at the same physical instant, not merely with whichever tokens happen to have nearby integer indices.

Architectural Scaling: Edge, Nano, and Super

  • All three Cosmos 3 variants retain the same dual-tower abstraction but scale the underlying dense transformer.
Variant Total MoT scale Base dense transformer Layers Hidden dim. Attention heads KV heads Head dim. FFN dim.
Cosmos3-Edge \(4\text{B}\) \(2\text{B}\) \(28\) \(2{,}048\) \(16\) \(8\) \(128\) \(9{,}216\)
Cosmos3-Nano \(16\text{B}\) \(8\text{B}\) \(36\) \(4{,}096\) \(32\) \(8\) \(128\) \(12{,}288\)
Cosmos3-Super \(64\text{B}\) \(32\text{B}\) \(64\) \(5{,}120\) \(64\) \(8\) \(128\) \(25{,}600\)

  • The distinction between “base dense transformer” and total Cosmos 3 scale follows directly from the MoT construction. Each transformer layer contains independent Reasoner and Generator parameter sets. Thus a model based on a dense \(8\text{B}\) transformer architecture becomes approximately a \(16\text{B}\) dual-tower model, while the \(32\text{B}\) backbone yields the approximately \(64\text{B}\) Super model.

  • Nano and Super adapt the \(8\text{B}\) and \(32\text{B}\) Qwen3-VL architectures, respectively. Qwen3-VL Technical Report by Bai et al. (2025) describes the underlying VLM family from which Cosmos 3 inherits much of its language, vision, DeepStack, and multimodal-positioning foundation.

  • Edge instead uses a \(2\text{B}\) dense transformer trained from scratch, broadly following the Qwen3-1.7B design but removing QK normalization and replacing the feed-forward activation with ReLU-squared. Duplicating its pathways under the MoT construction produces the approximately \(4\text{B}\) Cosmos3-Edge architecture.

  • The result is a remarkably consistent scaling strategy:

    \[\boxed{ \text{modality encoders} \rightarrow \text{AR/DM token packing} \rightarrow \text{dual-tower MoT} \rightarrow \\ \text{shared multimodal attention} \rightarrow \text{reasoning or denoising outputs} }\]

  • The task itself is encoded largely through the modalities supplied, their ordering, their positional coordinates, and whether each diffusion token is presented as clean context or noisy prediction target. This is the architectural mechanism that allows Cosmos 3 to move between VLM reasoning, image and video synthesis, audiovisual generation, forward simulation, inverse dynamics, and policy generation without introducing a separate foundation-model architecture for each capability.

Data and Data Curation

  • Cosmos 3 uses two complementary data curricula because its two computational pathways learn fundamentally different capabilities. The Reasoner learns to interpret and reason about the physical world from paired vision-language supervision, while the Generator learns to reconstruct, synthesize, simulate, and eventually act from large-scale image, video, audio, and action data. Cosmos 3: Omnimodal World Models for Physical AI by NVIDIA et al. (2026) organizes these curricula into progressively more specialized stages rather than training all modalities and capabilities from the same mixture from the outset.

  • The overall progression is approximately

    \[\text{broad pre-training} \rightarrow \text{Physical AI specialization} \rightarrow \text{task/domain specialization}\]

  • For the Reasoner, the final stage is supervised fine-tuning. For the Generator, NVIDIA uses pre-training, mid-training, and then optional post-training into specialized generators or policies. This distinction is important: the base Cosmos3-Nano and Cosmos3-Super checkpoints are the products of Generator mid-training, while models such as Cosmos3-Super-Text2Image, Cosmos3-Super-Image2Video, and Cosmos3-Nano-Policy-DROID are specialized descendants of those checkpoints.

Reasoner Data Curriculum

  • The Reasoner curriculum contains approximately \(24.2\text{M}\) multimodal samples:

    \[22.0\text{M pre-training} + 2.2\text{M SFT}\]

  • The exact reported counts are \(22{,}002{,}013\) pre-training examples and \(2{,}171{,}673\) supervised fine-tuning examples. Pre-training emphasizes broad image understanding, OCR, grounding, and general multimodal competence. SFT changes the distribution substantially toward video, temporal reasoning, robotics, autonomous driving, and smart-infrastructure tasks.

  • The following figure (source) shows how the Reasoner mixture changes between its \(22.0\text{M}\)-sample pre-training stage and \(2.2\text{M}\)-sample supervised fine-tuning stage.

  • The modality-level shift is particularly revealing:
Modality Pre-training Supervised fine-tuning
Image-text 18,814,952 1,051,513
Video-text 1,016,299 1,079,200
Text-only 2,170,762 40,960
Total 22,002,013 2,171,673
  • Thus, video-text data moves from a relatively small part of pre-training to roughly half of the SFT media mixture. The intended progression is from broad visual-language competence toward spatiotemporal reasoning about dynamic physical environments.

Reasoner Pre-Training Mixture

  • Approximately \(19.7\text{M}\) Reasoner examples are subselected from NVIDIA’s Nemotron Nano 2 data collection, with another \(2.3\text{M}\) examples added to strengthen mathematics, video understanding, spatial grounding, and instruction following. The resulting corpus is not consumed directly. Cosmos 3 applies semantic deduplication followed by AI-judge filtering before constructing the final mixture.

  • The final \(22\text{M}\)-sample mixture is heavily weighted toward capabilities that establish visual-language alignment and spatial understanding. OCR accounts for \(9.44\text{M}\) samples, or \(42.9\%\) of the mixture. 2D grounding contributes \(3.62\text{M}\), or \(16.5\%\), followed by visual QA at \(2.48\text{M}\) and image reasoning at \(1.66\text{M}\). Video is intentionally lighter at this stage, with approximately \(0.99\text{M}\) video-QA examples and much smaller video-captioning and video-reasoning components.

  • The design can be viewed as establishing three foundations before Physical AI specialization:

    \[\text{perception} + \text{spatial grounding} + \text{general reasoning}\]

  • Temporal and embodied reasoning are then amplified during SFT rather than dominating the initial model distribution.

Semantic Deduplication

  • Cosmos 3 performs Reasoner deduplication at the conversation level. A “conversation” is the entire training example, such as an image plus instruction-response pair, a video plus instruction-response pair, or a text-only instruction-response sample.

  • For image-text and text-only samples, semantic embeddings are produced using Qwen3-VL-Embedding-8B. Video-text examples use Perception Encoder PE-Core-G14-448. Importantly, the representation combines the media semantics with the associated instruction-response semantics rather than deduplicating purely from pixels or text.

  • The pipeline first partitions examples with K-means and then performs cosine-similarity comparisons within clusters. Near duplicates satisfying \(\operatorname{cos}(z_i,z_j)>0.95\) are removed. This hierarchical strategy avoids the prohibitive cost of exhaustive pairwise comparison while still allowing semantically redundant examples to be identified within local neighborhoods. The procedure removes \(4.23\%\) of candidate training examples as near duplicates.

AI-Judge Quality Filtering

  • After deduplication, Cosmos 3 applies a VLM judge to the remaining Reasoner data. Gemma-4-31B-it evaluates each example on three dimensions:

    • Faithfulness: whether claims in the response are supported by the supplied image, video, or textual context.

    • Completeness: whether the response adequately addresses the requested task.

    • Correctness: whether the answer and reasoning are factually and logically correct.

  • Each dimension receives an integer score from \(1\) to \(5\). The judge additionally produces a rationale to facilitate auditing.

  • Filtering is conjunctive rather than based on an average. For threshold \(\tau\), an example is retained only when

    \[s_{\mathrm{faithfulness}}\geq\tau\] \[s_{\mathrm{completeness}}\geq\tau\] \[s_{\mathrm{correctness}}\geq\tau\]

  • This prevents a high score in two dimensions from compensating for a severe defect in the third. In particular, an otherwise detailed response with unsupported visual assertions is removed for insufficient faithfulness.

  • Cosmos 3 deliberately uses different quality thresholds at different training stages. Pre-training uses the lowest non-trivial threshold, \(\tau_{\mathrm{PT}}=2\) which retains approximately \(78\%\) of judged data and preserves broad capability coverage. SFT instead uses \(\tau_{\mathrm{SFT}}=5\) which retains only about \(46\%\), prioritizing annotation precision over scale. This reflects a broader data strategy: broad but filtered supervision establishes the representation, while later specialization uses substantially higher-confidence examples.

Physical AI Supervised Fine-Tuning

  • The \(2.2\text{M}\)-sample Reasoner SFT corpus is organized around general spatial and temporal reasoning plus three major application areas: autonomous vehicles, robotics and embodied AI, and smart infrastructure.

Spatial Understanding and Grounding

  • The spatial curriculum combines 2D and 3D grounding from real and simulated environments. Tasks include object and region localization, pointing, counting, OCR/layout grounding, multi-image correspondence, referring expressions, trajectories, free-space prediction, reachability, and relative spatial relationships. Boxes and points are normalized into a unified structured representation.

  • For 3D tasks, Cosmos 3 directly supervises camera-relative 3D boxes rather than requiring a two-stage pipeline of 2D localization followed by 3D reconstruction. A typical target includes:

    \[b_{\mathrm{3D}} = (x,y,z,w,h,d,\theta,\ldots)\]
    • representing object position, dimensions, and orientation in the relevant camera frame.

  • Simulator-derived data complements real-world supervision. Ground-truth answers can be computed directly from camera parameters, object poses, bounding boxes, depth, segmentation masks, visibility, and feasible action regions. This allows the curriculum to train not only descriptive spatial relationships but also actionable spatial reasoning, such as whether a region is reachable or where an object can be placed.

Temporal Understanding and Physical Plausibility

  • Temporal supervision targets three capabilities: understanding events over time, judging physical plausibility, and expanding sparse prompts into structured spatiotemporal scene descriptions.

  • For event understanding, the data includes densely annotated egocentric activities and a larger corpus of \(55\text{K}\) videos totaling approximately \(2.6\text{K}\) hours. That corpus provides \(743\text{K}\) event triplets of the form

    \[(t_{\mathrm{start}},t_{\mathrm{end}},\text{caption})\]
    • supporting both event enumeration and query-conditioned temporal localization. Additional supervision targets fine-grained motion differences and camera-motion patterns such as panning and zooming.

  • Physical-plausibility supervision includes \(13.5\text{K}\) human-graded video-question-answer tuples derived from \(1\text{K}\) generated videos. The labels cover visual integrity, temporal stability, geometry, anatomy, motion plausibility, and physical commonsense. Cosmos 3 additionally incorporates VideoPhy-2 by Bansal et al. (2025), containing action-centric videos scored for adherence to physical laws. The resulting supervision teaches the Reasoner to assess properties such as gravity, collisions, conservation behavior, causality, and spatial constraints rather than merely recognize objects.

Structured Prompt Upsampling

  • Another part of Reasoner SFT trains the model to transform underspecified generation requests into dense structured descriptions. Inputs may be text, an image, or both. Targets describe subjects, attributes, layout, camera framing, interactions, and plausible future motion.

  • This creates a direct Reasoner-to-Generator interface:

    \[\text{short user prompt} \xrightarrow{\text{Reasoner}} \text{structured scene specification} \xrightarrow{\text{Generator}} \text{media}\]

  • Different instruction styles and requested levels of detail are synthesized during training so the Reasoner learns to produce the same underlying structured output contract from heterogeneous user requests.

Autonomous-Driving Reasoning Data

  • The autonomous-vehicle curriculum includes action chain-of-thought, event localization, and metric 3D vehicle grounding.

  • The action-CoT component begins with more than \(10\text{K}\) human-labeled driving videos containing explicit driving decisions and contextual information about weather, lighting, road conditions, traffic rules, ego behavior, critical objects, and causal relationships. NVIDIA then scales this supervision to roughly \(1.1\text{M}\) decision-rich internal driving videos using automatic annotation. The pipeline identifies a meta-action transition keyframe as the decision point and uses video, ego trajectories, dynamic states, meta-actions, and VLMs to generate structured decisions and concise reasoning traces.

  • Temporal event data also incorporates dashcam footage containing collisions, near collisions, hard braking, harsh acceleration, and sharp cornering. Clips are sampled at \(6\text{ FPS}\) and limited to \(300\) seconds, with dense descriptions of scene state, interacting agents, ego behavior, and temporal context.

  • For metric 3D grounding, Cosmos 3 uses MADS data with synchronized cameras, 3D annotations, camera calibration, and ego poses. The task asks the model to enumerate or locate vehicles according to properties such as relative position, lane, motion state, or distance, returning camera-frame 3D boxes.

Robotics and Embodied Reasoning Data

  • Robot Action-CoT converts a high-level manipulation instruction and current observation into a structured image-plane motion plan. Instead of supervising an unconstrained natural-language rationale, the target identifies task-relevant objects, affordance locations, collision-free regions, grounding points, and finally 2D end-effector waypoints.

  • The annotation pipeline decomposes the problem across specialized models. Qwen3-VL-72B-Instruct supplies grounding rationales and movement reasoning, Molmo-7B performs referring-expression localization, and motion-plan targets are derived from MolmoAct or tracked DROID demonstrations. DROID: A Large-Scale In-the-Wild Robot Manipulation Dataset by Khazatsky et al. (2024) provides diverse real-world robot trajectories that can anchor this form of embodied supervision.

  • Longer-horizon reasoning includes \(3.6\text{K}\) Omniverse-rerendered manipulation videos across six tasks and \(83\text{K}\) BEHAVIOR-1K samples mapping scene observations and candidate actions to a ground-truth next action. Additional robotics QA targets affordances, task planning, failure detection, physical commonsense, localization, relationships, and trajectory prediction.

  • The Reasoner curriculum even extends to robotic surgery, with \(398\text{K}\) multi-turn conversations over \(2.2\text{M}\) images drawn from operating-room, robotic-detail, and console viewpoints. Tasks include tool recognition and localization, scene graphs, text transcription, personnel counting, action labeling, and surgical-phase recognition.

Smart-Infrastructure Data

  • Smart-infrastructure supervision covers warehouse spatial intelligence, pedestrian localization, and traffic/anomaly reasoning.

  • The warehouse component uses a synthetic Omniverse corpus spanning \(44\) warehouse scene collections and \(40\) camera views. From \(93\text{K}\) RGB-D images and \(873\text{K}\) QA pairs, NVIDIA samples \(80\text{K}\) balanced examples covering counting, metric distance, grounding, and spatial relations among pallets, boxes, forklifts, shelves, and operator zones.

  • Dense pedestrian localization uses \(208\text{K}\) images from \(44\) scenes with \(5.6\text{M}\) manually annotated person boxes. The report states that personally identifiable information is blurred before annotation and release. Traffic reasoning combines synthetic collision scenarios, real traffic events, causal and temporal reasoning, and surveillance anomaly detection.

Generator Data Curriculum

  • The Generator operates at a much larger data scale. Its curriculum progressively activates modalities and tasks rather than presenting the complete omnimodal problem from the beginning.

  • The following figure (source) summarizes which Generator modes are active during pre-training, mid-training, and specialized post-training.

  • The reported training-mode counts illustrate the progression:
Training mode / modality Pre-training Mid-training Post-training
Image 767M 16M 8M
Video 348M 75M 20K
Audio 139M 19M -
Action - 8M 58K
Video transfer - 4M -
  • These rows correspond to multiple conditional tasks. Video supports text-to-video, image-to-video, and video-to-video; audio enables joint audiovisual generation; action data supports forward dynamics, inverse dynamics, and policy configurations; and transfer data introduces spatially controlled video generation.

Image and Video Pre-Training at Web Scale

  • The retained visual pre-training corpus contains approximately \(767\text{M images}\) and \(347.7\text{M video clips}\) processed from approximately \(7.8\text{B raw images}\) and \(3\text{B raw source videos}\).

  • The large difference between raw and retained counts makes curation a core component of the training recipe rather than a peripheral preprocessing step.

  • The retained corpus is multi-resolution. For images, \(26.8\%\) are 720p and \(26.0\%\) are 480p; for videos, \(36.4\%\) are 720p and \(30.8\%\) are 480p. Another \(25.2\%\) of images and \(12.2\%\) of videos are 1080p or higher. The dominant aspect ratio is \(16:9\), representing \(52.0\%\) of images and \(97.3\%\) of videos.

  • Long source videos are first segmented into temporally coherent clips using TransNetV2. Black borders are detected and removed with ffmpeg cropdetect, and video is re-encoded into a canonical representation to standardize downstream processing.

Generator Deduplication and Quality Filtering

  • Images are embedded using Qwen3-VL-Embedding-8B, while videos use NVIDIA Cosmos-Embed1-448p. NVIDIA samples \(147\text{M}\) images and \(400\text{M}\) clips and independently clusters each modality into \(20{,}000\) K-means clusters. Near-duplicate removal is then performed within clusters using cosine similarity.

  • The corpus is also classified into \(47\) hierarchical semantic categories covering general and Physical AI content.

  • Image filters include aesthetic quality, photorealism, collage detection, watermarks, white backgrounds, and NSFW content. Video filtering combines DOVER aesthetic and technical quality scores with VTSS training-suitability scores and approximately \(100\) binary artifact labels. Major defects such as split-screen layouts, rotated footage, and static videos trigger rejection, whereas some lesser artifacts such as text overlays, motion blur, and compression noise can remain in pre-training.

  • This distinction becomes important later: pre-training prioritizes coverage, whereas mid-training applies considerably stricter filtering.

Generator Mid-Training

  • Mid-training serves two roles. First, it raises visual generation quality by concentrating training on higher-quality and harder examples. Second, it introduces capabilities that were absent during broad pre-training, particularly actions and control-conditioned video transfer.

  • The visual mixture draws from three broad sources:

    • higher-quality real images and videos;

    • targeted synthetic data; and

    • control-conditioned video-transfer data.

  • For images, the effective mid-training composition is \(60\%\) real images, \(36\%\) synthetic images, and \(4\%\) text-rendering images. For video, \(46.0\%\) comes from a more aggressively filtered subset of pre-training video, \(43.9\%\) from high-quality domain-specific robotics, autonomous-driving, human-activity, and egocentric interaction footage, and the remaining \(10.1\%\) from capability-oriented hard cases such as complex human motion, high-speed motion, and fine-grained manipulation.

  • The resulting philosophy is essentially

    \[\text{pre-training} \approx \text{coverage and diversity}\] \[\text{mid-training} \approx \text{quality + Physical AI density + new capabilities}\]

Synthetic Physical AI Data

  • Synthetic data is used explicitly to correct the long-tailed distribution of web-scale visual data. NVIDIA identifies several Physical AI domains where naturally occurring web video does not provide sufficient coverage of rare interactions, controlled dynamics, or safety-critical scenarios.

  • The synthetic corpus is divided into five major families:

    • SDG-PhyxSim: rigid-body collisions, articulated-object dynamics, deformable materials, fluids, and optical effects.

    • SDG-RobotSim: manipulation and locomotion across roughly six to eight robot embodiments.

    • SDG-DriveSim: routine and corner-case autonomous-driving scenarios.

    • SDG-SynHuman: human dynamics, camera-motion priors, and multi-character interactions.

    • SDG-Warehouse: warehouse operations and human-forklift safety scenarios.

  • These datasets are not merely intended to increase sample count. They deliberately increase probability mass on physical interactions and environments that are underrepresented in general video distributions.

Video-Transfer Data

  • Control-conditioned generation is introduced during mid-training using approximately \(3\text{M}\) selected videos, emphasizing high-quality footage and Physical AI domains such as robotics and autonomous driving.

  • The control inputs include:

    \[\text{edge}, \quad \text{blur}, \quad \text{depth}, \quad \text{segmentation}, \quad \text{world-scenario maps}\]

  • Canny edge detection, Gaussian blur, and bilateral filtering generate some controls dynamically during training. Depth is precomputed with Video Depth Anything, while segmentation uses SAMv2. SAM 2: Segment Anything in Images and Videos by Ravi et al. (2024) provides the video-capable segmentation model underlying this component of the pipeline.

  • For autonomous driving, MADS contributes \(1.1\text{M}\) samples with seven synchronized camera views and corresponding world-scenario maps containing lane lines, road boundaries, traffic lights, and dynamic 3D boxes.

  • The training relation is:

    \[(\text{text},\text{control video}) \rightarrow \text{RGB video}\]

  • Because control inputs and targets share the Cosmos 3 visual latent space, this capability fits directly into the clean-conditioning/noisy-target sequence representation introduced in the architecture section.

Structured Caption Annotation

  • Cosmos 3 does not rely primarily on conventional free-form dense captions for Generator training. NVIDIA reports that free-form captions tend to achieve reasonable precision but omit important details in complex scenes. Instead, all Generator training stages use structured JSON annotations designed to systematically capture scene content.

  • For images, fields describe subjects, background, lighting, aesthetics, and cinematography. Video annotations additionally encode temporal events, object interactions, human motion, state changes, camera motion, transitions, and other temporal structure.

  • Two Qwen3-VL-8B models are fine-tuned as dedicated image and video captioners. Caption quality is then evaluated at the atomic-assertion level using separate precision and recall procedures. Precision asks whether individual generated claims are actually supported by the source media, while recall compares the generated description against human-curated atomic ground-truth assertions.

  • This effectively optimizes the annotation pipeline around two failure modes:

    \[\text{hallucinated detail} \rightarrow \text{lower precision}\] \[\text{missing scene information} \rightarrow \text{lower recall}\]

  • The structured format is reported to substantially improve recall while retaining high precision.

Audio Data and Audio-Visual Alignment

  • Audio curation is designed around a particularly important distinction for world models: correlation is not necessarily physical causation. A narration may describe an object visible in a video while having no physical relationship to what the object is doing. Likewise, background music may correlate with a scene while providing no information about its dynamics.

  • Pre-training therefore favors breadth, while mid-training explicitly targets stronger audio-visual correspondence.

  • The pre-training audio corpus is derived from the video corpus and contains \(138.9\text{M}\) clips with usable audio. These include speech, voiceover, music, background music, ambient sound, and physical events. For the \(62.5\text{M}\) clips shorter than 30 seconds, Qwen3-Omni-Captioner produces synthetic audio descriptions.

  • Mid-training reduces this to a much more selective \(18.8\text{M}\)-clip pool:

    \[12.8\text{M non-speech} + 6.0\text{M synchronized speech}\]

  • The key rule is to retain speech when it is synchronized with a visible speaker, suppress off-screen speech when constructing non-speech examples, and remove background music when it would obscure the physically grounded audio signal.

  • The pipeline combines source separation, lip synchronization, audio-event detection, instrument detection, silence filtering, ASR, and re-captioning. SyncNet identifies visible lip-synchronized speech, with the report defining synchronization using \(\texttt{has\_face}=\mathrm{True}\) and \(\texttt{lip\_sync\_confidence}\geq3.0\).

  • Audio modified by source separation is re-captioned so that the textual description corresponds to the waveform actually used as the training target, rather than describing speech or music that has already been removed.

  • This curation is central to learning

    \[\text{visible physical event} \leftrightarrow \text{temporally aligned sound}\]
    • rather than merely learning broad semantic co-occurrence between video and audio.

Action Mid-Training Data

  • Actions appear only during Generator mid-training. The final action corpus contains approximately \(8.4\text{M episodes}\) and \(61.3\text{K hours}\) across four Physical AI domains: egocentric motion, autonomous vehicles, robotics, and camera motion.

  • The following figure (source) shows the action-data distribution by hours.

  • The distribution is:
Domain Hours Share
Egocentric motion 41.3K 67.4%
Autonomous vehicles 10.0K 16.3%
Robotics 5.4K 8.7%
Camera motion 4.6K 7.5%

Egocentric Motion

  • Egocentric data is the dominant action source, contributing \(1.7\text{M}\) episodes and \(41.3\text{K}\) hours of bimanual manipulation captured from a head-mounted RGB camera. Frames are synchronized with head-camera pose and 3D hand-pose annotations. This allows the same training examples to teach both ego-motion and dexterous hand motion.

Autonomous Vehicles

  • Driving contributes approximately \(10.0\text{K}\) hours from NVIDIA Hyperion logs. Rather than accepting the naturally occurring driving distribution, scenarios are mined to match a target mixture spanning weather, lighting, road conditions, and longitudinal and lateral maneuvers. Vehicle trajectories are transformed into the front-wide camera coordinate frame to match the common action representation.

Robotics

  • The robotics subset contains approximately \(5.4\text{K}\) hours, \(516.7\text{K}\) episodes, and \(90.4\text{K}\) tasks across several embodiments, including AgiBot, Franka Panda, Google Robot, WidowX-250, UMI, and UR systems. Both successful and failed demonstrations are retained, exposing the world model to off-nominal action consequences rather than only successful trajectories.

  • DROID: A Large-Scale In-the-Wild Robot Manipulation Dataset by Khazatsky et al. (2024) is one of the open robotics sources incorporated into this mixture and provides diverse manipulation trajectories collected across environments, tasks, and operators.

Camera Motion

  • Camera-motion supervision is mined from ordinary pre-training videos rather than requiring explicit control logs. ViPE and DepthAnything3 estimate metric camera trajectories; clips with unstable estimates, excessive jitter, or abnormal intrinsics are removed. This procedure yields approximately \(1.9\text{M}\) clips and \(4.6\text{K}\) hours of camera-motion action data.

  • This is an important example of converting passive observational video into action-conditioned world-model training data:

    \[\text{ordinary video} \xrightarrow{\text{camera pose estimation}} (\text{video},\text{pseudo-action trajectory})\]

Normalizing Actions Across Embodiments

  • After each action source is mapped through the unified action representation described in the architecture section, Cosmos 3 computes per-dimension normalization statistics from the training corpus. Action channels are rescaled to approximately \([-1,1]\).

  • This prevents domains with numerically larger coordinate scales from disproportionately influencing optimization. Multi-camera observations are concatenated into a visual canvas, with camera layout stored as metadata. Idle actions are deliberately retained rather than discarded, and their counts are stored so downstream samplers can explicitly control the active-to-idle ratio.

From Broad Data to Physical World Modeling

  • The complete data strategy can be summarized as progressively increasing the density of physically meaningful supervision:

    \[\begin{aligned} \text{Reasoner:}\quad &\text{general multimodal data}\\ &\rightarrow\text{high-quality spatial/temporal supervision}\\ &\rightarrow\text{robotics + driving + infrastructure reasoning}, \[6pt] \text{Generator:}\quad &\text{web-scale images/videos/audio}\\ &\rightarrow\text{high-quality + synthetic Physical AI data}\\ &\rightarrow\text{action + control-conditioned data}\\ &\rightarrow\text{specialized generators/policies} \end{aligned}\]

  • The key point is that Cosmos 3’s omnimodality does not come from simply concatenating every available modality into one web-scale corpus. The curriculum progressively transforms a general multimodal model into a physical world model by increasing data quality, temporal grounding, physical-event density, cross-modal synchronization, explicit action supervision, and domain-specific coverage. The architecture provides a unified representational interface, but the data curriculum determines whether that interface learns merely to model multimodal media or to connect perception, dynamics, causality, and action.

Training

  • Cosmos 3 uses a staged training strategy in which the Reasoner is trained first and then used to initialize the Generator. This ordering is important. Rather than training the generative tower independently from random initialization, Cosmos 3 transfers semantic and world knowledge from the trained multimodal Reasoner into the Generator before learning image, video, audio, and action generation. Cosmos 3: Omnimodal World Models for Physical AI by NVIDIA et al. (2026) describes the overall progression as Reasoner pre-training and Physical AI SFT, followed by Generator pre-training, mid-training, and task-specific post-training.

  • At a high level,

    \[\text{Reasoner PT} \rightarrow \text{Reasoner SFT} \rightarrow \text{Generator initialization} \rightarrow \text{Generator PT} \rightarrow \\ \text{Generator MT} \rightarrow \text{specialized post-training}\]

  • This creates a useful separation of concerns. The Reasoner first learns representations suitable for understanding the world. The Generator then inherits that semantic foundation while learning how continuous physical states evolve and how images, video, audio, and actions can be synthesized.

Training Objectives Across Pre-Training, Mid-Training, and Post-Training

  • Cosmos 3 uses two complementary objective families across its training curriculum. The Reasoner is optimized autoregressively with next-token prediction, while the Generator uses rectified flow matching over continuous image, video, audio, and action latents. The Generator keeps the same basic denoising formulation as training progresses from broad pre-training to Physical AI mid-training and downstream specialization, while changing the modalities, conditioning patterns, data mixture, and loss weighting.

Reasoner Objective

  • The Reasoner is first pre-trained on multimodal image-text, video-text, and text corpora and then supervised fine-tuned on curated Physical AI tasks. In both stages, its learning problem remains autoregressive next-token prediction; what changes is the training distribution and sampling strategy. During pre-training, Cosmos 3 additionally applies square-root normalized per-token loss weighting to balance short and long sequences.
Multimodal Pre-Training

  • Let \(y_t\) denote the target language token at position \(t\), \(y_{<t}\) the preceding autoregressive context, \(m\) the associated image/video context encoded by the ViT when present, and \(w_t\) the square-root-normalized per-token weight. A compact representation of the Reasoner pre-training objective is

    \[\mathcal{L}_{\mathrm{Reasoner\text{-}PT}} = -\sum_{t=1}^{T} \underbrace{w_t}_{\text{length-balancing weight}} \underbrace{\log p_{\theta}\!\left(y_t \mid y_{<t},m\right)}_{\text{next-token log-likelihood}}\]
    • where \(w_t\) denotes the paper’s square-root-normalized per-token weighting, used to balance the contributions of short and long sequences. The report specifies this weighting strategy but does not provide its explicit closed-form formula, so it is left abstract here rather than introducing an unsupported expression.

  • All Reasoner components are trained jointly from the beginning of pre-training rather than using a separate projector-only alignment stage. Thus, the objective updates the language model, multimodal projector, and ViT together.

Supervised Fine-Tuning

  • Supervised fine-tuning preserves the autoregressive prediction objective but changes the data distribution toward high-quality Physical AI instruction and reasoning examples. If \(\mathcal{D}_{\mathrm{SFT}}\) is the curated SFT mixture and \(\mathcal{D}_{\mathrm{PT}}^{\mathrm{HQ}}\) is the filtered pre-training replay stream, the stage can be summarized as

    \[\mathcal{L}_{\mathrm{Reasoner\text{-}SFT}} = \mathbb{E}_{(y,m)\sim \underbrace{\mathcal{D}_{\mathrm{SFT}}\cup\mathcal{D}_{\mathrm{PT}}^{\mathrm{HQ}}}_{\text{importance-aware SFT mixture + pre-training replay}}} \left[ -\sum_{t=1}^{T} \underbrace{\log p_{\theta}\!\left(y_t\mid y_{<t},m\right)}_{\text{supervised next-token loss}} \right]\]
    • where the filtered pre-training stream is retained at a fixed \(1{:}4\) pre-training-to-SFT sampling-budget ratio. This replay is intended to preserve broad reasoning, visual understanding, and instruction-following capabilities while the model specializes to Physical AI.

Generator Objective

  • The Generator is trained with rectified flow matching across its continuous modalities. For a clean target latent \(x_0\), Gaussian noise \(\epsilon\sim\mathcal{N}(0,I)\), and noise level \(\sigma\in[0,1]\), Cosmos 3 constructs a point along a straight interpolation path between data and noise

    \[\underbrace{x_{\sigma}}_{\text{noisy/interpolated latent}} = \underbrace{\sigma\epsilon}_{\text{noise contribution}} + \underbrace{(1-\sigma)x_0}_{\text{clean-target contribution}}\]

  • Because this path has constant velocity with respect to \(\sigma\), its target velocity is

    \[\underbrace{v^{*}}_{\text{target flow velocity}} = \underbrace{\epsilon}_{\text{Gaussian noise}} - \underbrace{x_0}_{\text{clean target latent}}\]

  • The denoiser \(v_{\theta}(x_{\sigma},\sigma,c)\), conditioned on context \(c\), is trained to predict this velocity with a masked mean-squared error. Let \(M\) equal one on target tokens and zero on clean conditioning tokens. The common Generator objective can therefore be written as

    \[\mathcal{L}_{\mathrm{RF}} = \mathbb{E}_{x_0,\epsilon,\sigma} \left[ \underbrace{ \left\| \underbrace{M}_{\text{target-token mask}} \odot \left( \underbrace{v_{\theta}(x_{\sigma},\sigma,c)}_{\text{predicted velocity}} - \underbrace{(\epsilon-x_0)}_{\text{target velocity}} \right) \right\|_2^2 }_{\text{masked velocity MSE}} \right]\]
    • where clean conditioning tokens, such as observed frames in image-to-video or the control stream in transfer generation, are masked out of the loss. Noise levels are sampled independently by modality. The paper uses logit-normal noise sampling for image, audio, and action batches and mode sampling for video batches.

Generator Pre-Training

  • Pre-training applies the rectified-flow objective jointly to image, video, and audio generation. The task mixture includes Text-to-Image, Text-to-(Video+Audio), Image-to-(Video+Audio), and Video-to-(Video+Audio), so the same objective learns both generation from text and conditional continuation from clean prefixes.

  • Writing \(\mathcal{M}_{\mathrm{PT}}=\{\mathrm{image},\mathrm{video},\mathrm{audio}\}\), the stage-level objective is

    \[\mathcal{L}_{\mathrm{Gen\text{-}PT}} = \sum_{m\in\mathcal{M}_{\mathrm{PT}}} \underbrace{\lambda_m}_{\text{modality/task sampling contribution}} \underbrace{ \mathbb{E} \left[ \left\| M_m\odot \left( v_{\theta}^{(m)}(x_{\sigma_m},\sigma_m,c) - (\epsilon_m-x_0^{(m)}) \right) \right\|_2^2 \right] }_{\text{rectified-flow velocity loss for modality }m}\]
    • where \(\sigma_m\) is sampled independently for each modality and \(M_m\) excludes clean conditioning tokens. The \(\lambda_m\) notation summarizes the contribution induced by the training mixture; the paper specifies sampling mixtures rather than a separate explicit analytical coefficient for each pre-training modality.

  • During Generator pre-training, only generation-specific parameters are updated and the Reasoner tower remains frozen. This preserves the inherited language and visual-understanding capabilities while the Generator learns continuous multimodal synthesis.

Generator Mid-Training

  • Mid-training keeps the same rectified-flow objective but expands the modeled space to action and control-conditioned transfer. The mixture contains image, video, video+audio, action, general transfer, and driving-transfer streams. Action supervision is instantiated in forward-dynamics, inverse-dynamics, and policy configurations by changing which video and action tokens are clean conditioning variables and which are noisy prediction targets.

  • The report explicitly states that the total mid-training loss is the sum of per-modality velocity MSEs with modality-specific loss scales. Action losses receive a \(10\times\) multiplier because normalized action vectors otherwise produce smaller per-element MSEs. A stage-level expression is

    \[\mathcal{L}_{\mathrm{Gen\text{-}MT}} = \underbrace{ \sum_{m\in\mathcal{M}_{\mathrm{non\text{-}action}}} \lambda_m\,\mathcal{L}_{\mathrm{RF}}^{(m)} }_{\text{image, video, audio, and transfer velocity losses}} + \underbrace{ 10\,\lambda_{\mathrm{act}}\,\mathcal{L}_{\mathrm{RF}}^{(\mathrm{act})} }_{\text{10× scaled action velocity loss}}\]
    • where \(\mathcal{M}_{\mathrm{non\text{-}action}}\) covers the non-action streams and \(\mathcal{L}_{\mathrm{RF}}^{(\mathrm{act})}\) applies the same velocity-prediction formulation to action tokens. Action inherits the vision noise schedule during this stage.
Forward Dynamics, Inverse Dynamics, and Policy Masks

  • The action objective does not require three unrelated losses. Instead, Cosmos 3 changes the clean/noisy mask over the same video-action sequence. For video latents \(v\) and action latents \(a\), the three configurations can be summarized as

    \[\underbrace{\mathcal{L}_{\mathrm{FD}}}_{\text{forward dynamics}} = \underbrace{\mathcal{L}_{\mathrm{RF}}\!\left(\tilde v\mid v_{\mathrm{context}},a\right)}_{\text{denoise future video conditioned on clean actions}}\] \[\underbrace{\mathcal{L}_{\mathrm{ID}}}_{\text{inverse dynamics}} = \underbrace{\mathcal{L}_{\mathrm{RF}}\!\left(\tilde a\mid v\right)}_{\text{denoise actions conditioned on clean visual transitions}}\] \[\underbrace{\mathcal{L}_{\mathrm{Policy}}}_{\text{joint video-action prediction}} = \underbrace{\mathcal{L}_{\mathrm{RF}}\!\left(\tilde v,\tilde a\mid v_{\mathrm{context}}\right)}_{\text{jointly denoise future visual states and actions}}\]
    • where a tilde denotes a noise-perturbed diffusion target. Forward dynamics, inverse dynamics, and policy learning therefore share the same Generator and rectified-flow machinery, while differing primarily in which modalities are observed and which are reconstructed.

Generator Post-Training

  • Post-training specializes the mid-trained Generator for particular downstream capabilities. The paper describes Text-to-Image, Image-to-Video, and DROID robot-policy post-training as supervised fine-tuning stages. It does not introduce a new mathematical loss family for these stages; instead, they continue from the Generator training setup while narrowing the task distribution, resolution, horizon, and target domain.
Text-to-Image Specialization

  • Cosmos3-Super-Text2Image uses two-stage SFT. Stage 1 performs broad T2I specialization for 20K steps, followed by a 2K-step refinement on 470K ultra-high-quality image-caption pairs. Since the report states that the remaining hyperparameters follow the mid-training setup and does not specify a replacement objective, the post-training objective is most faithfully represented as the same masked rectified-flow loss restricted to T2I examples

    \[\mathcal{L}_{\mathrm{T2I\text{-}PT}} = \mathbb{E}_{(x_0,c)\sim \underbrace{\mathcal{D}_{\mathrm{T2I}}}_{\text{curated image-caption specialization data}}} \left[ \underbrace{ \left\| v_{\theta}(x_{\sigma},\sigma,c) - (\epsilon-x_0) \right\|_2^2 }_{\text{image-latent velocity prediction}} \right]\]
    • where the principal change from mid-training is specialization of the data distribution rather than introduction of a new objective. Stage 2 further concentrates that distribution on ultra-high-quality examples to improve aesthetics, prompt following, text rendering, and human preference alignment.
Image-to-Video Specialization

  • Cosmos3-Super-Image2Video is fine-tuned on filtered and targeted video data using the I2V formulation, with 20% T2I image tokens retained for semantic alignment. The corresponding masked objective predicts noisy future video latents while excluding the clean conditioning image from the loss

    \[\mathcal{L}_{\mathrm{I2V\text{-}PT}} = \mathbb{E} \left[ \underbrace{ \left\| \underbrace{M_{\mathrm{future}}}_{\text{future-frame loss mask}} \odot \left( \underbrace{v_{\theta}(x_{\sigma},\sigma,c_{\mathrm{image}},c_{\mathrm{text}})}_{\text{conditioned video velocity prediction}} - \underbrace{(\epsilon-x_0)}_{\text{target velocity}} \right) \right\|_2^2 }_{\text{future-video rectified-flow loss}} \right]\]
    • where the clean initial image is conditioning context rather than a reconstruction target. The report specializes this checkpoint to 480p and 189 frames and trains it for 10K iterations.
Robot Policy Specialization

  • Cosmos3-Nano-Policy-DROID resumes from the mid-trained Nano checkpoint and specializes the policy configuration to DROID. It predicts 32 future absolute joint-position actions together with auxiliary RGB video frames. The report initializes new action-related modules and applies a \(5\times\) learning-rate multiplier to those parameters, while otherwise following the mid-training setup.

  • The policy objective can therefore be expressed using the inherited joint video-action rectified-flow form

    \[\mathcal{L}_{\mathrm{DROID}} = \underbrace{ \lambda_{\mathrm{video}}\mathcal{L}_{\mathrm{RF}}^{(\mathrm{video})} }_{\text{auxiliary future-RGB prediction}} + \underbrace{ 10\,\lambda_{\mathrm{action}}\mathcal{L}_{\mathrm{RF}}^{(\mathrm{action})} }_{\text{scaled 32-step action prediction}}\]
    • where the \(10\times\) action loss scale comes from the inherited mid-training setup. The paper states that the robot-policy stage follows the mid-training hyperparameters except for the reported policy-specific changes, but it does not separately restate a closed-form post-training loss. The equation above therefore makes the inherited objective explicit rather than claiming an additional loss introduced only for DROID.

Objective Progression Across the Curriculum

  • The overall curriculum can be summarized as a progression in what is conditioned on and what is predicted, rather than a sequence of unrelated objective functions

    \[\underbrace{\mathcal{L}_{\mathrm{NTP}}}_{\text{Reasoner: predict language tokens}} \quad\longrightarrow\quad \underbrace{\mathcal{L}_{\mathrm{RF}}^{\mathrm{image/video/audio}}}_{\text{Generator pre-training: broad generative priors}} \quad\longrightarrow\quad \\ \underbrace{\mathcal{L}_{\mathrm{RF}}^{\mathrm{image/video/audio/action/control}}}_{\text{mid-training: Physical AI dynamics and controls}} \quad\longrightarrow\quad \underbrace{\mathcal{L}_{\mathrm{RF}}^{\mathrm{task}}}_{\text{post-training: T2I, I2V, or policy specialization}}\]

  • This continuity is central to Cosmos 3. The Reasoner establishes semantic and multimodal understanding through next-token prediction; the Generator converts that foundation into continuous multimodal generation through flow matching; mid-training expands the same denoising objective to actions and controls; and post-training specializes the resulting world-action prior without requiring a new task-specific architecture.

Reasoner Training

  • The Reasoner is trained in two stages:

    \[\text{multimodal pre-training} \rightarrow \text{Physical AI supervised fine-tuning}\]

  • Pre-training establishes general image, video, text, grounding, and reasoning capabilities. SFT shifts the optimization toward robotics, autonomous driving, spatial intelligence, temporal understanding, and smart infrastructure while retaining a portion of general pre-training data to reduce catastrophic forgetting.

Reasoner Initialization

  • Reasoner pre-training starts with three components:

    \[\text{language model} + \text{ViT} + \text{multimodal projector}\]

  • The initialization differs by model scale. Cosmos3-Edge uses NVIDIA’s internally pretrained model, Cosmos3-Nano starts from Qwen3-VL-8B, and Cosmos3-Super starts from Qwen3-VL-32B.

  • A notable departure from some VLM training recipes is that Cosmos 3 does not first freeze the language and vision backbones and train only the multimodal projector. All components are optimized jointly from the beginning of Reasoner pre-training.

  • Conceptually, if \(h_v=P(E_{\mathrm{ViT}}(x_v))\) is the projected visual representation, the model does not first optimize only \(P\) while keeping \(E_{\mathrm{ViT}}\) and the LLM fixed. Instead:

    \[\theta_{\mathrm{LLM}}, \theta_{\mathrm{ViT}}, \theta_P\]
    • are all trainable from the start.

Reasoner Pre-Training Objective

  • The Reasoner uses conventional autoregressive next-token prediction over its multimodal sequence. For output language tokens

    \[y_1,\ldots,y_T\]

    • the underlying objective is

      \[\mathcal{L}_{\mathrm{AR}} = -\sum_{t=1}^{T} w_t \log p_\theta \left( y_t \mid y_{<t}, x_{\mathrm{vision}} \right)\]

  • Cosmos 3 trains for two epochs over the complete Reasoner pre-training mixture using a no-replacement sampler that uniformly concatenates the component datasets.

  • Training sequences are capped at \(16{,}384\) tokens, with per-sample limits of \(2{,}048\) image tokens and \(8{,}192\) video tokens. The report connects these limits to the need for efficient reasoning and low-latency inference in Physical AI applications.

Square-Root Normalized Loss

  • Naively averaging token loss can cause long sequences to dominate optimization simply because they contain more supervised tokens. Cosmos 3 instead applies square-root normalized per-token loss weighting to balance examples of different lengths.

  • For an example containing \(N\) supervised tokens, the principle can be understood as scaling its aggregate contribution proportionally to roughly

    \[\frac{1}{\sqrt{N}} \sum_{i=1}^{N}\ell_i\]
    • rather than allowing the contribution to scale linearly with \(N\).

  • The report states that this strategy improved both downstream benchmark performance and training stability.

Reasoner Pre-Training Hyperparameters

  • Optimization uses AdamW with separate learning rates for the language/projector path and vision encoder:
Component Peak learning rate
Language model \(5\times10^{-5}\)
Multimodal projector \(5\times10^{-5}\)
ViT \(5\times10^{-6}\)

  • The ViT therefore receives a learning rate one order of magnitude below the language and projection components.

  • The remaining settings are:

    \[(\beta_1,\beta_2) = (0.9,0.999)\] \[\lambda_{\mathrm{wd}} = 0.05\]

    • and global gradient clipping at:

      \[\Vert g\Vert_2\leq1.0\]

  • The learning rate uses a \(10\%\) linear warm-up followed by cosine decay to \(0.1\times\) the peak value.

Reasoner Supervised Fine-Tuning

  • Reasoner SFT changes not only the data but also how datasets are sampled. Pre-training approximately treats the mixture uniformly across epochs. SFT instead assigns each dataset a fixed sampling budget according to its importance, quality, and scale. This makes the sampling distribution itself part of the specialization strategy.

  • Cosmos 3 also preserves a filtered portion of the pre-training distribution. The sampling-budget ratio is

    \[\text{pre-training}:\text{SFT} = 1:4\]

  • The purpose is to specialize toward Physical AI without erasing broad visual reasoning and instruction-following behavior. An additional \(800\text{K}\)-sample instruction-following dataset is included to stabilize conversational behavior during adaptation.

  • SFT runs for \(8{,}200\) iterations with a global batch size of \(512\).

  • The learning rates are reduced relative to pre-training:

    \[\eta_{\mathrm{LLM/projector}} = 10^{-5}\] \[\eta_{\mathrm{ViT}} = 10^{-6}\]

  • AdamW uses:

    \[(\beta_1,\beta_2) = (0.9,0.95)\]
    • weight decay \(0.1\), and global gradient clipping at \(1.0\). The schedule uses \(1{,}000\) warm-up steps followed by cosine decay to \(0.1\times\) the peak learning rate.

Transferring the Reasoner into the Generator

  • Once Reasoner training is complete, its weights initialize the Generator pathway. This is possible because the two pathways use compatible transformer-block structures even though their attention semantics and objectives differ.

  • The initialization can be summarized as:

    \[\theta_{\mathrm{Generator}}^{(0)} \leftarrow \theta_{\mathrm{Reasoner}}\]

  • The Generator therefore begins with representations already encoding language, visual semantics, objects, spatial relationships, and aspects of physical-world reasoning. It then learns a continuous denoising objective instead of next-token prediction.

  • This transfer is one of the important architectural-training interactions in Cosmos 3:

    \[\boxed{ \text{learn to understand} \rightarrow \text{reuse those representations to learn to generate} }\]
    • rather than training understanding and generation as entirely unrelated systems.

Generator Training Objective

  • All Generator modalities use a common rectified-flow objective. Let \(x_0\) denote a clean target latent, which can correspond to an image, video, audio segment, or action trajectory. Gaussian noise is sampled as:

    \[\epsilon\sim\mathcal{N}(0,I)\]

  • At noise level \(\sigma\in[0,1]\), Cosmos 3 constructs:

    \[x_\sigma = \sigma\epsilon + (1-\sigma)x_0\]

  • At \(\sigma=0\) the sample is completely clean:

    \[x_\sigma=x_0\]

  • At \(\sigma=1\), it is pure Gaussian noise:

    \[x_\sigma=\epsilon\]

  • The trajectory between them is a straight interpolation. Its target velocity is therefore constant:

    \[v^* = \epsilon-x_0\]

  • The Generator learns a velocity field

    \[v_\theta(x_\sigma,\sigma,c)\]

    • where \(c\) represents conditioning information such as language, clean video frames, control signals, or actions. The loss is a masked mean-squared error between predicted and target velocities:

      \[\mathcal{L}_{\mathrm{flow}} = \frac{ \sum_i m_i \left\| v_\theta(x_{\sigma,i},\sigma,c) - (\epsilon_i-x_{0,i}) \right\|_2^2 }{ \sum_i m_i }\]

  • Conditioning tokens have \(m_i=0\) while tokens the model must generate have \(m_i=1\).

  • Consequently, an image-to-video example can include clean initial frames in the same diffusion sequence as noisy future frames without forcing the model to reconstruct the conditioning frames.

Per-Modality Noise Sampling

  • Although the Generator uses one general flow-matching objective, Cosmos 3 does not require every modality in a multimodal sample to share the same noise level. Noise time is sampled independently for image/video, audio, and action modalities.

  • Conceptually:

    \[\sigma_v \sim p_v(\sigma)\] \[\sigma_a \sim p_a(\sigma)\] \[\sigma_{\mathrm{act}} \sim p_{\mathrm{act}}(\sigma)\]

  • This means, for example, that an audiovisual sequence can contain video at one denoising stage and audio at another. The model must learn cross-modal conditioning across varying uncertainty levels rather than assuming synchronized diffusion time.

  • Cosmos 3 uses logit-normal noise sampling for image, audio, and action batches and mode sampling for video. The latter follows the strategy used by Waver, with NVIDIA reporting better video generation quality from mode sampling.

Rectified-Flow Noise Shifting

  • The sampled diffusion time is further transformed using a shift parameter \(s\). With \(\bar t=1-t\).

  • Cosmos 3 defines:

    \[\sigma = \frac{s\,t} {1+(s-1)\bar t}\]

  • For \(s>1\), the distribution is biased toward higher noise levels.

  • The shift is resolution-dependent because higher-resolution generation benefits from a different noise distribution. During Generator pre-training:

    \[s_{256}=1\] \[s_{480}=3\] \[s_{720}=5\]

  • During mid-training these are increased to:

    \[s_{256}=3, \qquad s_{480}=5, \qquad s_{720}=10\]
    • to better address dynamics and high-resolution artifacts.

Multi-Resolution Generator Pre-Training

  • Generator pre-training jointly covers images, video, and audio and does not specialize the model to a single spatial resolution. Cosmos 3 simultaneously trains at 256p, 480p, and 720p across five aspect ratios and variable temporal lengths.

  • The following figure (source) shows the multi-resolution training configuration, fixed-context token packing, image-video sampling proportions, and video conditioning modes.

  • The supported configurations are:
Tier Video FPS Maximum frames Example 16:9 resolution Source restriction Pre-training noise shift
256p 10-30 400 \(320\times192\) Any source resolution \(s=1\)
480p 10-30 400 \(832\times480\) Native resolution \(\geq480p\) \(s=3\)
720p 10-30 300 \(1280\times720\) Native resolution \(\geq720p\) \(s=5\)
  • The highest tier is limited to \(300\) frames because of sequence-length constraints, while 256p and 480p support up to \(400\) frames.

Fixed-Budget Sequence Packing

  • Multi-resolution training creates highly variable sequence lengths. Padding every example to the longest sequence would waste substantial accelerator compute. Cosmos 3 instead packs variable-length samples into a fixed context budget of \(74{,}000\) tokens.

  • For example, a packed sequence might contain several 256p clips, fewer 480p clips, or one large 720p clip:

    \[S_{\mathrm{pack}} = [ S_1; S_2; \ldots; S_k ]\]

    • subject to:

      \[\sum_{i=1}^{k}|S_i| \leq74{,}000\]

  • This lets batches maintain a relatively constant computational footprint without padding every sample to the maximum video length.

  • The training mixture is \(80\%\) video and \(20\%\) image. At the aggregate level, the reported resolution mixture is \(20\%\) Video-256, \(40\%\) Video-480, \(20\%\) Video-720, with the remaining \(20\%\) divided evenly across the three image resolutions.

Learning Multiple Generation Modes Through Corruption

  • Cosmos 3 does not need separate models for text-to-image, text-to-video, image-to-video, and video-to-video. Instead, the training mode is controlled by how many initial visual latent frames remain clean.

  • Let a latent video be:

    \[X \in \mathbb{R}^{C\times T\times H\times W}\]

    • with:

      \[T = T_{\mathrm{cond}} + T_{\mathrm{noised}}\]

  • The first \(T_{\mathrm{cond}}\) latent frames remain clean. Only the remaining \(T_{\mathrm{noised}}\) frames receive noise and contribute to the denoising loss.

  • This yields four modes:

Mode Conditioning
T2I Text only; one visual frame generated
T2V \$$T_{\mathrm{cond}}=0\$$
I2V \$$T_{\mathrm{cond}}=1\$$ latent frame
V2V \$$T_{\mathrm{cond}}=2\$$ latent frames, corresponding to the first five video frames

  • The overall sampling proportions are

    \[20\%\ \mathrm{T2I}\] \[56\%\ \mathrm{T2V}\] \[16\%\ \mathrm{I2V}\] \[8\%\ \mathrm{V2V}\]

  • For T2V, duration, FPS, and timestamps are added to the structured JSON caption. This makes temporal extent an explicit conditioning variable rather than an implicit consequence of tensor shape.

FPS-Aware Training

  • Training videos are sampled at varying frame rates. A raw token offset of one therefore does not represent a fixed physical time interval.

  • Cosmos 3 addresses this both architecturally and through conditioning. The 3D MRoPE temporal coordinate is modulated according to physical time using a base rate of \(24\text{ FPS}\), as discussed in the architecture section. In parallel, FPS and duration are explicitly included in the textual conditioning.

  • The Generator can therefore distinguish between requests such as a short high-frame-rate sequence and a longer low-frame-rate sequence even if their latent token counts are similar.

Freezing the Reasoner During Generator Pre-Training

  • During Generator pre-training, only generation-specific parameters are updated. The Reasoner tower remains frozen.

  • Thus,

    \[\nabla_{\theta_{\mathrm{Reasoner}}} \mathcal{L}_{\mathrm{flow}} = 0\]

    • while:

      \[\nabla_{\theta_{\mathrm{Generator}}} \mathcal{L}_{\mathrm{flow}} \neq 0\]

  • This prevents the large-scale diffusion objective from overwriting the language and visual-understanding capabilities already established by Reasoner training.

  • Generator pre-training uses FusedAdamW with:

    \[\eta=10^{-4}\] \[(\beta_1,\beta_2) = (0.9,0.99)\] \[\lambda_{\mathrm{wd}}=0.05\]
    • and gradient clipping at \(1.0\). The learning rate decays linearly to \(0.30\times\) its peak after warm-up. A \(10\%\) text-dropout rate enables classifier-free guidance during inference.

Pre-Training Compute Scale

  • The reported token counts illustrate the scale of Generator training: \(\text{Cosmos3-Nano} = 31.05\text{T tokens}\) trained using \(1{,}024\ \text{NVIDIA GB200 GPUs}\) while \(\text{Cosmos3-Super} = 17.86\text{T tokens}\) uses \(2{,}048\ \text{GB200 GPUs}\).

  • The larger model therefore does not simply consume proportionally more tokens. It is trained with substantially more compute per token because of its much larger parameter count.

Generator Mid-Training

  • Pre-training establishes general image, video, and audio priors. Mid-training turns this into a more explicitly Physical AI-oriented model.

  • The stage has two goals:

    \[\boxed{ \text{domain specialization} + \text{multimodal integration} }\]

  • Domain specialization increases the density of robotics, driving, human activity, physical interaction, simulation, and other difficult physical-world examples. Multimodal integration adds action and control signals while retaining the original image, video, and audio generation modes.

  • The training mixture becomes:

Stream Modes Share
Image T2I 10%
Video T2V, I2V, V2V 32%
Video + Audio T2(V+Audio), I2(V+Audio), V2(V+Audio) 8%
Action Forward dynamics, inverse dynamics, policy 25%
General Transfer Edge, blur, depth, segmentation 20%
Driving Transfer World-scenario-map control 5%
  • A quarter of mid-training batches therefore involve explicit action modeling, while another quarter involve control-conditioned transfer.

Unified Action Training

  • The action stream uses the same rectified-flow formulation as visual and audio generation. Three different behaviors emerge from the corruption pattern.

  • For forward dynamics \(p(v_{t+1:t+H}\mid v_{\leq t},a_{t:t+H-1})\), actions remain clean while future visual states are denoised.

  • For inverse dynamics \(p(a_{t:t+H-1}\mid v_{t:t+H})\), the observed trajectory remains clean while actions are denoised.

  • For policy generation, future video and action are jointly noisy:

    \[p( a_{t:t+H-1}, v_{t+1:t+H} \mid v_{\leq t}, \text{instruction} )\]

  • The model consequently learns a bidirectional relationship between intervention and consequence: \(\text{action} \rightarrow \text{world evolution}\) and \(\text{world evolution} \rightarrow \text{action}\).

  • This is one of the central differences between Cosmos 3 and a conventional video diffusion model.

Action Loss Scaling

  • Normalized action vectors have substantially fewer elements than video latents and naturally produce smaller per-element MSE values. If the losses were summed without adjustment, action learning could be overwhelmed by the visual objective.

  • Cosmos 3 therefore scales action loss by \(\lambda_{\mathrm{action}}=10\).

  • The mid-training objective can be represented schematically as:

    \[\mathcal{L}_{\mathrm{MT}} = \lambda_v\mathcal{L}_v + \lambda_a\mathcal{L}_a + 10\mathcal{L}_{\mathrm{action}} + \lambda_c\mathcal{L}_{\mathrm{control}}\]
    • where each component uses the common velocity-prediction formulation.

  • Mid-training uses FusedAdamW with learning rate \(10^{-4}\), weight decay \(0.05\), gradient clipping at \(1.0\), and a LambdaLinear schedule with start factor \(0.4\) and cycle length \(100{,}000\).

  • The reported training scale is \(2.4\text{T tokens}\) for Cosmos3-Nano on \(1{,}024\) GB200 GPUs and \(1.9\text{T tokens}\) or Cosmos3-Super on \(2{,}048\) GB200 GPUs.

Text-to-Image Post-Training

  • Cosmos3-Super-Text2Image is produced by specializing the mid-trained Super checkpoint through two SFT stages. No architectural modification is required.

  • The first stage performs broad T2I specialization for \(20{,}000\) steps. Its data mixture is:

    \[45\%\ \text{high-quality real images}\] \[40\%\ \text{synthetic images}\] \[15\%\ \text{text-rendering data}\]

  • The base learning rate is \(10^{-4}\) with \(2{,}000\) warm-up iterations followed by linear decay.

  • The second stage is a much narrower quality-refinement pass:

    \[470{,}000\]
    • ultra-high-quality image-caption pairs for \(2{,}000\) steps.

  • Both stages use images above 720p and a fixed context length of \(70{,}000\) tokens. The objective of the second stage is to further improve aesthetics, prompt adherence, text rendering, and alignment with human preferences.

  • This provides a clear example of Cosmos 3’s specialization philosophy:

    \[\text{omnimodal checkpoint} \xrightarrow{\text{small targeted SFT}} \text{high-quality domain expert}\]

Image-to-Video Post-Training

  • Cosmos3-Super-Image2Video similarly begins from the general mid-trained Super checkpoint. The I2V dataset combines filtered pre-training videos, examples retrieved specifically around known model weaknesses, approximately \(1{,}000\) manually curated high-quality videos, and approximately \(20{,}000\) synthetic videos.

  • Although the video sequences themselves use only the I2V formulation, \(20\%\) of training tokens remain T2I image tokens to preserve semantic alignment.

  • The model is specialized for \(480p\) video containing \(189\) frames, corresponding to roughly \(8\text{ seconds at }24\text{ FPS}\).

  • Training runs for \(10{,}000\) iterations at \(10^{-5}\) learning rate and processes approximately \(50\text{B tokens}\).

Robot Policy Post-Training

  • Robot policy specialization demonstrates the most consequential extension of the generative formulation: the same diffusion Generator can be adapted to emit executable robot actions.

  • The pilot model, Cosmos3-Nano-Policy-DROID, is trained on DROID: A Large-Scale In-the-Wild Robot Manipulation Dataset by Khazatsky et al. (2024). DROID contains approximately \(76\text{K}\) trajectories, \(350\) hours of interaction, \(86\) tasks, and \(564\) scenes using a Franka Panda 7-DoF arm and Robotiq 2F-85 gripper.

  • For this specialization, Cosmos 3 introduces freshly initialized:

    1. Action encoder

    2. Action-decoding MLP

    3. Action embedding tokens

  • Action-specific parameters receive a \(5\times\) learning-rate multiplier to adapt more quickly than the pretrained backbone.

Policy Inputs

  • The policy receives both proprioception and three visual views. The wrist camera has raw resolution \(360\times640\) while each external camera contributes \(180\times320\).

  • The two external views are placed side by side beneath the wrist view, creating a single \(540\times640\) visual canvas.

  • Thus, multiple robot cameras enter the existing visual encoder through spatial packing rather than requiring independent camera-specific transformer towers.

Joint Action and Video Prediction

  • The policy predicts \(32\) future absolute joint-position actions together with auxiliary RGB future frames. It operates at \(15\text{ Hz}\).

  • The predicted action horizon therefore spans approximately \(\frac{32}{15} \approx 2.13\text{ seconds}\).

  • Joint prediction makes the policy objective more than pure action regression:

    \[(\text{observation},\text{instruction},\text{proprioception}) \rightarrow (\text{future actions},\text{future visual states})\]

  • The visual future serves as an auxiliary world-modeling target while the action trajectory provides the executable control output.

Fast Diffusion for Closed-Loop Control

  • A diffusion policy is only practical if sampling latency is low enough for control. Cosmos3-Nano-Policy-DROID therefore uses only \(4\) diffusion steps at inference with shifted noise schedule \(s=5\).

  • Classifier-Free Guidance (CFG) uses scale \(3\) and CFG branches are executed in parallel. Video latent decoding is skipped because the auxiliary predicted future video is unnecessary for actually executing the action.

  • These optimizations allow the policy server to run on two NVIDIA RTX Pro 6000 GPUs. A downstream joint-position controller executes the predicted \(32\)-action chunks at \(15\text{ Hz}\).

  • The resulting execution loop can be represented as:

    \[o_t,s_t,\text{instruction} \rightarrow \text{Cosmos 3} \rightarrow \hat a_{t:t+31} \rightarrow \text{joint controller} \rightarrow \text{robot} \rightarrow o_{t+1}\]

The Training Strategy as a Whole

  • Cosmos 3’s training recipe is best understood as progressive capability acquisition rather than one monolithic multimodal optimization run:

    \[\boxed{ \begin{aligned} \text{Reasoner PT} &\rightarrow \text{general multimodal understanding} \\ \text{Reasoner SFT} &\rightarrow \text{Physical AI reasoning} \\ \text{Generator PT} &\rightarrow \text{image/video/audio generation} \\ \text{Generator MT} &\rightarrow \text{action + control + Physical AI dynamics} \\ \text{Post-training} &\rightarrow \text{specialized generators and policies} \end{aligned} }\]

  • Three design choices tie these stages together. First, Reasoner weights initialize the Generator, explicitly transferring understanding into generation. Second, rectified flow provides one continuous-generation objective for vision, audio, and action instead of introducing a different training framework for each modality. Third, task identity is frequently represented through conditioning and corruption patterns rather than architectural changes.

  • This means that text-to-video generation, video continuation, audiovisual generation, forward dynamics, inverse dynamics, control-conditioned transfer, and robot policy learning increasingly become different views of the same underlying optimization problem:

    \[\boxed{ \text{given the known parts of a multimodal physical trajectory, reconstruct the unknown parts} }\]

  • That formulation is the training-level counterpart to Cosmos 3’s architectural goal of treating reasoning, generation, simulation, and action as capabilities of one Physical AI foundation model.

Infrastructure

  • Cosmos 3 is supported by an integrated infrastructure stack spanning the full model lifecycle, from raw multimodal data to trained checkpoints, serving, and continuous evaluation. Cosmos 3: Omnimodal World Models for Physical AI by NVIDIA et al. (2026) organizes the system into four pillars: data engineering, large-scale distributed training, low-latency serving, and benchmarking and validation.

  • The pipeline can be summarized as:

    \[\text{raw multimodal data} \rightarrow \text{curated WebDatasets} \rightarrow \text{distributed training} \rightarrow \text{checkpoints} \rightarrow \\ \begin{cases} \text{serving}\\ \text{benchmarking} \end{cases}\]

  • The important systems idea is that these are not independent components. Data curation has to continuously feed training at very large scale, training has to support highly variable multimodal sequences, and the same checkpoints must move quickly into inference and regression evaluation.

  • The following figure (source) shows the four-pillar infrastructure architecture.

SILA: Scalable Infrastructure for Large-Scale Data Processing and Annotation

  • Cosmos 3 begins from tens of billions of candidate images and videos. NVIDIA built SILA, or Scalable Infrastructure for Large-scale data processing and Annotation, to manage the repeated processing required to turn this raw corpus into training data. SILA combines storage, metadata, distributed processing, semantic retrieval, dataset inspection, and annotation into one system.

  • A typical sample may pass through:

    \[\text{ingestion} \rightarrow \text{transcoding} \rightarrow \text{embedding} \rightarrow \text{deduplication} \rightarrow \text{filtering} \rightarrow \text{tagging} \rightarrow \\ \text{captioning} \rightarrow \text{sharding}\]

  • This process is iterative. As captioners, embeddings, filters, or model requirements improve, some stages need to be rerun without recomputing the entire pipeline.

  • SILA therefore separates pipeline logic from infrastructure mechanics. Researchers specify typed processing stages, including the columns consumed and produced, while the platform manages sharding, distributed execution, fault tolerance, checkpointing, metadata updates, and asset registration.

A Unified Columnar Data Layer

  • Earlier Cosmos infrastructure used a table-per-pipeline design in which individual pipelines wrote to separate PostgreSQL tables and outputs were synchronized into Databricks using Change Data Capture. At increasing scale, reconstructing a sample’s complete state required increasingly expensive joins across pipeline-specific tables.

  • SILA instead uses Lance as a unified columnar data representation. A row represents one data sample, while typed columns represent progressively added curation signals such as:

    \[\{ \text{caption}, \text{embedding}, \text{quality score}, \text{taxonomy}, \text{annotation}, \ldots \}\]

  • A pipeline therefore enriches the existing sample:

    \[x_i \rightarrow (x_i,c_i) \rightarrow (x_i,c_i,e_i) \rightarrow (x_i,c_i,e_i,q_i) \rightarrow\cdots\]
    • rather than creating a new relational entity at every processing stage.

  • This makes incremental recomputation particularly natural. If a new quality model is introduced, SILA can add or recompute the corresponding column while reusing previously generated captions, embeddings, and annotations.

Fragment-Level Distributed Coordination

  • At Cosmos 3 scale, many workers and independent jobs may operate over the same evolving dataset. SILA coordinates work at the Lance-fragment level rather than letting workers randomly select samples.

  • A worker discovers an unfinished fragment and obtains a time-limited lease:

    \[F_j \xrightarrow{\text{lease}} W_k\]

  • The worker periodically emits heartbeats. If the worker fails or is preempted, its lease eventually expires:

    \[\text{heartbeat stops} \rightarrow \text{lease expires} \rightarrow F_j\text{ becomes reclaimable}\]

  • Another worker can then resume the fragment without manual intervention.

  • Individual fragments may require hours of model inference, so SILA subdivides a claimed fragment into smaller processing segments. Completed segments are durably checkpointed, but the fragment becomes visible to downstream consumers only after all segments complete and the system atomically commits the fragment.

  • This separates

    \[\text{recovery granularity} \neq \text{visibility granularity}\]

  • A failed job can therefore restart from its most recent completed segment without exposing partially processed fragment state.

Staged Ray Execution

  • Multimodal curation mixes very different workloads. Video decoding is primarily an I/O and CPU problem, VLM captioning is GPU-heavy, writing results is storage-bound, and committing metadata has yet another resource profile.

  • SILA executes these operations through a staged Ray pipeline rather than one homogeneous worker pool. Framework-managed stages handle loading, writing, and committing, while user-defined preprocessing, compute, and postprocessing stages run in independently sized Ray actor pools.

  • Conceptually,

    \[\text{load} \rightarrow \text{preprocess} \rightarrow \text{GPU inference} \rightarrow \text{postprocess} \rightarrow \text{write} \rightarrow \text{commit}\]

  • Backpressure is applied between stages. If GPU inference becomes the bottleneck, upstream decoding is prevented from producing an unbounded queue of intermediate objects and forcing the Ray object store to spill excessively to disk.

Node-Local Model Serving for Curation

  • Many curation operations themselves invoke foundation models. Captioning, embedding generation, scoring, and taxonomy assignment can therefore look like large inference-serving workloads nested inside a distributed ETL pipeline.

  • Instead of routing all requests to a centralized inference cluster, SILA can launch node-local model servers using systems such as vLLM. Pipeline workers communicate directly with the local endpoint.

  • This gives a topology approximately like:

    \[\begin{array}{ccc} \text{Node}_1 & & \text{Node}_2\\ \downarrow && \downarrow\\ \text{local vLLM} && \text{local vLLM}\\ \uparrow && \uparrow\\ \text{Ray workers} && \text{Ray workers} \end{array}\]

  • The design avoids turning a central inference endpoint into a throughput bottleneck and allows curation jobs to exploit fragmented GPU capacity.

Opportunistic GPU Utilization

  • Data curation does not necessarily receive a dedicated contiguous GPU cluster. It must coexist with large training jobs whose resource consumption changes over time.

  • SILA therefore decomposes curation into fine-grained distributed jobs that can consume available capacity opportunistically. The report describes execution through environments including NVIDIA DGX Cloud Lepton and Slurm.

  • Instead of requiring \(N\) simultaneously available GPUs before starting a large monolithic job, work can progress incrementally as smaller GPU groups become available.

  • This is especially useful because data curation is persistent. New filters, annotations, captions, and embeddings may need to be produced throughout model development rather than during a single preprocessing phase.

Agentic Data-Pipeline Operations

  • SILA also exposes operational functionality through CLIs, reusable skills, and structured job metadata so long-running agents can monitor curation jobs.

  • The report describes agents that periodically inspect logs and execution metadata, identify unhealthy or stalled workers, relaunch failed stages, verify coverage, trigger incremental recomputation after filtering or model changes, and notify engineers about failures and recovery.

  • This creates an operational loop of the form:

    \[\text{observe} \rightarrow \text{diagnose} \rightarrow \text{recover} \rightarrow \text{verify} \rightarrow \text{continue}\]

  • According to NVIDIA’s reported production measurements, moving to SILA reduced pipeline startup latency from roughly 30 to 60 minutes to approximately 5 minutes, while the overall architecture achieved about a \(10\times\) throughput improvement over its predecessor. Individual production stages could process billions of row-level annotations per day.

Embedding Storage and Semantic Retrieval

  • Cosmos 3 data curation also requires similarity search across billions of multimodal samples for retrieval, clustering, and deduplication.

  • A previous architecture separated embeddings into a vector database while keeping metadata in relational storage. That creates a synchronization problem:

    \[\text{vector state} \leftrightarrow \text{metadata state}\]

  • As embedding models and filtering metadata change, both systems have to remain consistent.

  • SILA instead stores embeddings alongside the underlying sample metadata in Lance and uses LanceDB to build indexes directly over the dataset.

  • The production configuration described for Cosmos 3 contains \(4096\) dimensional embeddings across tens of billions of rows. Approximate nearest-neighbor search uses IVF-PQ with cosine similarity and \(64\text{K}\) IVF partitions.

  • The lookup problem is therefore approximately:

    \[j^* = \arg\max_j \frac{ e_q^\top e_j }{ \|e_q\|_2\|e_j\|_2 }\]
    • but IVF first restricts the candidate space to relevant coarse partitions, while product quantization compresses stored vectors and accelerates approximate similarity computation.

  • This same infrastructure underlies semantic retrieval, clustering, and deduplication operations used in the data curriculum described earlier.

Training Infrastructure

  • Cosmos 3 presents several unusual systems challenges for distributed training. The Reasoner and Generator have different attention patterns, multimodal examples vary dramatically in sequence length, video tokenization can itself become a computational bottleneck, and the Generator operates with contexts reaching approximately \(74{,}000\) tokens.

  • The training stack consequently combines optimized multimodal data loading, custom variable-length attention, Hybrid Sharded Data Parallelism, context parallelism, selective activation checkpointing, compilation, optimized VAE tokenization, and asynchronous checkpointing.

Joint Multimodal Data Loading

  • The Generator’s training mixture includes image, video, video-audio, action, and transfer streams with different sample sizes and token footprints. A conventional batch loader can easily create imbalance: one stream may stall while another overproduces examples, or batches may underfill the available token budget.

  • Cosmos 3 uses a Joint Data-Loader that treats training as token-budget packing rather than simply choosing a fixed number of examples.

  • For a target token capacity \(B\), it seeks samples satisfying approximately:

    \[\sum_i L_i \lesssim B\]
    • where \(L_i\) is the sequence length of sample \(i\).

  • The data loader tracks per-stream buffers and uses look-ahead to improve packing efficiency while maintaining the configured sampling ratios. This matters because a 720p video and a single image can differ dramatically in token count even though each is technically one “sample.”

  • The loader also explicitly prewarms every stream before the first training iteration. Worker processes are spawned, file metadata is cached, and deserialization paths are exercised before a distributed barrier releases training. This avoids a particularly damaging failure mode in which the slowest rank experiences large first-batch latency while other ranks have already entered an NCCL collective.

Training Data Observability

  • The Joint Data-Loader emits packing statistics on every iteration. These records are aggregated across distributed ranks and logged to Weights & Biases.

  • Tracked signals include the empirical versus target sampling ratio for each stream, number of samples packed per iteration, buffer occupancy, buffer wait time, look-ahead saturation, and token-budget utilization.

  • These metrics target a class of distributed-training failures that may not crash the job but silently alter the effective training distribution. A stalled data stream, for example, could change \(p_{\mathrm{actual}}(m)\) away from the intended \(p_{\mathrm{target}}(m)\) for modality \(m\) while the training loop continues normally.

Two-Way Flat Attention

  • The Mixture-of-Transformers architecture creates an unusual attention problem. Reasoner tokens require causal self-attention:

    \[R_i \rightarrow R_{\leq i}\]

    • while Generator tokens require bidirectional attention over both the Reasoner and Generator context:

      \[G_i \rightarrow [R,G]\]

  • A generic masking operator can express this pattern correctly, but the report finds that doing so leaves substantial hardware performance unused.

  • Cosmos 3 therefore implements a custom two-way flat attention mechanism using two variable-length scaled dot-product attention calls per layer.

  • The following figure (source) shows the two kernel calls and how Reasoner and Generator sequences from multiple packed samples are represented.

  • For the Reasoner, packed sequences \(R_0,R_1,\ldots,R_n\) are processed with standard causal variable-length SDPA.

  • For the Generator, queries contain only Generator tokens, while keys and values are interleaved by sample:

    \[[R_0,G_0,R_1,G_1,\ldots,R_n,G_n]\]

  • The sequence boundaries are supplied through cumulative sequence-length arrays. Thus, Generator queries for sample \(i\) see \([R_i,G_i]\) but cannot cross into another sample’s block.

  • The result is a rectangular block-diagonal attention pattern:

    \[G_i \longrightarrow [R_i,G_i]\]

  • NVIDIA reports that this specialized formulation improves end-to-end Cosmos3-Nano training throughput by \(22\%\) relative to a FlexAttention-based baseline.

Hardware-Specific Attention Backends

  • The variable-length attention interface dispatches to different kernels depending on GPU architecture.

  • On Hopper GPUs such as H100 and H200, Cosmos 3 uses FlashAttention-3 by Shah et al. (2024), which exploits Hopper’s WGMMA instructions and Tensor Memory Accelerator.

  • On Blackwell GB200 GPUs, Cosmos 3 uses NATTEN, whose variable-length kernels are implemented through CUTLASS and tuned for Blackwell’s newer tensor cores and memory hierarchy.

  • Both sit behind the same dispatch interface, allowing the higher-level transformer implementation to remain unchanged.

Hybrid Sharded Data Parallelism

  • Reasoner and Generator training combine Hybrid Sharded Data Parallelism, or HSDP, with Context Parallelism.

  • HSDP partitions

    \[\{ \text{parameters}, \text{gradients}, \text{optimizer states} \}\]
    • within replica groups while replicating those groups across the cluster. The purpose is primarily model-state memory reduction.

  • Context Parallelism addresses a different bottleneck:

    \[\text{activation memory} \propto \text{sequence length}\]

  • This becomes especially important for the Generator’s long multimodal contexts. The HSDP and CP degrees can therefore be chosen independently according to model size, sequence length, available GPU memory, and topology.

Ulysses Context Parallelism

  • Cosmos 3 implements context parallelism using the DeepSpeed-Ulysses approach by Jacobs et al. (2023).

  • Outside attention, tokens are partitioned along the sequence dimension:

    \[X = [X^{(1)},X^{(2)},\ldots,X^{(P)}]\]

  • Before attention, an all-to-all operation redistributes Q, K, and V from sequence sharding to head sharding. Each GPU then receives the complete sequence but only a subset of attention heads:

    \[\text{sequence-sharded} \xrightarrow{\mathrm{all\text{-}to\text{-}all}} \text{head-sharded}\]

  • Attention can then execute locally. A second all-to-all restores sequence sharding:

    \[\text{head-sharded} \xrightarrow{\mathrm{attention}} \text{head-sharded output} \xrightarrow{\mathrm{all\text{-}to\text{-}all}} \text{sequence-sharded output}\]

  • The two-way Reasoner/Generator interleaving is performed after redistribution, when each rank already holds the complete sequence for its assigned heads. As a result, the same variable-length attention kernels can be reused without implementing special context-parallel variants.

  • The maximum CP degree is bounded by the number of query heads \(CP_{\max}=32\) for Cosmos3-Nano and \(CP_{\max}=64\) for Cosmos3-Super.

Why Ulysses Instead of Ring Attention

  • The report explicitly discusses ring attention as an alternative. For Cosmos 3, it would require materializing the combined interleaved Reasoner-Generator sequence before distributing it across context-parallel ranks.

  • That conflicts with the architecture’s desire to independently shard the two pathways and makes the mixed causal/bidirectional masks more difficult to implement under the rotating ring schedule.

  • Ulysses instead permits the two streams to remain independently sharded until after head redistribution. NVIDIA also cites the favorable all-to-all bandwidth available on NVLink-connected systems as a reason for the choice.

Selective Activation Checkpointing

  • Standard transformer activation checkpointing reduces memory by storing only a limited number of forward activations and recomputing discarded operations during backpropagation.

  • If a transformer block computes \(h_{l+1}=F_l(h_l)\) standard block-level checkpointing may preserve only \(h_l\). During backward, \(F_l\) is executed again to regenerate its internal activations.

  • This dramatically reduces activation memory but, according to the Cosmos 3 report, adds roughly \(33\%\) more forward-equivalent FLOPs.

  • Cosmos 3 instead uses Selective Activation Checkpointing. Intermediate tensors are ranked according to a cost-benefit criterion:

    \[\rho(o) = \frac{ \text{FLOPs required to recompute }o }{ \text{bytes required to retain }o }\]

  • Operations with high \(\rho\) are preferentially stored until the activation-memory budget is exhausted.

  • Attention outputs are especially attractive because attention recomputation grows quadratically with sequence length, while the output activation itself grows only linearly with sequence length and hidden dimension.

  • For Cosmos3-Nano at a \(74{,}000\)-token batch budget, retaining attention outputs through SAC produces a reported \(13\%\) end-to-end training-throughput improvement with no change in numerical results.

Compiling the Transformer

  • Cosmos 3 applies torch.compile with \(\texttt{fullgraph=True}\) and \(\texttt{dynamic=True}\).

  • Full-graph compilation reduces CPU overhead and enables operator fusion, while dynamic shapes are required because multimodal batches can have substantially different sequence lengths from one iteration to another.

  • NVIDIA reports a \(41\%\) training-throughput improvement for Cosmos3-Nano Generator training from this optimization.

Video Tokenization as a Systems Bottleneck

  • The Generator does not train directly on raw pixels. Video must first pass through the frozen Wan2.2 VAE described earlier.

  • For smaller Cosmos3-Edge and Cosmos3-Nano models, NVIDIA found that this preprocessing could dominate the forward-pass time. The transformer had become fast enough that the supposedly auxiliary tokenizer was now on the critical path.

  • The solution combines larger encoding chunks, compilation, and static-shape specialization.

Chunked VAE Encoding

  • The Wan2.2 causal tokenizer normally processes a one-frame prime chunk followed by groups of four pixel frames per latent chunk. That granularity does not provide enough work per kernel launch to fully utilize the GPU.

  • Cosmos 3 batches more frames into each VAE invocation. The empirically selected chunk sizes are

    1. 68 frames at 256p.

    2. 24 frames at 480p.

    3. 12 frames at 720p.

  • The decreasing chunk size reflects the larger memory cost of each frame at higher spatial resolution.

Compiling the Video Tokenizer

  • The tokenizer is also compiled with torch.compile, which NVIDIA reports reduces encoding latency by an additional \(52\%\).

  • Static compilation introduces another problem. Cosmos 3 has \(3\) resolutions, \(5\) aspect ratios per resolution, and \(3\) tokenizer call modes.

  • Thus, \(3\times5\times3 = 45\) static graphs must be compiled.

  • Serially compiling all 45 graphs on every training rank adds roughly 15 minutes of startup time. Cosmos 3 instead distributes compilation across data-parallel ranks using AOTInductor.

  • The following figure (source) shows this sharded compilation process.

  • With at least 45 ranks, each rank compiles one graph:

    \[g \mapsto \operatorname{rank} ( g\bmod N )\]

  • Artifacts are written to shared storage, after which all ranks load the complete set.

  • This reduces tokenizer compilation warm-up from approximately \(15\text{ minutes}\) to \(<1\text{ minute}\).

  • For arbitrary frame counts, videos are right-padded to a compatible static shape before encoding and the resulting latent sequence is cropped back to its true temporal length. For datasets with known fixed frame counts, such as some robot-action datasets, compilation specializes directly to those exact shapes and avoids the padding overhead.

Asynchronous Checkpointing

  • Large distributed models can lose substantial training time if every checkpoint requires all workers to stop while model state is written to storage.

  • Cosmos 3 overlaps checkpointing with training. Following the approach used in TorchTitan, checkpoint I/O is routed through a dedicated Gloo process group instead of the NCCL communicator used for training collectives.

  • Conceptually,

    \[\begin{array}{rcl} \text{GPU training} &:& \text{NCCL}\\ \text{checkpoint I/O} &:& \text{Gloo} \end{array}\]
    • execute concurrently.

  • A long-lived child process handles CPU-side checkpoint operations and storage writes while the GPUs continue training.

  • Relative to synchronous checkpointing every 30 minutes, NVIDIA reports an end-to-end training-time reduction of approximately \(4\%\) for Cosmos3-Nano and \(9\%\) for Cosmos3-Super.

  • The checkpoint save plan is also computed once and reused because it depends deterministically on the topology of the state dictionary. NVIDIA reports that this save-plan memoization reduces checkpointing overhead by approximately \(60\%\).

Infrastructure as Part of the Model Design

  • The Cosmos 3 infrastructure is notable because several optimizations are direct consequences of the model architecture rather than generic large-model engineering.

  • The MoT architecture creates mixed causal and bidirectional attention, motivating two-way flat attention. Omnimodal training creates highly heterogeneous sequence lengths, motivating token-budget packing and variable-length kernels. Long video contexts make activation recomputation expensive, motivating selective checkpointing. Online video encoding makes the VAE part of the training critical path, motivating chunking and distributed compilation. Continuous large-scale curation makes traditional ETL assumptions impractical, motivating SILA’s fragment-level execution model.

  • The stack can therefore be summarized as:

    \[\boxed{ \begin{aligned} \text{SILA} &\rightarrow \text{continuous billion-scale data curation} \\ \text{Joint Data-Loader} &\rightarrow \text{heterogeneous multimodal packing} \\ \text{Two-way flat attention} &\rightarrow \text{efficient MoT attention} \\ \text{HSDP + Ulysses} &\rightarrow \text{model-state + long-context scaling} \\ \text{SAC + compilation} &\rightarrow \text{memory/compute efficiency} \\ \text{optimized VAE} &\rightarrow \text{high-throughput video tokenization} \\ \text{async checkpointing} &\rightarrow \text{reduced training stalls} \end{aligned} }\]

  • The broader lesson is that Cosmos 3’s model and systems designs are tightly coupled. Omnimodal world modeling introduces workloads that differ substantially from conventional text-only LLM training, and much of the infrastructure is specifically designed around those differences rather than simply scaling a standard LLM training stack.

Evaluation and Results

  • Cosmos 3 is evaluated as a unified Physical AI model rather than as a conventional VLM or video generator. The evaluation therefore spans both sides of the architecture: the Reasoner is tested on multimodal understanding and physical reasoning, while the Generator is tested on image, video, audiovisual, control-conditioned, dynamics, and action-generation tasks. Cosmos 3: Omnimodal World Models for Physical AI by NVIDIA et al. (2026) emphasizes this breadth because the central claim of Cosmos 3 is not simply improved generation quality, but that one omnimodal architecture can support perception, reasoning, simulation, and action.

  • The evaluation can be viewed as measuring four increasingly physical capabilities:

    \[\text{perception} \rightarrow \text{reasoning} \rightarrow \text{world prediction} \rightarrow \text{action}\]

  • A useful aspect of the evaluation methodology is that NVIDIA does not rely on a single aggregate benchmark. Different failure modes require different evaluation mechanisms, including exact-match scoring, geometric metrics, VLM judges, perceptual metrics, reference-based physical measurements, and human evaluation.

Benchmark Infrastructure

  • Evaluation itself is implemented as a reproducible infrastructure layer. Every benchmark run records the model checkpoint, code version, benchmark selection, generation configuration, benchmark-specific parameters, and associated datasets. Generated artifacts are stored separately from evaluation results.

  • For Generator benchmarks, generation and scoring are deliberately decoupled:

    \[\text{checkpoint} \rightarrow \text{generation} \rightarrow \text{stored artifacts} \rightarrow \text{scoring}\]

  • This means a generated video corpus can later be evaluated with a new metric without rerunning an expensive generation job.

  • Reasoner evaluation uses VLMEvalKit together with vLLM, while Generator evaluations can execute through either the native PyTorch path or optimized serving backends. Automated scores, human annotations, run metadata, and generated examples are exposed through a benchmark portal for checkpoint-to-checkpoint inspection.

Reasoner Evaluation

  • The Cosmos 3 Reasoner is evaluated across \(48\) benchmarks divided into four categories:

    \[\{ \text{General}, \text{Robotics}, \text{Smart Infrastructure}, \text{Driving} \}\]

  • This decomposition is important because aggregate VLM scores can obscure capabilities particularly relevant to Physical AI.

General Multimodal Reasoning

  • The general suite contains 19 benchmarks covering visual question answering, spatial reasoning, OCR and document understanding, video reasoning, causal understanding, hallucination resistance, and instruction following.

  • Examples include MMBench, RealWorldQA, CVBench, AI2D, DocVQA, OCRBench-v2, MVBench, VideoPhy2, CausalVQA, MMMU-Pro, HallusionBench, and IFBench.

  • This suite asks whether the model can construct a reliable semantic representation before testing whether that representation can support physical reasoning.

Robotics and Embodied Reasoning

  • The robotics suite contains 17 benchmarks. It focuses more directly on capabilities such as affordance reasoning, feasible-action prediction, spatial grounding, object relationships, trajectory reasoning, manipulation understanding, and multi-view perception.

  • Representative tasks include:

    \[\text{Where is the object?}\] \[\text{Where can it be placed?}\] \[\text{What actions are physically feasible?}\] \[\text{How will the scene change after an action?}\]
    • rather than conventional object recognition alone.

  • The suite includes Cosmos-ER, Cosmos-CS, RefSpatial, VSI-Bench, SparBench, ERQA, RoboSpatialHome, Where2Place, and several RynnBrain tasks, among others.

Smart Infrastructure and Driving

  • Smart-infrastructure evaluation uses VANTAGE-Bench and Traffic Anomaly Reasoning to measure fixed-camera reasoning in settings such as warehouses and transportation systems. Tasks include event verification, visual question answering, pointing, temporal localization, dense captioning, tracking, anomaly detection, and causal reasoning.

  • Driving evaluation includes LingoQA and specialized safety-critical event benchmarks. This shifts evaluation toward questions such as collision understanding, stopping behavior, road-scene reasoning, and causal interpretation of dynamic traffic events.

  • The key point is that Reasoner evaluation progressively moves from generic multimodal understanding toward reasoning about environments in which actions have physical consequences.

Image Generation

  • Cosmos 3 evaluates text-to-image generation along several complementary dimensions rather than relying only on aesthetic preference.

  • The benchmark suite includes UniGenBench for instruction following, CVTG for scene-text generation, HPSv3 for learned human preference, and Aesthetic V2 for prompt-independent visual quality.

  • This decomposition separates:

    • prompt adherence
    • text rendering
    • human preference
    • visual aesthetics

  • That distinction matters for Physical AI. An aesthetically strong generated scene can still be unusable if signage, labels, spatial relationships, or requested objects are incorrect.

  • The specialized Cosmos3-Super-Text2Image checkpoint substantially improves text rendering relative to the general Cosmos3-Super checkpoint. For example, its CVTG-500L GNED increases from \(66.77 \rightarrow 80.88\), while its PNED increases from \(70.97 \rightarrow 89.08\).

  • The result illustrates the role of the post-training stage discussed previously: a general omnimodal checkpoint can be specialized substantially without changing the underlying architecture.

Video Generation

  • Video evaluation is more difficult because visual quality alone does not establish whether generated motion is physically coherent.

  • Cosmos 3 therefore combines automated benchmarks with human evaluation. The principal automated evaluations are PAIBench-G, RBench, and Physics-IQ, while Cosmos HUE and Human World Bench provide human assessments targeted at Physical AI failure modes.

PAIBench-G

  • PAIBench-G evaluates Physical AI video generation across six domains:

    \[\{ \text{Human}, \text{Autonomous Vehicle}, \text{Common Sense}, \text{Robotics}, \text{Physics}, \text{Industry} \}\]

  • The benchmark contains \(1{,}044\) image-text prompt pairs.

  • Its score combines general video quality with domain-specific correctness:

    \[S_{\mathrm{overall}} = 0.5S_{\mathrm{quality}} + 0.5S_{\mathrm{domain}}\]

  • Quality captures properties such as temporal consistency, smooth motion, aesthetics, and video-text alignment. Domain scoring uses VLM-based verification of physical and semantic correctness.

  • Cosmos3-Super records \(80.0\) on the report’s T2V evaluation and \(82.8\) on I2V, while Cosmos3-Nano records \(79.4\) and \(82.7\) respectively.

  • These results are particularly close between Nano and Super, suggesting that increased model scale does not translate uniformly into large gains on every generation metric.

Robotics Video Generation with RBench

  • RBench narrows the evaluation to embodied robotics. It contains \(650\) image-text cases spanning task-oriented scenarios and embodiment-specific scenarios.

  • The latter include:

    \[\{ \text{dual-arm}, \text{humanoid}, \text{single-arm}, \text{quadruped} \}\]

  • Rather than evaluating only visual realism, RBench combines task completion with visual quality:

    \[S_{\mathrm{RBench}} = \frac{ S_{\mathrm{task\ completion}} + S_{\mathrm{visual\ quality}} }{2}\]

  • Cosmos3-Nano and Cosmos3-Super score \(58.4\%\) and \(58.1\%\) respectively, compared with \(50.7\%\) for Wan2.2-A14B in the report’s evaluation.

  • Again, the Nano and Super results are extremely close. Model scale alone is therefore not sufficient to explain embodied video-generation performance.

Physics-IQ: Testing Physical Prediction Directly

  • Many video benchmarks include physical plausibility as one component of a larger quality score. Physics-IQ instead attempts to isolate physical prediction.

  • It covers five physical domains:

    \[\{ \text{solid mechanics}, \text{fluid dynamics}, \text{optics}, \text{thermodynamics}, \text{magnetism} \}\]

  • The benchmark contains \(396\) real-world scenes captured from three fixed viewpoints.

  • In I2V mode, the model sees a switch frame and predicts what happens next. In V2V mode, it receives a three-second conditioning video and predicts the following five seconds.

  • The critical distinction is that generated motion is compared against an actual recorded continuation:

    \[\text{observed initial state} \xrightarrow{\text{model}} \hat{x}_{t+1:t+H}\]

    • versus:

      \[\text{observed initial state} \xrightarrow{\text{physical world}} x_{t+1:t+H}\]

  • Evaluation then measures how closely \(\hat{x}_{t+1:t+H}\) matches \(x_{t+1:t+H}\) in space and time.

  • The benchmark combines spatial overlap, temporal alignment, magnitude-aware spatial agreement, and pixel-level error into a normalized 0 to 100 score.

  • Cosmos3-Super obtains a direct score of \(43.8\) for I2V and \(59.7\) for V2V. With WMReward-based best-of-\(N\) reranking, these rise to \(48.9\) and \(63.4\) respectively.

  • The improvement from reranking is also instructive. The generative distribution can contain physically stronger trajectories than the default single sample exposes, making inference-time selection an additional axis for world-model performance.

Why Human Evaluation Remains Necessary

  • The Cosmos 3 report explicitly argues that automated metrics become less discriminative as video generators improve. A VLM judge may capture broad prompt alignment while missing transient hand deformation, impossible contacts, inconsistent object permanence, or subtle violations of physical dynamics.

  • The report notes that, on the same prompt distribution, T2V models span roughly \(10\) points under human evaluation but only about \(4\) points on the comparable automated PAIBench-G score.

  • Cosmos 3 therefore introduces Cosmos HUE and Human World Bench.

Cosmos HUE

  • Cosmos HUE replaces a conventional holistic Likert judgment with atomic binary verification.

  • Instead of asking an annotator how realistic a video is, the system decomposes the evaluation into questions closer to:

    \[\text{Is object A present?}\] \[\text{Does the requested interaction occur?}\] \[\text{Does the object remain geometrically consistent?}\] \[\text{Is the resulting motion physically plausible?}\]

  • Each question supports

    \[\{\text{Yes},\text{No},\text{Unclear}\}\]

  • Questions are phrased so that “Yes” consistently represents the desired behavior. Two annotators independently evaluate each video-question pair, and disagreements are sent to quality control.

  • Question generation itself uses a three-stage model pipeline:

    \[\text{Domain Strategist} \rightarrow \text{Scene Parser} \rightarrow \text{Auditor}\]

  • The Domain Strategist identifies the relevant Physical AI domain. The Scene Parser constructs a structured representation of the requested or reference scene. The Auditor converts that representation into atomic questions.

  • Human reviewers then examine outputs from strong models and add questions covering failure modes the automated pipeline missed. This produces an iterative benchmark-development loop:

    \[\text{model failures} \rightarrow \text{human discovery} \rightarrow \text{new atomic checks} \rightarrow \text{stronger evaluation}\]

  • The final questions are grouped into four dimensions:

    \[\boxed{ \begin{aligned} &\text{Semantic Alignment}\\ &\text{Physical Laws}\\ &\text{Geometric Reasoning}\\ &\text{Visual Integrity} \end{aligned} }\]

  • This decomposition is especially useful for a world model because it distinguishes a semantically incorrect video from one that follows the instruction but violates geometry or physics.

Human World Bench

  • Human World Bench targets egocentric I2V generation for human manipulation. It contains \(180\) examples sourced from EgoVerse.

  • Annotators independently evaluate two high-level properties: \(S_{\mathrm{instruction}}\) for whether the requested objects and actions appear, and \(S_{\mathrm{physics}}\) for temporal coherence and physical plausibility.

  • The final score is:

    \[S_{\mathrm{HWB}} = \frac{ S_{\mathrm{instruction}} + S_{\mathrm{physics}} }{2}\]

  • The reported HWB scores are \(71.9\) for Cosmos3-Super and \(66.9\) for Cosmos3-Nano, compared with \(60.7\) for Wan2.2-A14B and \(67.8\) for Veo-3.1 in the evaluation reported by NVIDIA.

  • The result is notable because HWB focuses specifically on a known weak point of video generators: physically credible hands, objects, contacts, and manipulation over time.

Audiovisual Generation

  • Cosmos 3 also evaluates whether generated audio corresponds to what occurs visually. This requires more than independent video and audio quality.

  • For a hammer striking an object, for example, a valid generation requires

    \[\text{visible impact at }t \Longleftrightarrow \text{acoustic transient near }t\]

  • A realistic hammer sound several seconds away from the visual collision is still an audiovisual failure.

  • Cosmos-SoundBench contains \(144\) prompts derived from FoleyBench and focuses on non-speech events such as impacts, tools, vehicles, water, ambient scenes, and object interactions.

  • The benchmark defines semantic audiovisual quality as:

    \[S_{\mathrm{AV}} = 0.60S_{\mathrm{audio}} + 0.30S_{\mathrm{AVAlign}} + 0.10S_{\mathrm{visual\ support}}\]

  • A separate production-quality metric measures low-level acoustic fidelity.

  • The final audiovisual quality score is:

    \[S_{\mathrm{AVQ}} = 0.5S_{\mathrm{AV}} + 0.5S_{\mathrm{PQ}}\]

  • The evaluation reveals an interesting distinction. Cosmos 3 is particularly strong on semantic audiovisual grounding and event alignment, while the strongest proprietary systems in the comparison retain an advantage in low-level audio production quality.

  • That distinction matters for Physical AI: correctly associating a sound with a collision, machine, or interaction can carry information about an event even when perceptual audio fidelity is imperfect.

Control-Conditioned Video Transfer

  • Video transfer asks the Generator to synthesize a photorealistic video while obeying an explicit structural control signal such as depth, segmentation, edges, or blur.

  • Cosmos 3 uses two independently adjustable classifier-free guidance weights \(w_c\) for control adherence and \(w_t\) for text adherence.

  • Each denoising step evaluates three conditions: full text plus control, text without control, and control paired with a fixed negative text prompt. The predictions are combined so structural fidelity and caption fidelity can be tuned separately.

  • This is particularly useful because the correct tradeoff differs by task. Depth-conditioned generation may require aggressive geometric adherence, while a weaker edge constraint may permit more freedom in appearance.

PAIBench-C

  • PAIBench-C evaluates four control modalities:

    \[\{ \text{depth}, \text{segmentation}, \text{blur}, \text{edge} \}\]

  • Its \(600\) clips cover robotics, autonomous driving, and egocentric everyday activity.

  • Rather than asking a VLM whether the result “looks controlled,” the relevant control signal is re-extracted from the generated video and compared with the reference:

    \[\text{generated RGB} \xrightarrow{\text{control extractor}} \hat c\] \[\hat c \leftrightarrow c_{\mathrm{reference}}\]

  • Metrics are modality-specific: depth uses scale-invariant RMSE, segmentation uses mIoU, blur uses SSIM, and edges use F1. DOVER provides a complementary perceptual-quality score.

  • This evaluation directly tests one of the architectural claims from earlier sections: Cosmos 3 can represent different controls inside a unified token sequence instead of requiring a separate ControlNet branch for every control modality.

Autonomous-Driving World Transfer

  • AVBench-C specializes transfer evaluation to autonomous driving. The benchmark contains \(486\) single-view driving clips.

  • The control input encodes a rendered world scenario containing map structure and dynamic actors:

    \[c= \{ \text{lanes}, \text{road boundaries}, \text{signals}, \text{signs}, \text{vehicles}, \text{pedestrians}, \ldots \}\]

  • The Generator must transform this structured scenario into a photorealistic video while preserving the geometry and dynamics specified by the control.

  • Evaluation combines visual-odometry drift, dynamic- and static-object correspondence, environment consistency, and human judgments of video quality and lane-line fidelity.

  • This benchmark is particularly representative of the world-model interpretation of Cosmos 3:

    \[\text{structured hypothetical world state} \rightarrow \text{plausible visual realization}\]

What the Evaluation Shows

  • The most useful interpretation of the Cosmos 3 results is not a single leaderboard position. Different parts of the benchmark suite expose different strengths and limitations.

  • The Reasoner shows that Physical AI specialization can coexist with broad multimodal understanding. The video benchmarks show that visual quality and physical correctness are related but distinct. Physics-IQ demonstrates the value of comparing generated futures against real physical outcomes. HUE shows why increasingly capable video generators require failure-specific human evaluation. SoundBench separates audiovisual grounding from raw audio fidelity. Control benchmarks test whether the model can preserve explicit geometry while generating realistic worlds.

  • Taken together, the evaluation hierarchy increasingly approaches the central world-model question:

    \[\boxed{ \text{Given partial observations, instructions, controls, and actions,} }\] \[\boxed{ \text{does the model produce a plausible continuation of the physical world?}\]

  • That is a stricter requirement than either VLM accuracy or generative visual quality in isolation. It also sets up the final section naturally: how the Reasoner and Generator are composed at inference time into practical Physical AI workflows, and what the broader Cosmos 3 design implies for world models and embodied agents.

Inference, Applications, and Key Takeaways

  • Cosmos 3 becomes most distinctive at inference time, where the same foundation model can be configured for several different Physical AI workflows. The Generator supports image, video, audiovisual, transfer, forward-dynamics, inverse-dynamics, and policy generation, while the Reasoner can serve both as a standalone multimodal model and as a planning layer that converts underspecified user intent into structured inputs for generation. Cosmos 3: Omnimodal World Models for Physical AI by NVIDIA et al. (2026) frames this as a general-purpose world-action model rather than a collection of independent task-specific models.

  • Conceptually, the inference stack is

    \[\text{user intent} \rightarrow \text{Reasoner} \rightarrow \text{structured world specification} \rightarrow \text{Generator} \rightarrow \begin{cases} \text{image}\\ \text{video}\\ \text{audio}\\ \text{future state}\\ \text{action} \end{cases}\]

  • This architecture is useful because reasoning about a world and rendering or acting within that world become separate but composable operations.

Generator Inference Envelope

  • The base Cosmos 3 Generator supports a relatively broad inference envelope. Visual generation can operate at \(10\text{--}30\ \mathrm{FPS}\) with sequences ranging from \(5\text{ to }400\) frames, at \(256p,\ 480p,\ \text{or}\ 720p\) and common aspect ratios including:

    \[1{:}1,\quad3{:}4,\quad4{:}3,\quad9{:}16,\quad16{:}9\]

  • This flexibility follows directly from the training curriculum. Rather than training a separate model for each resolution, duration, or aspect ratio, Cosmos 3 exposes those properties as part of the generation configuration.

  • Action generation also spans several temporal regimes. The base Nano and Super models support native control frequencies from roughly \(10\text{--}30\ \mathrm{Hz}\) and horizons ranging from \(16\text{ to }400\) steps, depending on the action task. Post-trained policy models can specialize this interface. Cosmos3-Nano-Policy-DROID, for example, operates at \(15\ \mathrm{Hz}\) and predicts \(32\) future actions per inference call.

Structured Prompting as a Model Interface

  • The Generator was trained on structured JSON captions rather than only short natural-language prompts. These representations explicitly describe properties such as subjects, background, lighting, aesthetics, cinematography, actions, state transitions, camera motion, and temporal segments.

  • Inference works best when it recreates this training-time structure.

  • The pipeline is therefore not simply:

    \[\text{short prompt} \rightarrow \text{Generator}\]

  • Instead, Cosmos 3 uses:

    \[\boxed{ \text{short prompt} \rightarrow \text{prompt upsampler} \rightarrow \text{structured scene program} \rightarrow \text{Generator} }\]

  • The structured representation also contains explicit media controls such as duration, FPS, height, width, and aspect ratio.

  • A request such as: “A robot picks up a red cup from a table.”

  • can therefore be expanded conceptually into

    \[\begin{aligned} \text{Scene} &: \text{robot, table, cup, environment}\\ \text{Geometry} &: \text{relative positions and orientation}\\ \text{Action} &: \text{reach}\rightarrow\text{grasp}\rightarrow\text{lift}\\ \text{Camera} &: \text{viewpoint and motion}\\ \text{Timing} &: \text{temporal ordering of actions}\\ \text{Audio} &: \text{contact and environmental sounds}\\ \text{Media} &: \text{FPS, duration, resolution, aspect ratio} \end{aligned}\]

  • The Generator consequently receives a much denser description of the intended world evolution than the original request contained.

Reasoner as Prompt Upsampler

  • The Cosmos 3 Reasoner can itself perform this prompt expansion.

  • Prompt upsampling is treated as multimodal instruction following rather than simple text rewriting. The Reasoner takes a compact request and constructs a typed JSON representation containing scene semantics, visual attributes, temporal structure, audio descriptions, and generation parameters.

  • The process can be understood as three conceptual stages:

    \[\boxed{ \text{world state} \rightarrow \text{temporal rollout} \rightarrow \text{multimodal consequences} }\]

  • First, the Reasoner infers a coherent initial world state. It then determines how that state should evolve over time. Finally, it derives additional consequences, including audio events that should coincide with visible interactions.

  • This is a significant extension of conventional prompt expansion. Systems such as DALL-E 3 by Betker et al. (2023) use caption upsampling to translate short requests into richer descriptions better matched to the image generator’s training distribution. Cosmos 3 applies a related idea to an explicitly temporal and multimodal world representation.

  • The output is therefore closer to a scene program than a verbose prompt.

The Prompt Contract

  • The upsampler receives four major pieces of information:

  • Instructions: task-level rules governing the expected output and formatting.

  • Description: the user’s semantic request and, where relevant, conditioning media.

  • Task constraints: resolution, duration, FPS, aspect ratio, timing conventions, entity-preservation requirements, and task-specific grounding rules.

  • Output schema: the JSON structure defining where subjects, scene properties, temporal events, audio, camera, lighting, style, and generation controls must appear.

  • For image-to-video generation, the conditioning image is treated as authoritative evidence for the initial state:

    \[x(t=0)=x_{\mathrm{condition}}\]

  • The Reasoner can elaborate future events, but it should not rewrite visible facts in the conditioning frame.

  • This gives the interface a useful division of responsibilities:

    \[\text{constraints} \rightarrow \text{how to reason}\]

    • while:

      \[\text{schema} \rightarrow \text{how to represent the result}\]

  • Because the intermediate representation is explicit JSON, the reasoning-to-generation boundary is inspectable rather than hidden entirely inside model activations.

Sampling Configuration

  • Cosmos 3 uses different diffusion configurations for different tasks. The report’s default settings are summarized below.
Generation mode Steps Guidance Shift Additional control
Nano/Super audiovisual 50 6 10 full-range CFG
Super Text2Image 50 4 3 full-range CFG
Super Image2Video 50 6 5 full-range CFG
Forward/inverse dynamics 50 1 5 full-range CFG
Nano Policy DROID 4 3 5 full-range CFG
Transfer 50 3 10 control guidance (=1.5)

  • The differences are revealing. High-quality media generation uses \(50\) denoising steps, whereas the robot policy uses only \(4\).

  • The policy cannot afford the latency budget of an offline video generator. Its sampling configuration therefore sacrifices iterative refinement for control-loop responsiveness.

Negative Prompt Selection

  • Negative prompts are also task-specific rather than globally fixed.

  • NVIDIA evaluates candidate negative prompts using automated benchmarks, considering natural-language descriptions, keyword lists, instruction-style prompts, physical-consistency constraints, identity-preservation constraints, and the null string.

  • Interestingly, more prompting is not always better. The null negative prompt performs best for Cosmos3-Super-Text2Image and action generation, while the Image2Video model benefits from negative prompts derived from the user’s request.

  • This reinforces a broader point about diffusion inference:

    \[\text{optimal conditioning strategy} \neq \text{universal constant}\]

  • It depends on the task and post-training distribution.

Forward Dynamics

  • Forward dynamics asks the model to predict what happens after an action:

    \[p(x_{t+1:t+H}\mid x_{\leq t},a_{t:t+H})\]

  • For robotics, this can mean predicting future visual observations given a sequence of robot commands.

  • For camera motion, it can mean synthesizing how the scene changes under a requested trajectory.

  • For autonomous driving, it can mean predicting future observations conditioned on ego motion.

  • This mode turns Cosmos 3 into a learned simulator:

    \[\boxed{ \text{current world} + \text{proposed action} \rightarrow \text{predicted future world} }\]

  • The action evaluations show that initializing downstream models from the action-midtrained checkpoint consistently improves over adapting directly from the generic pretrained checkpoint across camera motion, egocentric motion, robotics, and autonomous-driving settings.

  • This provides empirical support for treating action midtraining as a reusable world-action prior.

Inverse Dynamics

  • Inverse dynamics reverses the conditional direction:

    \[p(a_{t:t+H}\mid x_{\leq t},x_{t+1:t+H})\]

  • Given an observed state transition, the model infers the action that could have produced it.

  • Conceptually,

    \[\boxed{ \text{current state} + \text{future state} \rightarrow \text{action} }\]

  • This is useful when large amounts of video are available but explicit action labels are scarce. If motion can be translated into a common geometric action representation, the model can learn relationships between observed changes and the transformations that caused them.

  • Forward and inverse dynamics are therefore complementary:

    \[a_t \xrightarrow{\mathrm{FD}} x_{t+1}\] \[(x_t,x_{t+1}) \xrightarrow{\mathrm{ID}} a_t\]

  • Their coexistence inside the same architecture is one reason the action representation introduced earlier is central to Cosmos 3.

Policy Generation

  • Policy mode predicts future actions directly from the current observation and task context:

    \[\pi(a_{t:t+H}\mid x_{\leq t},g)\]
    • where \(g\) represents the task or instruction.

  • Cosmos3-Nano-Policy-DROID is the clearest example. It receives current proprioception together with three visual views and predicts a \(32\) step action chunk at \(15\ \mathrm{Hz}\).

  • Unlike a conventional discriminative policy head, this action sequence is generated through the same diffusion formulation used for the Generator.

  • The progression from world model to policy can therefore be written as:

    \[\boxed{ \text{predict worlds} \rightarrow \text{understand action effects} \rightarrow \text{predict actions} }\]

Cross-Domain Action Transfer

  • One of the more important action experiments asks whether very different action domains provide useful training signal to one another.

  • The evaluated domains include autonomous driving, camera motion, egocentric human motion, and several robot embodiments.

  • These differ substantially:

    \[\text{vehicle trajectory} \neq \text{camera trajectory} \neq \text{human hand motion} \neq \text{robot end-effector motion}\]

  • Yet all encode transformations of entities through a physical environment.

  • The Cosmos 3 synergy experiments find several examples of positive transfer. Camera-motion forward dynamics, for example, improves when co-trained with autonomous-driving data and multiple robot datasets.

  • The report interprets this as evidence that action-conditioned prediction shares reusable priors such as

    \[\{ \text{object persistence}, \text{scene geometry}, \text{motion correspondence}, \text{interaction dynamics} \}\]

  • Robot datasets also show cross-domain transfer, particularly for lower-resource or weaker single-domain configurations.

Human Motion as a Robot Prior

  • A particularly interesting experiment connects egocentric human motion with robot manipulation.

  • Cosmos 3 first adapts the base model to egocentric human action data and then fine-tunes it on AgiBot robot manipulation. This is compared with directly adapting the same pretrained checkpoint to AgiBot.

  • The egocentric-warmed model performs better throughout the measured AgiBot adaptation trajectory. The reported forward-dynamics PSNR improvement is \(+0.94\) at 5K steps and approximately \(+1.3\text{--}1.6\) at later checkpoints.

  • The following figure (source) illustrates the pairwise transfer and warm-start experiment.

  • The result does not imply that human and robot actions are interchangeable. Instead, it provides evidence that learning from human interaction video can establish visual and dynamical priors that make subsequent robot adaptation easier.

  • This is potentially important for Physical AI because human video is vastly more abundant than robot demonstration data.

From World Generation to World-Action Models

  • The defining architectural progression in Cosmos 3 is that action is no longer treated merely as metadata attached to video.

  • A conventional video model learns approximately:

    \[p(x_{t+1:t+H}\mid x_{\leq t},c)\]
    • where \(c\) may be text.

  • Cosmos 3 expands the state space \(p(x, s, a, l)\) where:

    \[x=\text{visual state}\] \[s=\text{audio}\] \[a=\text{action}\] \[l=\text{language}\]

  • Different applications then correspond to conditioning on different subsets and generating the remainder.

  • For example: \(p(x_{t+1}\mid x_t,a_t)\) is forward dynamics, \(p(a_t\mid x_t,x_{t+1})\) is inverse dynamics, \(p(a_{t:t+H}\mid x_t,l)\) is policy generation, and \(p(x,s\mid l)\) is audiovisual generation.

  • The architectural ambition is therefore to replace a set of separately trained mappings with one learned multimodal joint distribution.

Applications to Physical AI

  • This formulation naturally supports several application classes.

  • For robotics, Cosmos 3 can reason about a scene, simulate a proposed manipulation, infer actions from demonstrations, or directly produce policy actions.

  • For autonomous driving, it can reason over road scenes, infer ego motion, generate photorealistic scenarios from structured world controls, and predict future observations.

  • For smart infrastructure, the Reasoner can analyze long-running fixed-camera streams for events, anomalies, localization, and causal relationships.

  • For synthetic-data generation, the Generator can convert simulation-derived depth, segmentation, edges, maps, or scenario representations into photorealistic training video.

  • For human-centered simulation, it can generate egocentric interactions and model the relationship between human motion and downstream robot adaptation.

  • The same model family therefore spans

    \[\boxed{ \text{observe} \rightarrow \text{understand} \rightarrow \text{simulate} \rightarrow \text{act} }\]

Limitations

  • The results also make clear that Cosmos 3 does not solve world modeling.

  • Physical correctness remains substantially below perfect performance. Even on Physics-IQ, where Cosmos3-Super produces strong reported results relative to the evaluated baselines, the direct I2V score is \(43.8/100\). This leaves considerable room between plausible-looking generation and faithful physical prediction.

  • Human evaluation remains necessary because automated VLM-based metrics miss subtle but important failures in geometry, contact, object permanence, human anatomy, and temporal dynamics.

  • Audio generation shows another gap. Cosmos 3 performs strongly on semantic audiovisual grounding, but the evaluation reports lower low-level production quality than the strongest proprietary audiovisual systems.

  • Action generation is also not equivalent to a universally deployable robot policy. The report’s demonstrated policy checkpoint is specialized to DROID after post-training, and the broader action results focus heavily on adaptation, forward dynamics, and inverse dynamics. Deployment on a new robot still requires embodiment-specific data and specialization.

  • Finally, inference cost varies dramatically across use cases. Offline media generation can use 50 diffusion steps, while real-time control requires aggressive reduction to four steps in the demonstrated DROID policy. World-model quality and control latency therefore remain an important systems tradeoff.

Key Takeaways

  • Cosmos 3 can be understood through five design decisions.

  • First, it makes modality a first-class architectural concept. Language, visual understanding, visual generation, audio, and actions use modality-specific representations while interacting through shared attention.

  • Second, it combines autoregressive reasoning and diffusion generation inside one Mixture-of-Transformers backbone. The Reasoner and Generator are therefore coupled without forcing all modalities into one homogeneous token-generation objective.

  • Third, it treats action as a native generative modality. Forward dynamics, inverse dynamics, and policy learning become alternative conditional-generation configurations rather than completely separate model families.

  • Fourth, it scales the full Physical AI pipeline rather than only the transformer. SILA, multimodal packing, two-way flat attention, Ulysses context parallelism, selective activation checkpointing, compiled VAE tokenization, asynchronous checkpointing, and optimized serving are all required to make the architecture practical.

  • Finally, Cosmos 3 reframes a world model as an omnimodal conditional model:

    \[\boxed{ p( \text{language}, \text{vision}, \text{audio}, \text{action} ) }\]
    • from which many Physical AI tasks can be derived by choosing which variables are observed and which must be generated.

  • That is the central idea connecting the entire system. Cosmos 3 is not simply a stronger VLM, video generator, or robot policy. It is an attempt to make those capabilities different interfaces to a shared model of physical environments.

  • The long-term hypothesis is that the same learned representation can support the complete loop:

    \[\boxed{ \text{perceive} \rightarrow \text{reason} \rightarrow \text{predict} \rightarrow \text{act} \rightarrow \text{observe again} }\]

  • Cosmos 3 provides evidence that these tasks can share architecture, data, and learned priors, while its remaining gaps in physical fidelity, audiovisual quality, embodiment transfer, and low-latency action generation show how far general-purpose Physical AI world models still have to progress.

References

Cosmos 3 and the NVIDIA Physical AI stack

Unified multimodal architectures

Generative modeling and flow matching

Vision, video, and multimodal preprocessing

Robot data and vision-language-action models

World models and interactive simulation

Evaluation and video quality

Prompt upsampling and structured generation

Distributed training and long-context systems

Data storage, retrieval, and model serving

Citation

If you found our work useful, please cite it as:

@article{Chadha2020DistilledCosmos3,
  title   = {Cosmos 3},
  author  = {Chadha, Aman and Jain, Vinija},
  journal = {Distilled AI},
  year    = {2020},
  note    = {\url{https://aman.ai}}
}