Cosmos 3: Omnimodal World Models for Physical AI
1. Paper overview
In one sentence: Cosmos 3 jointly denoises future video and actions inside a broader multimodal architecture, gaining transferable control priors while leaving compute-matched synergy and causal rollout reliability incompletely established. e-architecturee-modese-policye-pusht
| At a glance | What to know |
|---|---|
| Research problem | Author claim Physical AI systems commonly assemble separate understanding, video-generation and action models. The authors seek one adaptable architecture that can understand observations, simulate their evolution and produce actions across embodiments. e-scope |
| Core mechanism | Source description A Mixture-of-Transformers couples autoregressive and diffusion streams through asymmetric attention while retaining separate learned projections and feed-forward weights. e-architecture |
| A key reported result | RoboLab language-conditioned manipulation: 39.7% Task success rate. Nano-Policy-DROID; 120 simulated tasks, ten rollouts per task; specific instructions. π0.5: 28.1%; Nano PT-init: 30.2%. Gains are 11.6 and 9.5 percentage points. Complex/vague tasks remain weak: 4.1% versus PT-init 7.1%. e-policy |
| Reading caution | Reader analysis Aggregate benchmark leadership is protocol-specific. Automated video metrics miss physical failures; human evaluation and audio fidelity retain clear gaps. Table 19 does not support the prose's assertion of wins across every difficulty/instruction category. e-videoe-humane-audioe-policy |
Core contributions
- Source description
A Mixture-of-Transformers couples autoregressive and diffusion streams through asymmetric attention while retaining separate learned projections and feed-forward weights. e-architecture
- Reader analysis
A staged multimodal curriculum yields transferable initialization and specialized image, video and DROID policy checkpoints; the reported benefits depend on task and evaluation protocol. e-traininge-midtraininge-specializatione-dynamicse-policy
Figure 5. An asymmetric attention mask connects understanding to multimodal generation. Original paper, p. 11 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the token legend below the diagram: blue tones mark autoregressive vision and language, while orange, purple and green mark diffusion vision, audio and action. Dashed outlines indicate noisy inputs. Follow each stream downward through normalization, the shared attention operation and its own MLP, then read the matrix with queries on rows and keys on columns. The blue upper-left block is causal. The upper-right quadrant is zero, so autoregressive queries cannot read diffusion tokens. Both lower quadrants are enabled, allowing diffusion predictions to use the reasoner's context and the other generation modalities. Equations 7–8 on page 12 confirm these directions. e-architecturee-modese-training
What it supports. The integration occurs through attention over a common sequence, with separate learned transformations for the two streams. In policy mode, future video and actions occupy the same diffusion subsequence and can inform one another during denoising. This supplies an architectural route for joint prediction without requiring identical AR and DM weights.
Where the evidence stops. The clean outputs are schematic: training predicts flow velocity, and actual inputs can include clean conditioning tokens. The caption explicitly simplifies the diagram by showing all diffusion inputs noisy. Attention connectivity alone does not establish causal action grounding.
2. Motivation
2.1 The problem and the proposed response
Physical AI systems commonly assemble separate understanding, video-generation and action models. The authors seek one adaptable architecture that can understand observations, simulate their evolution and produce actions across embodiments. e-scope
2.2 What this reading follows
A robot policy needs actions that change the world; a video generator needs plausible future pixels. Cosmos 3 places both predictions in one diffusion stream, conditioned by an autoregressive understanding stream. This reading follows that connection from the attention mask through staged training to robot execution. The evidence is deliberately separated: image and video benchmarks measure generated content, dynamics tests measure predicted motion or reconstruction, and RoboLab measures executed task success in simulation. The most revealing appendix result is mixed: joint action-mode training improves inverse dynamics and policy coverage while reducing video PSNR. That trade-off helps frame what the unified architecture has actually demonstrated. e-architecturee-modese-policye-pusht
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | WAMs |
| Architecture | One Model |
| Prediction paradigm | Joint prediction |
| Quadrant | Q1 · One Model × Joint prediction |
3.1 Evidence-based assessment
Supports the recorded classification
Q1 is supported by a single integrated generator jointly denoising video and action targets, with executable DROID policy outputs. Separate AR/DM weights and specialist checkpoints qualify what 'One Model' means. Audio and action-alignment evidence support the broader subcategories; a distinct tactile sensing interface is not demonstrated by the listed encoders. e-encoderse-modese-architecturee-policy-recipee-consistency
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 From an observed scene to two kinds of future
Start with an observed video prefix and decide which physical quantity is known. If an action trajectory is supplied, the forward-dynamics layout treats that trajectory as clean conditioning and denoises future video. If a video trajectory is supplied, inverse dynamics instead denoises the corresponding actions. In policy mode, neither future quantity is given: video and action tokens are generated together. Figure 4 and Equations 3–6 make this distinction explicit. Figure 5 then explains how the layout is processed. The reasoner supplies a causal visual-language context that cannot read noisy targets, while diffusion tokens attend that context and one another. My interpretation is that this connectivity makes video-action coordination possible, but coordination must still be demonstrated through training and execution evidence. A shared attention mask by itself does not prove that changing an action produces the correct counterfactual future. e-modese-architecture
5.2 Why an action prior is more than an action decoder
The training curriculum changes what the generator has learned before a robot-specific decoder is fitted. Broad image and video pre-training establishes a visual prior; mid-training adds heterogeneous motion, synchronized audio and transfer tasks, while preserving other media training. Common pose representations and domain-specific projections let different embodiments enter the diffusion stream without requiring identical raw action dimensions. DROID post-training then changes the practical output interface to absolute joint-position chunks. Table 18 and LIBERO adaptation compare the consequences of starting from PT-init or MT-init, supporting the value of the later initialization. However, my causal reading is narrower than an action-only explanation: Table 6 adds several modalities and extra training simultaneously. Appendix E's separate reasoner and audio ablations help identify component benefits, yet they do not fully decompose the main policy improvement into data, compute and shared-representation effects. e-actionse-datae-midtraininge-policy-recipee-dynamicse-transfer-policye-component-ablations
5.3 Trace imagined motion all the way to execution
The strongest policy evidence requires leaving the generated video. DROID inference produces 32 joint-position actions at 15 Hz using four denoising steps; the control implementation skips video decoding and executes the action chunk. RoboLab then tests whether those actions complete language-conditioned tasks. Appendix E.5 asks a complementary question: when the predicted chunk is executed from the same simulator state, does the resulting video agree with the jointly predicted video? Its left-camera PSNR is higher than its wrist-camera PSNR, consistent with the source's explanation that moving viewpoints reveal harder-to-predict content. My interpretation is that these diagnostics address different failures: task success checks utility, while paired-video agreement checks reconstruction consistency. Neither currently demonstrates that model uncertainty identifies bad plans, or that a long sequence of imagined actions remains causally calibrated. The proposed perturbation check below targets that missing link. e-policy-recipee-policye-consistency
5.4 Training and inference
During training
Reasoner training uses next-token prediction followed by supervised fine-tuning; its corpora contain approximately 22.0M and 2.2M samples. Generator pre-training freezes the trained reasoner and learns flow velocity with masked MSE on noisy targets, excluding clean conditioning. e-reasoner-traininge-training
Generator data include 767M images, 347.7M video clips and 138.9M clips with usable audio. Action mid-training spans 8.4M episodes and 61.3K hours across robot, egocentric, vehicle and camera motion; substantial portions are proprietary. e-data
Mid-training mixes image/video, audiovisual, action and transfer tasks; action is 25% of Table 6's mixture and its loss is weighted 10×. Thus MT-init versus PT-init changes more than action exposure alone. Subsequent image, video and robot post-training produce distinct specialist checkpoints. e-midtraininge-specialization
DROID post-training removes idle/failed demonstrations, adds an action encoder/decoder interface, and trains 32 future absolute joint-position actions at 15 Hz with auxiliary future RGB video. Its three-view canvas combines wrist and two external cameras. e-policy-recipe
During inference
Media generation uses structured prompt upsampling and modality-specific sampling settings. Table 21 specifies 50 steps for base audiovisual and dynamics generation; conditioning representations from the reasoner can be cached across denoising iterations. e-promptinge-serving
DROID policy inference uses four denoising steps, guidance 3 and shift 5 on two RTX Pro 6000 GPUs. Franky executes the 32-action chunk at 15 Hz; video latents are not decoded during control. The described deployment does not use decoded imagined futures to rank candidate plans. e-policy-recipee-prompting
5.5 Implementation flow
- Encode different physical signals
Understanding uses a trainable vision transformer; generation uses a frozen Wan2.2 video VAE and frozen audio VAE. Video compression is temporal 4× with effective spatial 32×32; audio is represented at 25 tokens/second. Domain-specific linear action projections map normalized vectors into the shared hidden width. e-encoderse-actions
- Separate conditioning from prediction
Autoregressive tokens precede diffusion tokens; clean conditioning precedes noisy targets. Forward dynamics supplies actions and predicts future video. Inverse dynamics supplies video and predicts actions. Policy mode makes both future video and actions noisy targets, enabling joint denoising. e-modes
- Couple streams asymmetrically
Each layer has separate AR/DM normalization, QKV projections and MLP parameters. AR queries see only the causal AR prefix; diffusion queries see both streams fully. Physical-time-scaled 3D rotary positions align visual, audio and action rates. Nano and Super have 16B and 64B total parameters; the 4B Edge variant is described as a later release. e-architecturee-positione-variants
6. Experiments & results
Cosmos 3 integrates autoregressive understanding with diffusion generation of images, video, audio and actions. Its most relevant world-action result is a shared generator that can predict future observations and executable robot actions jointly, then specialize through post-training. Broad benchmark gains support useful transfer, while mixed ablations and limited consistency diagnostics leave the strength of causal action grounding unresolved.
6.1 Read the original evidence
Table 11. Image specialization improves aggregate alignment, with uneven gains across criteria. Original paper, p. 55 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the first three columns of scores as different UniGenBench populations: All combines 600 original prompts and 570 added physical prompts; Orig and Phys separate them. Then move to the English CVTG-500L and Chinese CVTG-102ch text-rendering columns, whose GNED and PNED entries are higher-is-better character-level metrics. The last columns measure aesthetics and predicted human preference, not task completion. Compare the first two green rows to distinguish the specialized image checkpoint from base Super. Finally read the footnote: Gemini's English text scores change substantially when casing errors are excluded. Pages 54–55 specify prompt rewriting, judging and generation resolution. e-imagee-specialization
What it supports. The specialized checkpoint reaches 91.36 overall versus base Super's 87.33 and Gemini 3 Pro Image's 90.69. Its physical-subset score is 89.54, slightly below both corresponding comparators, 89.64 and 89.74. Aggregate improvement therefore should not be restated as a universal improvement in physical correctness or every image-generation criterion.
Where the evidence stops. These are rewritten-prompt, model-judged or proxy-metric comparisons at 1024×1024. The table uses a specialized checkpoint and does not evaluate actions, temporal consistency or physical robot execution. Differences in casing treatment matter when comparing text rendering.
Table 12. Video rankings change with the benchmark and with the component being measured. Original paper, p. 57 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Treat the table as three evaluation blocks. PAIBench-G Text2Video and Image2Video each show Overall, Domain and Quality; Overall equally weights the two components. The rightmost RBench block scores generated robotics videos using task correctness and visual quality. Compare models within the same block: T2V and I2V have different conditioning, while RBench has a different case set and generation horizon. PAIBench-G uses five seeds and 189 frames per prompt; RBench uses one seed and 121 frames. Both use 720p output. The protocol on pages 56–57 also identifies structured prompt rewriting and an internal VLM judge, which are part of the reported setting. e-video
What it supports. Super leads PAIBench-G overall at 80.0 for T2V and 82.8 for I2V, versus Wan2.2-A14B's 78.0 and 81.3. RBench changes the ordering: Nano scores 58.4%, Super 58.1%, and closed Wan2.6 60.7%. Thus larger model size and stronger broad video scores do not guarantee the highest robotics-video score.
Where the evidence stops. The page's protocol footnote says internal scores use Qwen2.5-VL-72B-Instruct because public judging with Qwen3-VL-235B-A22B was not reproducible for the authors. These values should not be merged with public leaderboard scores or interpreted as executed robot success.
Table 18. Mid-training improves downstream dynamics, under task-specific measurements. Original paper, p. 65 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Begin with the labels in parentheses: MT-init starts from the mid-trained multimodal checkpoint; PT-init starts before action-domain mid-training. Compare matching model sizes across these rows before comparing external baselines. The vehicle block is inverse dynamics, inferring trajectories from video. Camera motion is forward dynamics, producing video from a commanded trajectory; its poses are subsequently estimated from the generated frames. RRE is relative rotation error, RTE relative translation error, and ATE absolute trajectory error, all lower-is-better. Egocentric and robot forward dynamics instead report higher-is-better PSNR. Dashes mean a baseline is not reported for that domain, rather than a zero score. e-dynamicse-midtraining
What it supports. For Super, camera errors improve from 0.293°/0.036 m/1.82 m with PT-init to 0.142°/0.026 m/0.99 m with MT-init. Robot prediction rises from 22.69 to 26.04 dB. These comparisons support a stronger downstream initialization across distinct action interfaces; they do not create a common score measuring all aspects of a world model.
Where the evidence stops. Matched downstream budgets do not match the full pre-training history: mid-training adds action and other data. Camera metrics depend on pose estimation; robot PSNR measures a short predicted sequence. Neither directly establishes reliable long-horizon planning or intervention fidelity.
Table 19. Executed policy success improves overall, while vague complex tasks expose a weakness. Original paper, p. 68 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each three-column group as the same task family tested with vague, default or specific language. The left group aggregates all RoboLab tasks; the remaining groups split task difficulty. The first row is the DROID-specialized Cosmos policy, while the next row starts post-training from PT-init. External policy baselines appear below. For specific language, compare 39.7 in the overall column with 30.2 for PT-init and 28.1 for π0.5. Then deliberately inspect Complex/Vague: the first row is 4.1, below both PT-init's 7.1 and π0.5's 5.3. The protocol spans 120 simulated tasks with ten rollouts per task. e-policye-policy-recipe
What it supports. Under specific instructions, Cosmos improves overall success by 9.5 percentage points over its PT-init comparison and 11.6 points over π0.5. Its overall results also fall substantially with vague language. The difficulty breakdown shows that a favorable average can coexist with a concrete regression in a demanding evaluation slice.
Where the evidence stops. The prose claims improvement across all difficulty levels, which the Complex/Vague cells contradict. It also gives DreamZero specific success as 25.2%, whereas the table reports 23.9%. These conflicts remain unresolved; Table 19 provides no confidence intervals.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Physical AI understanding Super reasoner; grouped benchmark averages, Table 10. | 57.8 / 79.3 Robotics / driving average score | Qwen3-VL-32B: 52.6 / 40.7. These measure understanding, not executed control. e-reasoner-results |
| Text-to-image semantic alignment Super-Text2Image; 1,170 UniGenBench prompts; 1024×1024; Claude Opus 4.7 rewriting and Gemini 3.1 Pro judging. | 91.36 / 89.54 All / physical-subset score | Gemini 3 Pro Image: 90.69 / 89.74; base Super: 87.33 / 89.64. The overall advantage does not extend to the physical subset. e-image |
| Video generation PAIBench-G: 1,044 pairs, five seeds, 720p, 189 frames; RBench: 650 cases, one seed, 121 frames. | Super: 80.0 / 82.8; 58.1% PAIBench-G T2V / I2V overall; RBench score | Wan2.2-A14B: 78.0 / 81.3; 50.7%. Nano reaches 58.4% on RBench; closed Wan2.6 reaches 60.7%. Internal PAIBench judging differs from the public leaderboard; RBench assesses generated videos. e-video |
| Physical continuation Physics-IQ: 396 real-world scenes; single-frame I2V or three-second V2V context; prompt upsampling. | Super: 43.8 / 59.7 Direct I2V / V2V score | Sora2 I2V: 42.3; Magi-1 V2V: 56.0. Scores of 48.9 / 63.4 additionally use WMReward best-of-N reranking and are a different inference protocol. e-physics |
| Human evaluation of video Cosmos-HUE: 100 prompts, five seeds; human verification of atomic criteria. | Super: 89.3 / 89.6 T2V / I2V score | Veo-3.1: 91.3 / 89.7; real-video references: 93.6 / 94.4. Human judgments expose remaining errors; HUE counts unclear responses as failures. e-human |
| Audiovisual generation SoundBench: 144 non-speech prompts, five seeds; semantic judging and audio-quality scoring. | Nano: 7.34 / 8.35 AVQ / semantic audiovisual score | Seedance-1.5-Pro: 7.64 / 8.21. Better semantic alignment coexists with lower overall audio quality. e-audio |
| Control-conditioned video transfer PAIBench-C: 600 clips, one control modality per generation. | Super: 0.58 Depth si-RMSE, lower is better | Cosmos-Transfer2.5: 0.68. Supports depth adherence without establishing closed-loop policy utility. e-transfer |
| Forward and inverse dynamics adaptation Matched downstream adaptation; PT-init versus MT-init. Camera FD uses 100 internal five-second clips; DROID FD predicts 16 frames from 16 actions. | MT: 0.142° / 0.026 m / 0.99 m; 26.04 dB Super camera RRE / RTE / ATE; robot PSNR | PT: 0.293° / 0.036 m / 1.82 m; 22.69 dB. MT AV-ID ATE: 0.90 m versus PT 1.32 m. MT improves initialization; camera poses are estimated and PSNR is a short-horizon reconstruction proxy. e-dynamics |
| RoboLab language-conditioned manipulation Nano-Policy-DROID; 120 simulated tasks, ten rollouts per task; specific instructions. | 39.7% Task success rate | π0.5: 28.1%; Nano PT-init: 30.2%. Gains are 11.6 and 9.5 percentage points. Complex/vague tasks remain weak: 4.1% versus PT-init 7.1%. e-policy |
| Policy transfer and embodiment adaptation Same DROID checkpoint on MolmoSpaces; separately post-trained Nano on LIBERO-10, 500 rollouts/checkpoint. | 39.0% (20 June 2026); MT 24.6% at 500 iterations and 97.4% at 2,000. MolmoSpaces oracle success; LIBERO success | LIBERO PT: 0.0% and 95.2%. RoboArena reports a first-place preference ranking dated 30 May 2026. Simulation success and real-robot pairwise preference are different evidence; LIBERO's advantage narrows with training. e-transfer-policy |
| Joint action-mode ablation PushT, Edge; three separate 2K-step specialists versus one 6K-step FD/ID/policy model; 500 policy rollouts. | Joint: 26.22 dB / 3.09×10^-4 / 77.3% FD PSNR / ID MSE / policy coverage | Single-mode: 27.13 dB / 1.11×10^-3 / 74.1%. Action accuracy improves while video reconstruction declines; per-mode exposure is matched, total per-model training is not. e-pusht |
| Video-action consistency Predicted DROID policy chunks executed in held-out RoboLab simulation; compare predicted and resulting video. | 23.19 / 17.33 dB Left / wrist camera PSNR | No wrong-action control or planning-calibration baseline is reported here. This directly checks paired predictions, but does not establish long-horizon causal calibration. e-consistency |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 31. Joint action training trades some video fidelity for better action-side metrics. Original paper, p. 109 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. The row labels encode the experimental budget, not just model names. Each single-mode specialist receives 2,000 steps; the joint model receives 6,000 steps spread across forward dynamics, inverse dynamics and policy. This matches optimization exposure per mode. Read each arrow before comparing: FD PSNR and policy coverage should increase, but ID MSE should decrease. Coverage is the overlap ratio of the T block with its target region, aggregated over 50 initializations and ten rollouts each. It is not the binary task-success metric used in RoboLab. The model in this appendix experiment is Cosmos3-Edge, whereas the main DROID policy uses Nano. e-pusht
What it supports. Joint training lowers ID MSE from 1.11×10^-3 to 3.09×10^-4, approximately a 72% reduction, and raises coverage from 74.1% to 77.3%. FD PSNR falls from 27.13 to 26.22 dB. The result supports useful interaction among action modes while documenting a measurable reconstruction trade-off rather than uniform improvement.
Where the evidence stops. The comparison matches per-mode steps, but the joint model receives three times the total training of any one specialist. A 6K-step specialist control is needed to separate shared-mode benefit from extra optimization; the table does not provide it.
7. Analysis & limitations
7.1 What the evidence leaves open
Aggregate benchmark leadership is protocol-specific. Automated video metrics miss physical failures; human evaluation and audio fidelity retain clear gaps. Table 19 does not support the prose's assertion of wins across every difficulty/instruction category. e-videoe-humane-audioe-policy
Synthetic-data gains are mixed: Table 26's all-source mixture changes overall PAIBench-G from 79.67 to 79.77 but Human from 85.46 to 84.99. The authors attribute human-domain degradation to a simulation-to-real gap. e-synthetic
Mid-training comparisons hold downstream budgets fixed, not complete training histories. PushT similarly leaves total-compute confounding. Joint generation and PSNR agreement alone do not demonstrate reliable counterfactual rollouts or a learned environment suitable for agent training. e-midtraininge-dynamicse-pushte-consistencye-scope
7.2 Questions for discussion
- Would joint action-mode gains persist against a specialist trained for the same total compute?
- Does low predicted-video error distinguish successful actions from plausible but ineffective motion?
- How much of MT-init's advantage comes from action data, other added modalities, or simply more training?
8. Reproducibility audit
8.1 Requirements and known gaps
Full retraining requires inaccessible proprietary data and substantial compute: generator pre-training reports 31.05T tokens on 1,024 GB200 GPUs for Nano and 17.86T on 2,048 for Super. Exact AV evaluation size, DROID FD holdout details and several visual filtering cutoffs are not reported in the relevant descriptions. e-datae-traininge-dynamics
Preserve unresolved source conflicts: Figure 10's caption says uniform video-mode sampling, while its graphic says 70/20/10 and prose gives overall T2I/T2V/I2V/V2V 20/56/16/8. Table 19 gives DreamZero specific success 23.9%, but nearby prose says 25.2%. Verify these against an author-supplied configuration before replication. e-sampling-conflicte-policy
A feasible proposed audit would compare joint and single-mode PushT training under both exposure-matched and compute-matched budgets, then perturb actions in RoboLab to test whether predicted-video agreement actually diagnoses execution failure. These checks are specified in the illustrated edition and were not run. e-pushte-consistency
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Separate cross-mode transfer from extra optimization
Proposed, not performed: repeat the Edge PushT comparison with three arms for each target mode: a 2K-step specialist, a 6K-step specialist, and the 6K-step joint FD/ID/policy model. Hold initialization, target dataset, preprocessing and evaluation states fixed; repeat training seeds and report uncertainty. Preserve the source's 50 initializations and ten policy rollouts per initialization. Report all three original metrics, including FD PSNR regressions, and distinguish per-mode exposure from total training compute. If joint training improves only over the 2K specialist, while the 6K specialist closes the gap, the evidence would favor additional optimization over a unique cross-mode benefit. e-pusht
Check 2: Test whether predicted-video agreement detects wrong actions
Proposed, not performed: reproduce the Nano-Policy-DROID RoboLab consistency test with four denoising steps, guidance 3 and shift 5. Save the joint video/action prediction and execute its action chunk from a saved simulator state. Reset to that state and execute shuffled or perturbed chunks as controls, comparing each resulting video with the original prediction. Measure left/wrist PSNR separately, object-state and contact errors, and eventual task success across horizons. If high PSNR persists for ineffective or altered actions, or fails to distinguish success from failure, it would falsify the use of reconstruction agreement alone as a planning-confidence signal. e-policy-recipee-promptinge-consistencye-policy
8.3 Reading coverage
Visual audit: The title/author/version page and Appendix G credits were visually inspected. Method inspection covered encoders, action layouts, attention masks, equations, temporal positions, model sizes, data, training, policy deployment and inference settings. Results inspection covered all pages needed for the retained quantitative comparisons and protocol qualifications, including appendix ablations and consistency diagnostics. Six final original crops were inspected: Figure 5 and Tables 11, 12, 18, 19 and 31. Figure 5's arrows and mask agree with Equations 7–8; its clean-output/noisy-input simplifications are explained. Table 11 retains its casing footnote. Table 12's separate body footnote on the judge was read on the full page and is explained in the visual caution. Figure 10's caption/sampling conflict and Table 19's prose/table discrepancies are preserved. Other PDF pages were read in the complete text pass but are outside visual inspection. Source omissions and unverified external resources remain as recorded in the base report.
PDF pages inspected for this edition: 1, 5, 7, 8, 9, 10, 11, 12, 13, 14, 15, 20, 21, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 47, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 65, 66, 67, 68, 69, 70, 73, 74, 80, 104, 105, 107, 108, 109, 110, 113, 114, 115, 116, 117. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- All 43 supplied text chunks, covering all 139 PDF pages, including the title, abstract and contents.
- Sections 1–2: Introduction; encoders, action representations, token layouts, Mixture-of-Transformers, positions and model variants.
- Sections 3–4: Reasoner and generator data; pre-training, mid-training and specialized post-training.
- Section 5: Data, training, serving and benchmark infrastructure.
- Section 6: Understanding, image, video, audio, transfer, dynamics and policy evaluations; inference and prompt upsampling.
- Sections 7–8: Related work and conclusion.
- Appendices A–G: Caption schemas; default prompts and upsampling templates; synthetic datasets; Edge LLM training; additional ablations; human evaluation; contributors and acknowledgments.
- References.
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- Identity is verified against the retained PDF: title-page credit NVIDIA; arXiv:2606.02800v4 dated 23 June 2026, with a separate report header dated 2026-6-24. The catalog expands credits with named contributors; Appendix G distinguishes contributors from acknowledgments. Metadata preserves the title-page credit. Earlier revisions were not supplied, so revision-to-revision changes cannot be established.
- No separate affiliation block was verified. Code, checkpoints, external leaderboards and separate supplements were not inspected; no experiment was reproduced.
- Visual inspection covers the pages declared in the illustrated edition, including all evidence needed for its claims. Other pages were read as supplied text; this is not a claim that every PDF page or every figure was visually inspected.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e-identityPDF p. 1, title block and arXiv sidebar; Appendix G, pp. 113–117.
The exact title is Cosmos 3: Omnimodal World Models for Physical AI. The title page credits NVIDIA and refers to Appendix G. The retained edition is arXiv v4, 23 June 2026; its report header reads 2026-6-24. Appendix G separately lists contributors and acknowledgments.
Go to primary source ↓e-scopePDF p. 5, Introduction and Figure 1; p. 80, Conclusion.
The paper motivates unifying understanding, generation and action. The conclusion describes better closed-loop training environments as an expected future role, not a demonstrated agent-training result.
Go to primary source ↓e-encodersPDF p. 7, §§2.1.1–2.1.3; p. 8, Figure 3.
ViT handles understanding; frozen Wan2.2-TI2V-5B VAE handles generation with temporal 4× and effective spatial 32×32 compression. Frozen audio VAE maps stereo 48 kHz audio to 25 tokens/second. Listed action domains are camera, vehicle, human and robot motion.
Go to primary source ↓e-actionsPDF pp. 8–9, §2.1.3, Figure 3 and Equations 1–2.
Motion uses previous-frame relative SE(3) transforms, translation and continuous 6D rotation. Grasp is current state. Domain-specific input/output linear maps connect normalized action vectors to hidden tokens.
Go to primary source ↓e-modesPDF pp. 9–10, §2.2, Equations 3–6 and Figure 4.
AR tokens come first; clean DM conditioning precedes noisy targets. FD predicts future video from observed video/actions; ID predicts actions from video; policy predicts video and actions jointly. Action a_t corresponds to the transition from v_(t-1) to v_t.
Go to primary source ↓e-architecturePDF pp. 11–12, Figure 5, §§2.3.1–2.3.2 and Equations 7–8.
AR/DM streams have separate normalization, projections and MLP weights. The top-right attention-mask quadrant is zero: AR queries cannot read DM keys. DM queries attend both streams. The diagram depicts clean outputs schematically and shows all DM inputs noisy for simplicity.
Go to primary source ↓e-positionPDF pp. 12–14, §2.4 and Figure 6.
3D MRoPE assigns temporal/height/width positions. Audio/actions have zero spatial coordinates. Temporal positions scale with actual token rate, using the 24-FPS video's compressed rate as the reference.
Go to primary source ↓e-variantsPDF p. 14, §2.5 and Table 2.
Edge, Nano and Super total parameter sizes are 4B, 16B and 64B; Nano/Super derive from Qwen3-VL 8B/32B. Edge is marked for later release.
Go to primary source ↓e-reasoner-trainingPDF p. 15, Table 3; pp. 26–27, §4.1.
Reasoner data contain 22,002,013 pre-training and 2,171,673 SFT samples. ViT/projector/language components are trained with next-token prediction; later SFT uses importance sampling and mixed pre-training/SFT budgets.
Go to primary source ↓e-dataPDF pp. 20–21, Figure 8 and §3.2.1; pp. 23–25, §§3.2.2–3.2.3.
767M images and 347.7M video clips feed generator training; 138.9M pre-training clips have usable audio. Action data span 8.4M episodes and 61.3K hours, including proprietary egocentric and internal driving material. Several image/video quality filters lack numeric cutoffs.
Go to primary source ↓e-trainingPDF pp. 27–29, §4.2.1.
Flow interpolation is x_sigma=sigma*epsilon+(1-sigma)*x_0 with velocity epsilon-x_0 and masked MSE. Noise schedules vary by modality. The reasoner stays frozen. FusedAdamW pre-training uses learning rate 1e-4 and weight decay 0.05; Nano/Super totals are 31.05T/17.86T tokens on 1,024/2,048 GB200 GPUs.
Go to primary source ↓e-sampling-conflictPDF p. 28, Figure 10, its caption and the generation-mode sampling paragraph.
The caption says video clips uniformly sample T2V/I2V/V2V. The figure labels 70%, 20%, 10%; prose gives all-task 20%, 56%, 16%, 8% including T2I. The latter two agree after conditioning on video, but disagree with uniform sampling.
Go to primary source ↓e-midtrainingPDF p. 30, §4.2.2 and Table 6.
Mixture weights: image 10%, video 32%, video-audio 8%, action 25%, general transfer 20%, driving transfer 5%. Action loss is multiplied by 10. Mid-training adds several kinds of supervision.
Go to primary source ↓e-specializationPDF pp. 30–32, §§4.2.3–4.2.5.
Separate post-training recipes specialize text-to-image, image-to-video and robot policy checkpoints; these should not be treated as one unchanged checkpoint winning every benchmark.
Go to primary source ↓e-policy-recipePDF pp. 31–32, §4.2.5.
DROID uses Franka Panda and Robotiq 2F-85. Post-training predicts 32 absolute joint-position actions at 15 Hz plus future RGB video from proprioception, instructions and a 540×640 three-view canvas. Inference uses four steps, shift 5, CFG 3, two RTX Pro 6000 GPUs and Franky; video decoding is skipped.
Go to primary source ↓e-servingPDF p. 47, §5.3.1, Reasoner KV Cache.
Reasoner conditioning is unchanged across diffusion steps, permitting reuse of per-layer keys/values.
Go to primary source ↓e-reasoner-resultsPDF p. 51, Table 10; pp. 52–53, §6.1.
Super's grouped robotics/driving averages are 57.8/79.3 versus Qwen3-VL-32B 52.6/40.7. These are understanding benchmarks, with task-specific metrics and protocols.
Go to primary source ↓e-imagePDF pp. 54–55, §6.2.1 and Table 11.
UniGenBench combines 600 original and 570 physical prompts; Gemini 3.1 Pro judges binary criteria. Claude Opus 4.7 rewrites prompts; resolution is 1024×1024. Super-Text2Image All/Orig/Phys are 91.36/93.34/89.54; Gemini 3 Pro Image is 90.69/92.81/89.74. The retained table footnote explains Gemini's case-sensitive English-text scoring.
Go to primary source ↓e-videoPDF pp. 56–57, §6.2.2 and Table 12, including p. 57 footnote.
PAIBench-G uses 1,044 pairs, five seeds and 189-frame 720p videos; overall averages Domain and Quality equally. The internal judge is Qwen2.5-VL-72B-Instruct, whereas the public leaderboard uses Qwen3-VL-235B-A22B. RBench uses 650 cases, one seed, 121 frames. Table 12 reports Super 80.0/82.8 and 58.1%, Nano RBench 58.4%, Wan2.6 RBench 60.7%.
Go to primary source ↓e-physicsPDF p. 58, Physics-IQ protocol and Table 13.
396 real-world scenes; I2V uses a switch frame, V2V uses three seconds to predict five. Prompt upsampling is used. Super direct scores 43.8/59.7; WMReward best-of-N scores 48.9/63.4 are separate. This description does not specify N.
Go to primary source ↓e-humanPDF pp. 59–60, Table 14 and HUE/HWB protocols; p. 110, Appendix F scoring.
HUE uses 100 prompts and five seeds, with binary human verification. Super T2V/I2V is 89.3/89.6, Veo 91.3/89.7 and real-video references 93.6/94.4. Unclear answers count as No and model-level aggregation weights questions. Nano I2V is 88.6 in Table 14 but 88.5 in nearby prose; §6.2.2 allows up to 20 questions while Appendix F says up to 16.
Go to primary source ↓e-audioPDF pp. 61–62, §6.2.3, Table 15 and Figure 20.
SoundBench has 144 non-speech prompts and five seeds. Table 15 separates AVQ, semantic audiovisual score and production quality: Nano 7.34/8.35/6.32, Seedance 7.64/8.21/7.06. The prose/formula alternates AQ/PQ labels. Figure 20 is one qualitative hammer-strike/spectrogram example.
Go to primary source ↓e-transferPDF pp. 62–63, §6.2.4 and Table 16.
Transfer uses independent text/control guidance with three denoiser evaluations. PAIBench-C contains 600 clips, 200 each from robot, driving and egocentric domains, evaluated with one control at a time. Super depth si-RMSE is 0.58 versus Transfer2.5 0.68.
Go to primary source ↓e-dynamicsPDF pp. 65–67, §6.2.5, Table 18 and Figures 22–23.
PT-init lacks action-domain mid-training; MT-init includes it. Downstream model size, data and compute are matched. Table 18 supplies the retained pose-error and PSNR values. AV uses internal six-second 10-FPS clips with no dataset size here; camera uses 100 five-second clips and estimated poses; DROID FD predicts 16 frames from 16 end-effector actions without a stated holdout count here.
Go to primary source ↓e-policyPDF pp. 67–68, RoboLab protocol and Table 19.
RoboLab has 120 tasks and ten rollouts each, across vague/default/specific instructions. Overall Cosmos rates are 20.6/36.8/39.7%; PT-init 16.7/28.1/30.2%; π0.5 15.2/28.0/28.1%. Complex/vague Cosmos 4.1% trails PT 7.1% and π0.5 5.3%. DreamZero specific is 23.9% in the table, 25.2% in prose. Table 19 supplies no confidence intervals.
Go to primary source ↓e-transfer-policyPDF pp. 68–70, RoboArena, MolmoSpaces and LIBERO descriptions; Figures 25–27; Table 20.
RoboArena uses double-blind pairwise real-robot preference; first place is dated 30 May 2026. MolmoSpaces All Combined oracle success is 39.0%, dated 20 June, with the same checkpoint/settings. LIBERO uses two views and 500 rollouts/checkpoint: MT/PT at 500 steps is 24.6/0.0%; at 2,000 steps 97.4/95.2%.
Go to primary source ↓e-promptingPDF pp. 73–74, Table 21 and §§6.3.1–6.3.2.
Structured prompts specify scene and temporal attributes. Base audiovisual defaults: 50 steps, guidance 6, shift 10; dynamics: 50/1/5; DROID policy: 4/3/5. Negative prompts and rewriting vary by model and task.
Go to primary source ↓e-syntheticPDF pp. 104–105, Table 26 and §C.7.
The all-source synthetic mixture raises overall score 79.67→79.77 but lowers Human 85.46→84.99; individual sources also reduce Human scores. The authors discuss simulation-to-real gaps and domain trade-offs.
Go to primary source ↓e-component-ablationsPDF pp. 107–109, §§E.1–E.3 and Tables 28–30.
At matched generator training, the Cosmos reasoner raises T2V Domain from 73.7 to 75.7 with Quality 73.0 unchanged. Combining text FPS control and MRoPE gives composite 9.81 versus 8.51 without control; the score combines perceptual quality and a motion proxy. Audio co-training raises T2V/I2V overall by 0.5 each, with T2V Quality 73.4→73.2 and I2V Quality unchanged.
Go to primary source ↓e-pushtPDF p. 109, §E.4 and Table 31.
Edge PushT specialists train 2K steps each; joint FD/ID/policy trains 6K, matching exposure per mode. FD PSNR 27.13→26.22; ID MSE 1.11×10^-3→3.09×10^-4; policy coverage 74.1→77.3%. Coverage measures T-block overlap across 50 initializations and ten rollouts each.
Go to primary source ↓e-consistencyPDF pp. 109–110, §E.5 and Figure 37.
The same predicted action chunk is executed in RoboLab from the same initial state; predicted versus simulator video PSNR is 23.19 dB for the left camera and 17.33 for wrist. This is held-out simulation for a DROID-trained policy; no shuffled-action control or long-horizon calibration analysis is supplied here.
Go to primary source ↓8.5 Primary sources
Cosmos 3: Omnimodal World Models for Physical AI ↗
PDF · 71,863 extracted words
Source fingerprint
df058bd2447fbdd83d13fd63be60a8b0aceb00ef6e50ba428b0a6bdbbed64b8d