World Simulation with Video Foundation Models for Physical AI
1. Paper overview
In one sentence: A shared video generator becomes useful for Physical AI through specialized conditioning and synthetic observations, while most reported gains still concern video prediction rather than closed-loop control. e-identitye-architecturee-transfere-camerae-actione-robot-resultse-pai
| At a glance | What to know |
|---|---|
| Research problem | Source description Physical AI needs diverse training observations and controllable simulations without collecting every condition on hardware. The paper addresses visual world generation and adaptation for robotics and driving, with controllability and temporal consistency as central requirements. e-intro |
| Core mechanism | Source description A shared 2B/14B video backbone combines flow matching, Cosmos-Reason1 text features and unified Text2World, Image2World and Video2World conditioning. e-flowe-architecture |
| A key reported result | Real-robot visual robustness: Transfer-augmented policy: 24/30. Successful executed trials. Apple-into-bowl bimanual task; 100 demonstrations, five synthetic variants each; ten scenarios, three trials per scenario with matched poses/configurations. Standard image augmentation: 5/30; unaugmented policy: 1/30. Evidence supports this policy’s visual robustness on one task. Page 24’s statement that the baseline succeeds in just one case is inconsistent with Table 13’s two nonzero scenario columns; table counts are retained. e-robot-setupe-robot-augmente-robot-resultse-robot-prose |
| Reading caution | Reader analysis Generation scores and human preference do not establish physical correctness or reliable closed-loop policy validation. RNDS normalizes each long-video DOVER ratio by its first chunk on 17 videos, so it measures relative quality evolution, not absolute dynamics accuracy. e-paie-longe-action |
Core contributions
- Source description
A shared 2B/14B video backbone combines flow matching, Cosmos-Reason1 text features and unified Text2World, Image2World and Video2World conditioning. e-flowe-architecture
- Source description
Progressive pretraining, domain-specific fine-tuning, model merging, reward-based post-training and timestep distillation improve or accelerate generation. e-pretraine-sfte-rle-distill
- Source description
Specializations cover spatial translation, multiview simulation, synthetic VLA data and action-conditioned prediction, with separate downstream evaluations. e-transfere-driving-methode-camerae-vlae-action
Figure 2. Two distinct conditioning paths connect language and visual prefixes to the video generator. Original paper, p. 10 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Begin at the left: tokenized text passes through Cosmos-Reason1, and activations from several transformer blocks feed the projection. The resulting text embeddings enter the right-hand DiT through cross-attention. At the top right, the three token grids represent generation with no visual prefix, an image prefix or a video prefix. Yellow prefix regions supply visual context through the video-token pathway. Timestep embeddings enter from the right to modulate scale, shift and gates around self-attention, cross-attention and the feed-forward layer. Follow these arrows separately: the demonstrated image/video prefix mechanism does not require the dashed vision-encoder branch on the left. e-architecturee-flowe-pretrain
What it supports. The backbone shares a video-generation architecture across three conditioning modes. Section 3.2 specifies a 4×8×8 VAE compression and 1×2×2 patchification, with multilayer language features projected to 1024 dimensions. These features guide generated observations; the figure contains no executable robot-action output.
Where the evidence stops. The caption reserves Cosmos-Reason1’s dashed image/video branch for future work. Also, the schematic labels output as denoised tokens, while Equations (2)–(3) specify velocity prediction. Read the graphic as a generation overview, not a literal training-target diagram.
2. Motivation
2.1 The problem and the proposed response
Physical AI needs diverse training observations and controllable simulations without collecting every condition on hardware. The paper addresses visual world generation and adaptation for robotics and driving, with controllability and temporal consistency as central requirements. e-intro
2.2 What this reading follows
Cosmos-Predict2.5 is best understood by following its conditions: language enters through cross-attention, images anchor video prefixes, and specialized variants add spatial maps, camera poses or recorded robot actions. The same family can therefore generate possible futures, translate demonstration videos or synthesize additional views. Those uses lead to different kinds of evidence. A video benchmark measures generated observations; the Bridge experiment measures prediction under supplied actions; a separate real-robot experiment tests a policy trained with translated observations. This illustrated reading follows all three distinctions, using the supplied February 2026 v2 paper and retaining its numerical and reporting limits. e-identitye-architecturee-transfere-camerae-actione-robot-resultse-pai
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Foundational work |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
This table preserves the labels recorded at reading time. The current major category is Components of WAMs. View the current classification.
3.1 Evidence-based assessment
Supports the recorded classification
The foundational video-backbone/neural-simulator classification is supported. A single model unifies three video-conditioning modes, but it does not jointly output future observations and executable actions. Actions condition the Bridge predictor, while IDM-based pseudo-labeling and Diffusion Policy are separate pathways. Keeping the action-model architecture, prediction paradigm and quadrant not applicable is justified for this family-level entry. e-architecturee-actione-vlae-robot-setup
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Separate denoising time from physical action
The first distinction is between the variable a model learns to predict and the conditions used to steer that prediction. Equations (1)–(3) interpolate a data sample with Gaussian noise and train the network to predict the resulting velocity. That velocity belongs to the denoising trajectory. It does not specify how a gripper should move. Figure 2 then shows where conditions enter: language reaches cross-attention, while an image or video prefix occupies part of the video-token sequence. The specialized Bridge model adds a further condition by embedding recorded actions and adding those features to DiT timestep embeddings. Its output is still future imagery. For longer action-conditioned rollouts, the last generated frame anchors the next chunk. A controller that acts on physical observations would require a separate decision and feedback mechanism. e-flowe-architecturee-action
5.2 Treat post-training as several separate interventions
Reader interpretation: the training recipe matters because different interventions repair different weaknesses, making a single explanation for every gain unreliable. The paper first increases resolution and task diversity during pretraining. It then specializes separate models on curated physical-AI domains, adds a high-quality-video cooldown, and merges model weights. This reduces the need to choose one fixed data mixture during domain fine-tuning, but introduces model-selection choices of its own. RL subsequently optimizes VideoAlign rewards with diffusion-loss regularization. Figure 5 supports preference gains relative to each model’s own pre-RL state; it does not compare the two starting models directly. Finally, distillation compresses sampling to four steps. The small benchmark changes after distillation should be assessed separately from the improvement attributed to SFT or RL, and separately again from deployment latency. e-pretraine-sfte-rle-distill
5.3 Follow three different routes from video to robotics
Reader interpretation: the paper’s robotics applications become clearer when the action source is tracked explicitly. In the real-robot augmentation experiment, actions already exist in teleoperation demonstrations. Transfer changes their associated images, and a separate Diffusion Policy learns to execute the task. In the DreamGen pathway, generated videos instead acquire pseudo-actions through a latent-action model or inverse-dynamics model, creating candidate VLA training examples. The reported DreamGen scores judge instruction following in those videos. In the Bridge pathway, actions are supplied before generation and the world model predicts their visual consequences. Table 13 therefore concerns executed policy success, Table 18 concerns generated instruction following, and Tables 19–20 concern conditional prediction. Together they show several uses for a video backbone, while leaving open how faithfully imagined outcomes support physical policy selection. e-robot-setupe-robot-augmente-robot-resultse-vlae-dreamgene-actione-bridgee-action-ablation
5.4 Training and inference
During training
Pretraining uses approximately 200 million retained clips from a pipeline of splitting, transcoding, cropping, filtering, captioning, semantic deduplication and sharding. General data are supplemented with robotics, driving, smart spaces, human dynamics and physics data. The mixture includes proprietary material. e-data
Training progresses from 256p images to joint image/video learning at 480p and 720p, finally adding text-only video generation. Noise sampling shifts toward higher noise; 5% of samples come from the highest 2% of the noise distribution. AdamW learning rates are 3×10⁻⁵ for 2B and 1.3×10⁻⁵ for 14B. e-pretrain
Separate domain SFT models each receive 30,000 iterations at batch 256. A 4K cooldown model joins model merging; model soup is selected. The described 2B RL procedure uses VideoAlign, eight outputs per condition, 20 diffusion steps, group-normalized rewards and diffusion-loss regularization, training for 256 updates at batch 32. e-sfte-rl
During inference
Encode conditions, run the conditional velocity network through denoising, and decode video latents. Image/video prefixes anchor continuation. In the action variant, each subsequent chunk is conditioned on the last generated frame, so feedback is generated imagery rather than a fresh physical observation. e-flowe-architecturee-action
rCM distillation permits four-step generation. Reported overall scores change from 0.768 to 0.764 for Text2World and from 0.810 to 0.816 for Image2World; these are benchmark scores, not measured end-to-end deployment latency. e-distill
5.5 Implementation flow
- Represent video and language
WAN2.1 VAE compresses time/height/width by 4×8×8, followed by 1×2×2 latent patchification. Multiple Cosmos-Reason1 layer activations are concatenated and projected to 1024-dimensional text features. The base generation unit is 93 pixel frames, or 24 latent frames, at 16 fps. e-architecture
- Predict flow with a conditional DiT
Repeated self-attention, text cross-attention and feed-forward blocks use timestep-dependent scale, shift and gates. Relative 3D RoPE replaces absolute positional embeddings. Conditional prefix frames are retained through frame replacement and binary conditioning flags. Figure 2’s additional Reason1 vision branch is future work. e-flowe-architecturee-pretrain
- Translate while preserving spatial controls
Transfer2.5 accepts edges, blur, depth or segmentation. Four control blocks are distributed one per seven main-branch blocks. Each modality branch is trained independently; the uniform multimodal evaluation weights each control by 0.25. e-transfer
- Supply camera or robot-action conditions
Driving views are concatenated along latent time with per-view embeddings and RoPE. Robot camera control adds projected Plücker raymaps before self-attention, training only self-attention and camera projection. The action variant instead adds an action-MLP output to DiT timestep embeddings. e-driving-methode-camerae-action
- Keep policy learning separate
For real-robot augmentation, translated images retain demonstration actions and joint states; a separate Diffusion Policy learns control. The VLA-data pathway generates instruction-conditioned videos and obtains pseudo-actions through a separate latent-action model or inverse-dynamics model. e-robot-augmente-robot-setupe-vla
6. Experiments & results
Cosmos-Predict2.5 turns text and optional visual context into video using a flow-matching transformer, then specializes that backbone for spatial control, multiple cameras and supplied robot actions. Cosmos-Transfer2.5 also augments observations for a separately trained robot policy. The strongest execution evidence is a small real-robot robustness experiment; most other evaluations measure generated-video quality, consistency or instruction following. This is a reading of the February 2026 v2 source, not a comparison with its 2025 submission.
6.1 Read the original evidence
Table 11. Image-conditioned video scores improve, but the components tell different stories. Original paper, p. 16 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each row across before comparing the overall column. The Domain Score is VQA-based across physical-AI domains, while the Quality Score combines eight metrics adapted from VBench; their average gives the reported Overall Score. The paired Cosmos rows separate pretraining from post-training within each model size. The 2B model rises from overall 0.799 to 0.810, and the 14B model reaches the same 0.810 through a different balance of domain and quality scores. Then inspect the Wan2.2-27B-A14B row: its domain score is the highest listed, despite its lower overall result. e-pai
What it supports. The two post-trained Cosmos models tie at 0.810 overall, above the listed Wan2.2-27B-A14B score of 0.806. This is not a clean size-scaling conclusion: 2B has the higher Cosmos domain score, while 14B has the higher quality score. The benchmark averages conceal that tradeoff.
Where the evidence stops. These are image-conditioned video scores without reported uncertainty. Table 10’s separate Text2World setting instead places Wan2.2-27B-A14B at 0.769 versus Cosmos at 0.768. Do not transfer the Image2World ordering to every conditioning mode.
Table 13. Translated demonstrations improve a separate policy’s real-robot robustness on one bimanual task. Original paper, p. 23 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Rows identify policy training data: Base uses the original demonstrations, Baseline adds conventional image augmentation, and Proposed uses Cosmos-Transfer2.5 translated observations. Columns hold the base setting and nine changes to objects, lighting, table or background; Combo combines the mangosteen, orange bowl and beige table changes. Every cell counts successes out of three matched trials. The underlying task remains grasping an apple and bowl with separate arms, placing the apple inside and setting the bowl down. The augmentation stage makes five visual variants of each of 100 demonstrations while retaining recorded actions and joint states. The video model therefore changes policy training observations rather than selecting deployment actions. e-robot-setupe-robot-augmente-robot-resultse-robot-prose
What it supports. The proposed policy completes 24/30 trials, compared with 5/30 for standard augmentation and 1/30 for original data alone. Its failures remain visible: beige and black tablecloth columns each show only 1/3. The result is stronger evidence for practical control utility than a generated-video preference score, within this narrow setup.
Where the evidence stops. Only three trials support each scenario. Page 24 says the baseline succeeds in just one case, whereas this table has two nonzero scenario columns and five successes. The edition retains the table’s explicit counts and leaves the prose discrepancy unresolved.
Table 19. Recorded robot actions condition a more accurate video predictor. Original paper, p. 34 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. The four arrows define the comparison: PSNR and SSIM rise for better agreement, while Latent L2 and FVD fall. Compare the lower 2B row with the earlier 7B action-conditioned baseline, keeping all four metrics together. Section 6.6 evaluates 100 randomly sampled episodes from the official Bridge test set, using 320×256 videos at 5 fps. Each prediction starts from an image and supplied robot actions; subsequent chunks use the last generated frame. The action vector describes relative gripper displacement, rotation and width. Thus the row reports how well a model predicts observations under an action sequence, rather than how well it chooses that sequence. e-actione-bridgee-action-ablation
What it supports. Predict2.5 improves all four reported metrics: PSNR rises from 21.14 to 24.95, SSIM from 0.82 to 0.85, latent error falls from 0.32 to 0.28, and FVD from 190 to 146. The source supports better prediction on the sampled Bridge episodes.
Where the evidence stops. The comparison changes model family, scale and training. It does not isolate the action-injection mechanism; Table 20 is the closer comparison. Nor does it establish that predicted rollouts correctly rank candidate robot policies in a physical environment.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| PAI-Bench Image2World generation PAI-Bench predict task; post-trained models. | 2B: 0.840 / 0.779 / 0.810; 14B: 0.838 / 0.781 / 0.810. Domain / Quality / Overall score, higher is better | Wan2.2-27B-A14B: 0.841 / 0.772 / 0.806; pretrained 2B overall: 0.799. Overall averages VQA-based domain and VBench-derived quality scores. The lead concerns this video benchmark; no uncertainty is reported. e-pai |
| PAI-Bench Text2World generation PAI-Bench predict task, text-only conditioning. | Post-trained 2B and 14B: 0.768 each. Overall score, higher is better | Wan2.2-27B-A14B: 0.769; pretrained 2B: 0.751. The post-trained Cosmos models do not lead the listed Text2World overall scores. e-pai |
| Real-robot visual robustness Apple-into-bowl bimanual task; 100 demonstrations, five synthetic variants each; ten scenarios, three trials per scenario with matched poses/configurations. | Transfer-augmented policy: 24/30. Successful executed trials | Standard image augmentation: 5/30; unaugmented policy: 1/30. Evidence supports this policy’s visual robustness on one task. Page 24’s statement that the baseline succeeds in just one case is inconsistent with Table 13’s two nonzero scenario columns; table counts are retained. e-robot-setupe-robot-augmente-robot-resultse-robot-prose |
| Bridge action-conditioned video prediction 100 randomly sampled official Bridge test episodes; 320×256 videos at 5 fps; image and recorded 7D actions condition prediction. | 24.95 / 0.85 / 0.28 / 146. PSNR↑ / SSIM↑ / Latent L2↓ / FVD↓ | Predict1-7B action-conditioned baseline: 21.14 / 0.82 / 0.32 / 190. All reported video metrics improve; policy ranking accuracy and executed task success are not tested here. e-actione-bridge |
| Bridge action-conditioning ablation TimeEmbedding, CrossAtten and ChannelConcat variants of the 2B action-conditioned model. | TimeEmbedding: 24.95 / 0.85 / 0.28 / 146. PSNR↑ / SSIM↑ / Latent L2↓ / FVD↓ | CrossAtten: 24.41 / 0.84 / 0.28 / 159; ChannelConcat: 23.11 / 0.78 / 0.35 / 267. Time conditioning wins or ties the listed metrics. Missing seed variability and detailed capacity matching limit causal attribution. e-actione-action-ablation |
| Multicamera robot video synchronization 80 validation robotic-manipulation videos, 16 camera trajectories; joint multiview versus single-target-view generation. | Multiview: 19.73; 0.08; 0.20. Sampson error (px)↓; translation error↓; rotation error (rad)↓ | Single-view: 26.61; 0.08; 0.19. View consistency improves while rotation error slightly worsens; translation-error units are not stated. e-camera |
| Driving control adherence 1,000 held-out multiview clips; BEVFormer cuboid evaluation against labels. Table captions say RDS-HQ-HL; evaluation prose says RQS-HQ. | Transfer2.5: 0.394. Cuboid LET-AP↑ | Transfer1: 0.243; real-video reference: 0.476. Detection improves but remains below the real-video reference. Dataset naming is unresolved; these are perception metrics, not closed-loop driving outcomes. e-driving-results |
| DreamGen GR1 instruction following SFT models; generated videos judged by GPT-4o and Qwen2.5VL on object, behavior and environment generalization. | Predict2.5: object 91.8 / 69.4; behavior 70.2 / 59.6; environment 69.0 / 69.0. Reported instruction-following scores, GPT / Qwen | Wan2.1 behavior GPT: 72.3; earlier Predict2 behavior Qwen: 61.7. Object scores lead, but behavior scores do not. This benchmark judges generated behavior rather than a VLA policy trained on the resulting data. e-vlae-dreamgen |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 5. Preference gains after RL coexist with a substantial fraction of ties. Original paper, p. 13 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Each bar is a separate before/after comparison, so first read the legend rather than treating the two green segments as a direct contest. For the pretrained model, green occupies 41.1%, light blue 18.9% and gray 40.0%. For the merged model, the corresponding shares are 37.0%, 16.3% and 46.7%. Keep the gray segment in the denominator: the green values are not win rates restricted to decisive votes. Section 4.2.2 describes the 2B post-training procedure, where VideoAlign scores eight generated outputs per condition and group-normalized rewards drive updates regularized by the diffusion loss. e-rl
What it supports. RL-generated videos win more often than their own pre-RL baselines in both comparisons. In the merged comparison, 37.0% prefer the RL output against 16.3% preferring the earlier output, while ties form the largest category at 46.7%. This supports a preference benefit without implying universal improvement.
Where the evidence stops. The figure supplies neither confidence intervals nor a vote count. It does not compare pretrained-plus-RL directly with merged-plus-RL. Reward optimization and human preference also do not by themselves test correct physical dynamics or robot task completion.
Table 20. Where actions enter the DiT affects prediction quality. Original paper, p. 35 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Hold the model name fixed and compare the second line of each method cell. TimeEmbedding is the proposed route: an MLP converts robot actions into tensors that are added to DiT timestep embeddings. CrossAtten and ChannelConcat are alternative conditioning routes. Read the same four metric directions used in Table 19. The proposed row has the best PSNR, SSIM and FVD, while its latent L2 ties CrossAtten at 0.28. ChannelConcat performs substantially worse on all four metrics. This table is useful because it narrows attention from an entire model-family change to a specific design choice inside the action-conditioned predictor. e-actione-action-ablation
What it supports. TimeEmbedding reports FVD 146 versus 159 for CrossAtten and 267 for ChannelConcat. Its PSNR is 24.95 versus 24.41 and 23.11. The result motivates timestep conditioning as the reported default, while preserving the latent-error tie with cross-attention rather than claiming a win on every metric.
Where the evidence stops. The paper does not provide seed variability, detailed capacity matching or a full fine-tuning schedule for these rows. Actions are inputs in all variants. This ablation cannot establish joint action-and-future prediction or a generally optimal action representation.
7. Analysis & limitations
7.1 What the evidence leaves open
Generation scores and human preference do not establish physical correctness or reliable closed-loop policy validation. RNDS normalizes each long-video DOVER ratio by its first chunk on 17 videos, so it measures relative quality evolution, not absolute dynamics accuracy. e-paie-longe-action
The real-robot study has only three trials per scenario and one manipulation task. Some test edits overlap the augmentation range; the paper distinguishes stronger distribution shifts separately. No uncertainty or multi-seed policy analysis is supplied. e-robot-results
Gains over earlier families combine data, architecture and training changes. Driving Table 14 improves visual metrics but worsens Transfer temporal/cross-view Sampson errors, so fidelity and geometric consistency should not be conflated. e-introe-driving-results
7.2 Questions for discussion
- Does better action-conditioned video prediction preserve the ranking of real robot policies?
- Does semantic video augmentation retain action-label validity under the strongest scene edits?
8. Reproducibility audit
8.1 Requirements and known gaps
Full pretraining requires the curated corpus and auxiliary models. Table 9 measures training efficiency on 4,096 H100 GPUs at 720p/93 frames; this is not a minimum inference requirement or a complete compute budget. Corpus access, total stage iterations and exact merge weights are not specified sufficiently for exact retraining. e-datae-pretraine-sfte-compute
For a smaller reproduction, preserve Bridge action coordinates, frame rate, split and rollout horizon. The source gives the seven action components but omits the fine-tuning schedule, seed list and complete inference configuration for this variant. Match these before interpreting Table 20 as a controlled comparison. e-actione-action-ablation
For robot augmentation, reproduce global edge control, robot-masked blur and guidance scale 3 while retaining actions. The paper names medium/very-low thresholds but leaves their numerical values and several default settings unstated. An executable reconstruction also needs the policy training recipe and original demonstrations. e-robot-setupe-robot-augment
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Does timestep conditioning use the supplied actions?
Reader-proposed check, not performed: fine-tune TimeEmbedding, CrossAtten and ChannelConcat from the same 2B checkpoint on the same Bridge split, with matched update budgets and disclosed trainable parameter counts. Fix the 100 held-out episodes, initial frames, rollout lengths, sampling settings and random seeds. Report all four Table 20 metrics with episode-level uncertainty, then repeat inference with temporally shuffled actions as a negative control. Genuine action use predicts worse matching to the recorded future after shuffling; an advantage that vanishes under matched budgets would weaken the architectural attribution. Publish the omitted action normalization and chunk configuration so the comparison is reproducible. e-actione-bridgee-action-ablation
Check 2: Does semantic augmentation preserve labels and improve robustness?
Reader-proposed check, not performed: use the same 100 demonstrations and policy architecture to compare original data, conventional augmentation and five Transfer variants per demonstration. Match policy updates and observation exposures so extra training volume is controlled. Inspect frame/action alignment and robot geometry before training, preserving global edge control, robot-only blur and guidance 3 while documenting numerical thresholds. Evaluate the ten original scenarios with additional matched trials and multiple policy seeds; also reserve new semantic edits that never appear in augmentation prompts. The proposed mechanism predicts gains concentrated on visual shifts with valid action labels. Disappearing gains after exposure matching, or geometry-altering edits that break labels, would challenge that explanation. e-robot-setupe-robot-augmente-robot-results
8.3 Reading coverage
Visual audit: Inspected the title and arXiv version, contributor page, flow equations, all twenty tables, and Figures 1–13 and 18–20 on the declared pages. This includes the architecture, all retained training and hardware details, benchmark definitions, robot protocols, multiview results, action representation and ablation evidence. All six final original crops were visually inspected; the robot table was recropped to retain every scenario and total. Figure 2’s arrows were checked against its caption and Equations (1)–(3), preserving the future-work vision branch and velocity-versus-denoised-token distinction. Numerical comparisons use the table cells; the robot prose discrepancy and driving dataset-name mismatch remain disclosed. Figures 14–17 were not visually inspected; their text and captions were read in the complete source. Separate supplements, linked code and full video playback remain outside this reading.
PDF pages inspected for this edition: 1, 3, 4, 5, 6, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 28, 31, 32, 33, 34, 35, 36, 37. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract; Contents
- 1. Introduction
- 2. Data; 2.1 Video Curation Pipeline; 2.2 Domain Specific Data; 2.2.1 Robotics; 2.2.2 Autonomous Driving; 2.2.3 Smart Spaces; 2.2.4 Human Dynamics; 2.2.5 Physics
- 3. Method; 3.1 Flow Matching; 3.2 Network Architecture
- 4. Training; 4.1 Pre-training; 4.2.1 Supervised Fine-tuning; 4.2.2 Reinforcement Learning; 4.2.3 Timestep Distillation; 4.3 Infrastructure
- 5. Results
- 6. Applications; 6.1 Cosmos-Transfer2.5, including results and long video generation; 6.2 Robot Policy Learning, including system, augmentation and experiments; 6.3 Driving Simulation, including architecture, datasets and experiments; 6.4 Multi-view Generation with Camera Control; 6.5 Synthetic Data Generation for VLA training; 6.6 Action-Conditioned World Generation
- 7. Related Work; 8. Conclusion
- A.1 Contributors; A.2 Acknowledgments
- References, PDF pp. 38–44
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- Identity and edition: the title matches exactly. The supplied PDF is arXiv:2511.00062v2, marked 24 February 2026, with a separate title-page date of 2026-2-26. The catalog records the 2025 submission; v1 was not supplied or compared, so revision-level scientific changes are not established.
- The title page credits NVIDIA and points to Appendix A for contributors. The catalog expands those contributor names and includes a stray colon token. Metadata preserves the observed corporate author credit; no individual affiliations are inferred.
- The acquisition noted that text extraction does not reconstruct figure images. This reading supplements the complete text with original PDF inspection of the figures and tables listed below; Figures 14–17 were read through their extracted captions and surrounding text but were not visually inspected.
- Separate supplemental material availability has not been fully verified; no separate supplement was supplied.
- All 11 supplied text chunks were individually read, covering all 44 pages. Code, checkpoints and linked external works were not inspected, and no experiments were reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e-identityPDF p. 1, title, author, date and arXiv margin; p. 37, Appendix A.1
The exact title is World Simulation with Video Foundation Models for Physical AI. Corporate author NVIDIA points to named contributors in A.1. The arXiv margin identifies v2, 24 February 2026; the title-page dateline reads 2026-2-26.
Go to primary source ↓e-introPDF p. 3, Section 1, Introduction
Motivates simulation and synthetic data for Physical AI; describes combined data, model and training changes and specialized applications.
Go to primary source ↓e-dataPDF pp. 4–6, Sections 2.1–2.2; Figure 1
Seven-stage curation retains approximately 200 million clips; sources include proprietary and internet videos. Qwen captioning and semantic deduplication precede sharding; five physical domains supplement general training.
Go to primary source ↓e-flowPDF p. 8, Section 3.1, Equations (1)–(4); p. 9, continuation
Defines linear data/noise interpolation, velocity target, conditional MSE loss and shifted logit-normal timestep sampling.
Go to primary source ↓e-architecturePDF pp. 9–10, Section 3.2, Table 3 and Figure 2 with caption
Defines 2B/14B DiTs, WAN2.1 VAE compression and patchification, Reason1 multilayer text features, relative positional encoding and prefix replacement. Figure 2 caption reserves Reason1 visual inputs for future work.
Go to primary source ↓e-pretrainPDF pp. 10–11, Section 4.1 and Table 4
Progressive resolution/task curriculum, conditioning-mask tokens, 93-frame sequences, shifted noise schedule, high-noise oversampling and AdamW settings. Final-stage text says 0/1/2 condition frames after earlier 1/5 pixel-frame description.
Go to primary source ↓e-sftPDF pp. 11–13, Section 4.2.1, Table 5, Figures 3–4 and continuation before Section 4.2.2
Domain-specific SFT uses 30k iterations and batch 256; 4K cooldown and merging follow. Candidate selection uses hand-picked challenges then larger human evaluation; model soup is selected.
Go to primary source ↓e-rlPDF p. 13, Section 4.2.2 and Figure 5; p. 14, Table 6
VideoAlign rewards guide 2B post-training with eight outputs, 20 diffusion steps, 256 training steps and batch 32. Figure 5 pretrained comparison: RL wins 41.1%, before-RL wins 18.9%, ties 40.0%; merged comparison: 37.0%, 16.3%, 46.7%. Table 6 reports reward components separately.
Go to primary source ↓e-distillPDF p. 14, Section 4.2.3 and Table 7; p. 15, Table 8
rCM combines consistency and distribution-matching distillation for four-step sampling. Overall teacher/distilled scores are 0.768/0.764 for T2W and 0.810/0.816 for I2W.
Go to primary source ↓e-computePDF pp. 14–15, Section 4.3; p. 15, Table 9
FSDP2, context parallelism and selective checkpointing support training. Table 9 reports 4,096 NVIDIA H100 GPUs, 720p and 93 frames; it is a training-efficiency measurement.
Go to primary source ↓e-paiPDF p. 15, Section 5, Benchmarking; p. 16, Tables 10–11, all rows and score columns
Defines domain, quality and averaged overall scores. Table 11 gives post-trained 2B and 14B overall 0.810 versus Wan2.2-27B-A14B 0.806. Table 10 gives both Cosmos post-trained models 0.768 versus Wan2.2-27B-A14B 0.769.
Go to primary source ↓e-humanPDF p. 16, Human Evaluation; p. 17, Figures 6–7
Pairwise human votes retain ties. Against Wan2.1-14B, post-trained 2B wins 33.0% and 14B wins 48.6%; these are separate comparisons.
Go to primary source ↓e-transferPDF pp. 18–19, Section 6.1; p. 19, Table 12
Spatial controls are edge, blur, depth and segmentation. Four control blocks are distributed through the backbone. Each branch trains for 100k iterations at batch 64; full multimodal evaluation uses uniform 0.25 weights on 600 videos.
Go to primary source ↓e-longPDF pp. 19–21, Section 6.1.2, Equation (5) and Figure 10
Seventeen videos span 30–120 seconds. RNDS divides the generated/ground-truth DOVER ratio at each chunk by that ratio at chunk one, then averages curves. Figure 10 compares long-video quality evolution across four controls.
Go to primary source ↓e-robot-setupPDF pp. 20–22, Section 6.2 and 6.2.1; Figure 11
A fixed-base bimanual platform uses Kinova arms, Robotiq grippers and a head camera. One apple/bowl task supplies 100 demonstrations. Separate UNet Diffusion Policy uses visual input and gripper state, predicting eight-step chunks at 10 fps.
Go to primary source ↓e-robot-augmentPDF pp. 22–23, Section 6.2.2 and Figure 12
Global edge control and robot-only blur use Grounding DINO/SAMv2 masks, medium edge threshold, very-low blur threshold and guidance 3. Five variants per original video change only observations; actions and joint states remain unchanged.
Go to primary source ↓e-robot-resultsPDF p. 23, Section 6.2.3 and Table 13, all scenario columns and Total
Three trials per ten settings use fixed poses/configurations for comparisons. Total successes are base 1/30, standard-augmentation baseline 5/30, proposed 24/30; proposed beige/black tablecloth outcomes are each 1/3.
Go to primary source ↓e-robot-prosePDF p. 24, paragraphs below Figure 13; compare p. 23, Table 13
Prose confirms proposed 24/30 but describes the baseline as succeeding in just one case. Table 13 instead has baseline 3/3 in Base and 2/3 in Orange Bowl, totaling 5/30.
Go to primary source ↓e-driving-methodPDF p. 25, Sections 6.3.1–6.3.2; p. 28, Sections 6.3.2–6.3.3
Views concatenate along latent time with independent tokenization, learned per-view embeddings and per-view RoPE. World scenario maps control generation. Training and evaluation configurations are specified separately.
Go to primary source ↓e-driving-resultsPDF p. 25, Tables 14–15; p. 28, Section 6.3.3
Table 15 LET-AP is 0.394 for Transfer2.5, 0.243 for Transfer1 and 0.476 for real video. Table 14 Transfer TSE/CSE worsen from 1.017/1.835 to 1.246/2.310 while visual metrics improve. Evaluation uses 1,000 disjoint clips; captions say RDS-HQ-HL and prose RQS-HQ.
Go to primary source ↓e-cameraPDF p. 28, Section 6.4 architecture; p. 31, architecture continuation, experiments and Table 17
Projected Plücker raymaps add camera conditioning before self-attention; other layers are frozen. On 80 videos/16 trajectories, multiview versus singleview yields Sampson error 19.73 versus 26.61 px, rotation 0.20 versus 0.19 rad and translation error 0.08 for both.
Go to primary source ↓e-vlaPDF p. 32, Section 6.5
Instruction-conditioned robot videos can receive pseudo-actions from a separate latent action model or IDM. DreamGen evaluates generated-video instruction following using VLMs across object, behavior and environment generalization.
Go to primary source ↓e-dreamgenPDF p. 34, Table 18, GR1 instruction-following columns and caption
Predict2.5 object GPT/Qwen 91.8/69.4, behavior 70.2/59.6 and environment 69.0/69.0. Wan2.1 has behavior GPT 72.3; earlier Predict2 has behavior Qwen 61.7. Caption identifies all entries as SFT models.
Go to primary source ↓e-actionPDF p. 33, Section 6.6
Image plus action sequence conditions future video; subsequent chunks use the last generated frame. Action MLP outputs add to DiT timestep embeddings. Bridge actions are relative gripper displacements, rotations and width; test set sample is 100 episodes at 320×256 and 5 fps.
Go to primary source ↓e-bridgePDF p. 34, Table 19 and Figure 20 with caption
New/old action-conditioned models report PSNR 24.95/21.14, SSIM 0.85/0.82, latent L2 0.28/0.32 and FVD 146/190. Selected frames qualitatively compare object permanence.
Go to primary source ↓e-action-ablationPDF p. 34, final paragraph; p. 35, Table 20, all rows
TimeEmbedding yields 24.95/0.85/0.28/146; CrossAtten 24.41/0.84/0.28/159; ChannelConcat 23.11/0.78/0.35/267 in PSNR/SSIM/latent L2/FVD order. Seed variation and capacity-matching details are not reported here.
Go to primary source ↓8.5 Primary sources
World Simulation with Video Foundation Models for Physical AI ↗
PDF · 18,755 extracted words
Source fingerprint
2597170ef9e8128c1ec3c87aa1eaf321cd599268451da1f61aead325eca576fe