VideoWorld 2: Learning Transferable Knowledge from Real-world Videos
1. Paper overview
In one sentence: A diffusion appearance prior helps compact dynamics codes transfer across visual settings, while autoregressive code prediction still requires separate supervision to become a robot policy. e02e03e05e09e12
| At a glance | What to know |
|---|---|
| Research problem | Source description Real-world demonstrations entangle useful manipulation changes with backgrounds, lighting, materials and camera motion. The authors ask whether compact dynamics representations can preserve multi-step task structure across new appearances. Their hypothesis assigns appearance reconstruction to pretrained video diffusion so latent codes carry transferable dynamics. Broad claims about learning rules, reasoning and planning are operationalized through task completion, rather than separate reasoning tests. e02e05e08 |
| Core mechanism | |
| A key reported result | Seven-step paper-folding video generation — OpenX plus Craft: 72.3 Final sequential success rate (%). Latent learning on OpenX plus Craft; AR policy trained on Craft; same craft evaluation. VideoWorld: 31.9; CoLA: 40.2. Gains are 40.4 points over VideoWorld and 32.1 over CoLA; OpenX improves VideoWorld 2 by 3.5 points over its Craft-only row. e09e11 |
| Reading caution |
Core contributions
- Source description
Video-CraftBench covers airplane/boat folding and tower/horse/person block construction: approximately seven hours, 9.5k clips and 150 separately collected test videos with changed appearances or arrangements. CALVIN separately tests action-label efficiency and cross-domain latent pretraining. e08e12
Figure 4. Compact codes guide diffusion directly; a detached coarse reconstruction supplies motion cues. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start at the video frames along the bottom. The causal encoder produces green visual features; learnable queries summarize growing prefixes in the left inset, where hatched cells indicate masked future features. Quantization produces the pink dynamics codes. The first-frame feature bypasses compression into the coarse decoder, while codes take both a decoder route and a direct route into the VDM. Follow the two red sg circles: one interrupts the decoder’s gradient to codes, and the other interrupts the reconstruction-to-ControlNet path. The direct code-conditioning route has no sg marker. Read the reconstruction objective together with the warm-up description; the diagram alone does not specify the later loss schedule. e04e05e06e13e14e17
What it supports. The coarse decoder remains useful even when its pictures are blurry: it supplies temporal motion structure that the appearance model can refine. The proposed benefit is a better learning signal for transferable codes, rather than simply a prettier final renderer. The gradient-path ablation tests this design more directly than the architecture picture does.
Where the evidence stops. Figure 4 prints K=(T−1)/s, while Section 3.1 prints K=1+floor((T−1)/s). Section 3.2 describes up-to-current-time attention; Section 5.5 says current-step-only. Algorithm 1 also incompletely specifies detachment and loss switching. These implementation conventions remain unresolved.
2. Motivation
2.1 The problem and the proposed response
Real-world demonstrations entangle useful manipulation changes with backgrounds, lighting, materials and camera motion. The authors ask whether compact dynamics representations can preserve multi-step task structure across new appearances. Their hypothesis assigns appearance reconstruction to pretrained video diffusion so latent codes carry transferable dynamics. Broad claims about learning rules, reasoning and planning are operationalized through task completion, rather than separate reasoning tests. e02e05e08
2.2 What this reading follows
A paper-folding video contains both a procedure and many incidental details: the tabletop, paper texture, hands and camera view. VideoWorld 2 tries to learn the procedure through discrete visual-dynamics codes while a pretrained diffusion model supplies appearance modeling. Its central engineering choice is to keep a coarse decoder for motion guidance but restrict its gradient paths, then train an autoregressive transformer on the resulting codes. The reported gains cover two different outcomes: generated craft sequences judged by a learned evaluator, and CALVIN simulator actions after labeled fine-tuning. Reading those protocols separately is essential to understanding the transfer claim. e02e03e05e09e12
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Not assigned |
| Architecture | Not assigned |
| Prediction paradigm | Not assigned |
| Quadrant | Not assigned |
This table preserves the labels recorded at reading time. The current major category is VLA. View the current classification.
3.1 Evidence-based assessment
Insufficient evidence to decide
The recorded taxonomy is unassigned. Evidence supports a modular latent encoder/decoder, diffusion generator and AR policy, followed by a CALVIN action head. It establishes neither one network jointly emitting future observations and physical actions nor an inverse-dynamics controller applied to generated frames. Interpret this as latent-policy pretraining with video reconstruction and supervised action adaptation; no quadrant is assigned. e03e05e12
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Why a blurry decoder survives inside a high-fidelity generator
Start with the original compression problem: an encoder and quantized query codes must explain visual changes, while a decoder reconstructs frames from those codes and the first-frame feature. Section 3.2 argues that forcing this bottleneck to explain every texture and lighting change makes the codes less transferable. VideoWorld 2 introduces a pretrained VDM to shoulder appearance modeling, but initially asking diffusion to infer the task’s motion from noise is slow and unstable. The warmed-up decoder therefore survives as a producer of coarse temporal cues. Its output can be visually weak and still help diffusion place the hand or object trajectory. Gradient stops restrict how that auxiliary route trains the codes, while direct code conditioning guides denoising. This is the intended mechanism; the inconsistent loss schedule and incomplete pseudocode prevent treating the diagram as a complete implementation specification. e04e05e06e17
Figure 6. Transfer is evaluated across appearances and arrangements within five established craft tasks. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the upper rows horizontally: airplane and boat each have an initial state and seven designated milestones. These are reference task steps, not seven independent chances to succeed. The evaluator counts a later step only when all earlier steps have been completed. The lower panels illustrate what changes between training and test environments, including table patterns, paper textures, camera viewpoints and block arrangements. Section 4 reports approximately seven hours of material split into 9.5k clips, with about 150 separately collected test videos. Folding typically lasts 40–80 seconds; the shorter block tasks use final-state judgments instead of the seven-step protocol. e08e09
What it supports. The benchmark tests whether a familiar procedure remains recognizable under changed visual conditions and accumulates correctly across a long sequence. Its folding and block metrics answer different questions: ordered milestone completion versus final arrangement. These example images explain the intended distribution shift but do not quantify its full diversity.
Where the evidence stops. The generator’s test environments are not wholly unseen by the evaluation classifier: classifier training includes frames from both training and testing environments and generated outputs. The figure therefore supports a generator-transfer setup, not an independently held-out evaluator claim.
5.2 A predicted dynamics token is not yet an executable robot action
The second training stage changes the target from reconstructing a clip to predicting its compressed dynamics sequence. The transformer receives the initial image and task instruction, then learns the flattened latent tokens. For crafts, inference returns generated task videos: diffusion decodes the predicted codes, and long sequences continue from the last generated frame of each segment. The CALVIN route has a different endpoint. After latent pretraining, the authors attach an MLP head to transformer hidden states and train it against ground-truth actions with an L2 loss. They use static-camera images and omit wrist views and state parameters. Thus the latent representation is a bridge into a supervised policy, rather than a motor command with a universal meaning. The paper does not specify a diffusion rollout planner or a complete online control schedule for this policy. e03e07e12
5.3 Read transfer through three distinct tests
Reader interpretation: the evidence is strongest when separated into appearance transfer, action-label efficiency and representation diagnostics. Video-CraftBench changes familiar tasks’ backgrounds, materials and arrangements; its final folding score requires a correct prefix of milestones. This is a demanding generation test, but it relies on a learned classifier whose training includes test-environment and generated frames. CALVIN tests something more operational: a policy that executes simulator actions after labeled adaptation, with separate limited-label and OpenX-pretrained regimes. Finally, UMAP illustrates cross-environment alignment for coarse motion directions without measuring full task semantics. Agreement among these tests makes the proposed mechanism plausible, but none independently proves complete disentanglement or general-purpose reasoning. Reproduction should preserve the original protocols while adding independent evaluator checks and controlled gradient-path comparisons. e08e09e12e13e15
5.4 Training and inference
During training
The implementation uses Cosmos AR 4B and Cosmos DiT 2B, 93-frame clips, 480px output and a reported 16 fps. Defaults are four queries and 1,000 FSQ codes from levels [8, 5, 5, 5]. Reconstruction-only warm-up precedes diffusion-guided latent learning and autoregressive cross-entropy training. The default pretrained VDM is fully fine-tuned. e06e16
For OpenX-enhanced craft results, train the latent representation on OpenX plus Craft, but the AR transformer on Craft codes alone. CALVIN instead pretrains the transformer on latent targets before action adaptation. Thus “unlabeled” means action labels are absent from latent pretraining; language conditioning and later action supervision remain present. e03e11e12
During inference
For crafts, an initial image and task instruction condition code prediction and diffusion decoding. Longer videos extend each segment from the preceding segment’s final generated frame. This feedback is generated imagery; the craft experiment does not execute a physical manipulator. e03e07
CALVIN evaluates executable actions from the supervised head in simulation. The source does not establish an inference-time diffusion planner, candidate-trajectory search, or a fully specified observation refresh/action execution schedule. e12
5.5 Implementation flow
- Compress changes causally
A MAGVITv2-style causal encoder produces visual features. Learnable queries cross-attend to progressively longer feature prefixes; quantization discourages copying full future features. The first-frame feature and codes feed a causal decoder. This is visual-transition compression, not an identified physical action space. e04
- Give diffusion two dynamics inputs
Projected codes condition the pretrained VDM through causal cross-attention. The original decoder supplies blurry motion cues through a ControlNet-like branch. Figure 4 marks stops on the code-to-decoder and reconstruction-to-ControlNet paths; the direct code-to-VDM path remains available for learning. e04e05
- Learn codes, then adapt actions
Flatten the codes and train the transformer conditioned on the initial image and task instruction. For CALVIN, extend its vocabulary with dynamics tokens, then add an MLP action head and fine-tune hidden-state-to-action mapping using an L2 loss. Wrist images and state parameters are omitted. e03e12
6. Experiments & results
VideoWorld 2 learns discrete visual-dynamics codes with a pretrained diffusion appearance prior, then trains an autoregressive transformer to predict those codes. It improves generated long-horizon craft sequences and transfers latent pretraining to a separately action-supervised CALVIN policy. Its evidence supports benchmark-specific transfer; generated craft success, simulator control and complete appearance disentanglement remain distinct claims.
6.1 Read the original evidence
Table 1. Complete-sequence gains are strongest in the final folding column; training regimes must stay separate. Original paper, p. 10 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. First separate the three groups: pretrained video generators fine-tuned with stepwise Craft-text, latent methods trained with Craft, and latent methods trained with OpenX plus Craft. In the last group, only representation learning uses both datasets; the AR transformer still trains on Craft. Move across folding columns 1–7 to see cumulative failure, then read the Human, Tower and Horse columns as final-state block judgments. Human is the table’s label for building a person from blocks. The last columns reward higher SSIM and lower LPIPS. Keep LAPA’s N.A. as unavailable: the authors attribute it to severe long-sequence decoding degradation, not a measured zero. e06e09e10e11e16
What it supports. VideoWorld 2 reaches 68.8% complete folding success with Craft and 72.3% with OpenX plus Craft. Under the latter regime, VideoWorld reaches 31.9% and CoLA 40.2%, so the gains are 40.4 and 32.1 percentage points. The same VideoWorld 2 row reports SSIM 0.774 and LPIPS 0.193.
Where the evidence stops. These are classifier-scored generated videos, not physical execution. Baselines differ in architecture and language conditioning; matched compute and uncertainty are not established. The source discussion mentions PSNR, but the retained table actually reports LPIPS.
Table 2. Latent pretraining improves a simulator policy after supervised action adaptation. Original paper, p. 11 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the pretraining and fine-tuning columns before comparing scores. In the upper block, rows 2–4 use the same limited action-label setting: the caption’s 10% data means 2k sampled trajectories, while latent pretraining uses 22k CALVIN trajectories. Row 1 instead uses the full action-labeled set and is a separate reference. In the lower block, OpenX pretraining precedes full CALVIN action fine-tuning; distinguish video targets in row 5 from latent targets in rows 6–7. Columns 1–5 count sequential task success, whereas Avg. Len. is average completed tasks. Appendix A explains the added MLP action head and static-camera-only CALVIN input. e12
What it supports. In-domain VideoWorld 2 reaches average length 1.87 versus 1.11 without latent pretraining and 1.49 for LAPA, but remains below the full-label oracle’s 2.36. Cross-domain VideoWorld 2 reaches 2.88 and 30.9% five-task success, compared with LAPA’s 2.51 and 27.0%. This supports useful pretraining followed by adaptation.
Where the evidence stops. CALVIN results concern simulation and require ground-truth action labels for fine-tuning. ABCD→D is the printed protocol, with D present on the training side; do not describe it as withholding D. Neither these scores nor the video figures establish zero-shot physical robot control.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Seven-step paper-folding video generation — Craft training Video-CraftBench test environments; three rollouts per initial image; every preceding step must succeed. | 68.8 Final sequential success rate (%) | VideoWorld: 0.0; Wan 2.2 14B with Craft-text: 0.0. A 68.8 percentage-point gain over these zero-success baselines, not a defined relative percentage gain. Craft refers to task-data training, not absence of pretrained backbones. e09e10e06 |
| Seven-step paper-folding video generation — OpenX plus Craft Latent learning on OpenX plus Craft; AR policy trained on Craft; same craft evaluation. | 72.3 Final sequential success rate (%) | VideoWorld: 31.9; CoLA: 40.2. Gains are 40.4 points over VideoWorld and 32.1 over CoLA; OpenX improves VideoWorld 2 by 3.5 points over its Craft-only row. e09e11 |
| Block-horse video generation — OpenX plus Craft Video-CraftBench novel test arrangements; classifier judges final state. | 85.8 Final-state success rate (%) | VideoWorld: 49.8; CoLA: 49.9. This shorter-task final-state measure does not impose the seven-step folding protocol. e09e11 |
| CALVIN in-domain latent pretraining with limited action labels ABCD→D; latent pretraining on 22k trajectories, action fine-tuning on 2k trajectories (table label: 10% data). | 1.87; 9.70 Average completed tasks; five-task success (%) | Limited-label oracle: 1.11; 0. LAPA: 1.49; 2.30. Full-label oracle: 2.36; 24.6. Latent pretraining helps label efficiency but does not match the full-label oracle. e12 |
| CALVIN cross-domain latent pretraining Pretraining on the reported 1.3M OpenX dataset; action fine-tuning on 22k CALVIN trajectories, ABCD→D. | 2.88; 30.9 Average completed tasks; five-task success (%) | LAPA: 2.51; 27.0. Video-pretrained oracle: 2.46; 23.0. Supports transfer followed by labeled adaptation, not zero-shot robot execution. e12 |
| Detached coarse-motion conditioning ablation Table 3a, Craft-only; compare pretrained VDM plus decoder stop-gradient, without/with ControlNet. | 68.8; 0.205 with ControlNet Paper success (%); LPIPS (lower is better) | 47.3; 0.275 without ControlNet. Motion guidance adds 21.5 success points and lowers LPIPS by 0.070. Nearby prose’s approximately 0.9 LPIPS improvement conflicts with these cells. e13 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 3a–f. Motion guidance and gradient routing matter alongside latent capacity and temporal context. Original paper, p. 12 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Begin with panel (a), whose checkmarks expose the mechanism most directly. After introducing pretrained VDM, compare decoder stop-gradient alone, ControlNet alone and their combination. The final row includes all three switches. Then inspect panels (b), (d) and (e) to see how query count, codebook size and context length alter representation capacity. Panel (c) varies projection and attention, while panel (f) distinguishes random initialization, freezing, LoRA and full fine-tuning of the VDM. Retain the caption’s protocol: Craft-only training and in-domain CALVIN evaluation. Compare within a subtable first; similarly labeled configurations across panels are not fully reconciled by the text. e13e14
What it supports. With VDM and decoder stop-gradient fixed, adding ControlNet raises paper success from 47.3% to 68.8% and lowers LPIPS from 0.275 to 0.205. More capacity is not uniformly better: eight queries improve LPIPS over four, yet lower paper success from 68.8% to 65.0%. Context also matters: T=2 yields 19.1%, versus 68.8% at T=93.
Where the evidence stops. The prose’s approximately 0.9 LPIPS improvement conflicts with the legible changes of 0.084 or 0.070. Panel (c) ends at 72.3/80.9 rather than the 68.8/77.5 found elsewhere; the source leaves that configuration difference unresolved. No error bars are reported.
Figure 7. Cross-environment mixing in a projection supports the hypothesis, but does not prove disentanglement. Original paper, p. 12 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use the legend before interpreting the point clouds. Each movement direction has two colors, one for CALVIN and one for Bridge; color therefore jointly encodes motion and environment. Compare each same-direction color pair within a panel, rather than measuring distances between the two separately displayed maps. With VDM on the left, corresponding environment colors intermingle more; without it on the right, they separate more strongly. The highlighted boxes and arrows connect rightward-motion regions to example robot frames. Section 5.5 says the diagnostic samples 4,000 trajectories and labels them with four coarse arm directions: up, down, left and right. e15e13
What it supports. The visible pattern is consistent with codes becoming less dominated by which environment produced the video. That is useful support for the paper’s explanation of transfer, especially alongside the performance ablations. It remains a qualitative representation diagnostic rather than a direct measurement of folding competence or robot-policy success.
Where the evidence stops. UMAP can alter neighborhood appearance and global distances. The source provides no quantitative invariance metric, projection uncertainty or test of fine contact distinctions here. Similar colors overlapping cannot establish that all appearance information was removed or all necessary dynamics retained.
7. Analysis & limitations
7.1 What the evidence leaves open
Appendix C acknowledges accumulated lighting, texture and color shifts during long generation. Continued scaling is future work. The paper offers no physical craft execution experiment. e07e19e20
The evaluator uses frames from training/testing environments and outputs of all evaluated models. Its 96.1% frame accuracy does not establish independent rollout-level reliability; classifier holdout construction is insufficiently specified. Tables give no uncertainty intervals, and compute-matched superiority is not established. e09e10e12e16
UMAP aligns same-direction CALVIN and Bridge codes better with VDM, but projection overlap does not prove complete appearance invariance or preservation of all task-relevant distinctions. Four coarse arm directions are a limited diagnostic. e15
Other source discrepancies remain: Table 1 reports SSIM/LPIPS although its discussion says SSIM/PSNR; Appendix B states a 55.3% folding-duration share, while Figure 10’s folding slices sum to 75.3%. Neither inconsistency is silently repaired. e10e18
7.2 Questions for discussion
- Would independent rollout-level judgments preserve the final-step advantage?
- Can latent directions transfer under appearance changes without collapsing distinct contact dynamics?
8. Reproducibility audit
8.1 Requirements and known gaps
Table 4 specifies AdamW, rates 1e-4/3e-4, batches 128/256 and maximum iterations 100k/50k for dLDM/AR. Hardware, runtime, diffusion sampling configuration and exact stage-transition scheduling are absent. Page 1 promises future code/data/model release; it does not verify availability. e01e06e16
Resolve Algorithm 1 before implementation: the encoder returns undefined z, the query function is not called, and the decoder body never uses z. It retains MSE despite Section 5.5 discarding reconstruction loss after warm-up, and does not explicitly detach the ControlNet input. These are source gaps. e17
Proposed checks: repeat the stop-gradient/ControlNet factorial comparison under fixed training budgets, and rescore held-out craft rollouts with blinded human labels and a strictly separated classifier split. These test mechanism and evaluator sensitivity; neither was performed here. e09e13
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Separate motion conditioning from its gradient path
Reader-proposed experiment, not performed: start from the same reconstruction warm-up checkpoint and run the four pretrained-VDM conditions crossing decoder stop-gradient on/off with ControlNet on/off. Hold clip length at 93, query count at four, FSQ vocabulary at 1,000, data split, AR training budget and evaluation initial images fixed. First explicitly document the detach locations and post-warm-up loss because Algorithm 1 is incomplete. Measure complete folding success, LPIPS and variation across repeated seeds; add a time-shuffled coarse-motion control. The mechanism predicts that correctly ordered detached motion cues improve both sequence correctness and visual quality. Equal gains from shuffled cues, or no reproducible advantage from detachment, would weaken the proposed explanation. e04e05e06e13e17
Check 2: Test whether the learned evaluator preserves the model ranking
Reader-proposed experiment, not performed: reserve complete source videos and environments before extracting classifier frames, and exclude evaluated generators’ outputs from that classifier’s training. Generate three rollouts for each fixed test initial image from VideoWorld 2, VideoWorld and CoLA under their reported OpenX plus Craft protocols. Have blinded human reviewers label all seven folding milestones and appearance failures separately. Compare the original-style classifier, the strictly held-out classifier and human judgments using complete-sequence success, per-step false positives and ranking stability, with uncertainty clustered by initial image. A much smaller or reversed advantage under independent labels would identify evaluator dependence; a stable advantage would strengthen the transfer claim without turning video success into physical execution. e08e09e11
8.3 Reading coverage
Visual audit: Personally viewed original PDF pages 1–14 and 20–23, covering title/authors/version, Figures 1–11, Tables 1–4, all retained method and numerical evidence, training configuration, evaluator construction and Algorithm 1. Read all eight complete text chunks, including reference pages 15–19. Inspected all six final crops; Table 3 was expanded and re-inspected to preserve its full protocol caption. Figure 4 arrows, masks and gradient stops were checked against Sections 3.1–3.2, Section 5.5 and Algorithm 1; unresolved notation, attention and loss-schedule discrepancies are disclosed. No external supplemental video, code or later edition was inspected. Static source frames do not establish continuous-video viewing.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 20, 21, 22, 23. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract and title page (p. 1)
- 1 Introduction (pp. 1–4)
- 2 Related Work, including 2.1–2.3 (pp. 4–5)
- 3 Approach, including 3.1–3.2 (pp. 5–8)
- 4 Video-CraftBench, including 4.1–4.2 (pp. 8–9)
- 5 Experiments, including 5.1–5.5 (pp. 9–14)
- 6 Further Discussion with Other Works and 7 Conclusion (p. 14)
- References (pp. 15–19)
- Appendix A Implementation Details (pp. 20–22, Table 4 and Algorithm 1)
- Appendix B Details in Video-CraftBench (pp. 20–21)
- Appendix C Visualizations (pp. 22–23)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Poppler layout text is faithfully retained without manual repairs. Mathematical symbols, table ordering, figure labels and ligatures require inspection against the original PDF; successful extraction is not scientific reading.
- Separate supplemental material availability has not been fully verified.
- All eight supplied text chunks were read individually. Original PDF pages 1–14 and 20–23 were visually inspected, including every figure and table; reference pages 15–19 were read as text. Six final original crops were inspected.
- The reviewed edition is arXiv:2602.10102v1, stamped 10 Feb 2026. The exact title and all eight byline authors match the catalog. Page 1 separately prints Date: February 11, 2026 and spells the corresponding author Xiaojiao Jin, whereas the byline says Xiaojie Jin. These source-internal discrepancies are retained; no other revision was supplied or compared.
- The supplied acquisition time is preserved; its manifest basis is the retained original PDF modification timestamp, not a separately recorded HTTP response time.
- External project material, code, checkpoints, datasets and supplemental videos were not inspected. No experiments were reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title, author byline, affiliation line, arXiv stamp and abstract footer
Title and eight authors match the catalog. Affiliations are ByteDance Seed and Beijing Jiaotong University. Stamp: v1, 10 Feb 2026; body date: February 11, 2026. Byline Xiaojie differs from correspondence Xiaojiao. The abstract promises future open release.
Go to primary source ↓e02PDF pp. 2–4, Introduction, knowledge footnote and contribution list
The work targets transferable long-horizon knowledge from diverse real-world demonstrations; knowledge includes rules, reasoning and planning. Appearance/dynamics entanglement is the authors’ motivating hypothesis.
Go to primary source ↓e03PDF p. 5, Figure 3 and Section 3.1; pp. 7–8, Section 3.2, Auto-regressive transformer
Compression precedes next-token policy training. Latents are flattened; prediction is conditioned on the initial frame and task instruction. Inference predicts codes and decodes video.
Go to primary source ↓e04PDF p. 6, Figure 4 and Section 3.1, Latent dynamic model
Queries attend feature prefixes; quantized dynamics and the first feature condition reconstruction. Figure 4 shows two stop-gradient markers. Its K=(T−1)/s differs from the text’s K=1+floor((T−1)/s).
Go to primary source ↓e05PDF p. 7, Section 3.2, Dynamics-enhanced latent dynamic model; p. 20, Appendix A, dLDM
Direct codes enter VDM through projection/causal cross-attention. Detached coarse decoder outputs provide ControlNet-like guidance. VDM takes initial frame, decoder outputs and codes; attention uses information only up to the current time.
Go to primary source ↓e06PDF p. 9, Section 5.1; p. 11, Section 5.5 final paragraph; p. 14, Training strategies of VDM
Cosmos AR 4B/DiT 2B, 93 frames, 480px, 16 fps, N=4 and FSQ [8,5,5,5] are specified. Reconstruction-only warm-up precedes disentangled training; default VDM weights are fully fine-tuned. Section 5.5 says reconstruction loss is subsequently discarded.
Go to primary source ↓e07PDF p. 8, Section 3.2 inference paragraph; p. 22, Appendix C
Long generated videos chain 93-frame VDM segments through preceding final frames. These generated demonstrations are separate from the simulator-action experiment.
Go to primary source ↓e08PDF p. 8, Figure 6 and Sections 4–4.1; p. 20, Appendix B, Train and test environments
Five craft tasks use roughly seven hours/9.5k clips and 150 separately collected test videos. Folding lasts 40–80 seconds, blocks 20–30 seconds. Test changes include backgrounds, textures, viewpoints, paper appearance, block arrangement and colors.
Go to primary source ↓e09PDF pp. 8–9, Section 4.2; p. 21, Appendix B, Task classifier of Video-CraftBench
Three rollouts per case; folding requires preceding steps; blocks use final-state success. SSIM/LPIPS separately assess visual quality. DINOv2-Base trains on about 25k frames: 15k from train/test environments and 10k from generated videos. Classifier accuracy is 96.1%; exact independent holdout construction is absent.
Go to primary source ↓e10PDF p. 10, Table 1, Craft/Craft-text rows and Section 5.3 discussion
Craft VideoWorld 2 folding: 97.2,95.3,90.0,83.3,81.4,74.6,68.8%; block Human/Tower/Horse: 70.0/81.5/80.9%; SSIM .770, LPIPS .205. Craft VideoWorld and Craft-text video baselines end at 0%. LAPA is N.A. due to decoding degradation. Prose mentions PSNR although the table reports LPIPS.
Go to primary source ↓e11PDF p. 10, Table 1, OpenX & Craft rows and Data scaling with OpenX; p. 11, Section 5.3 continuation
Latent training combines OpenX/Craft; AR training uses Craft. VideoWorld 2 final folding: 72.3%, versus VideoWorld 31.9 and CoLA 40.2. Block Human/Tower/Horse: 74.0/83.0/85.8%; SSIM .774, LPIPS .193. CoLA/VideoWorld Horse: 49.9/49.8.
Go to primary source ↓e12PDF p. 11, Table 2, rows 1–7 and Section 5.4; p. 20, Appendix A, Latent pre-training on CALVIN
In-domain: 22k CALVIN trajectories for pretraining, 2k action-labeled trajectories for adaptation. Cross-domain: reported 1.3M OpenX dataset, then 22k CALVIN. VideoWorld 2 average length/five-task success: 1.87/9.70% and 2.88/30.9%. Appendix A specifies language/initial-image conditioning, static-camera-only CALVIN input and a new supervised MLP action head.
Go to primary source ↓e13PDF p. 12, Table 3a, all rows, caption and Effect of the original decoder paragraphs
Paper/block/LPIPS: baseline 0.0/28.5/.312; VDM 30.3/45.2/.297; plus stop-grad 47.3/54.7/.275; VDM plus ControlNet 51.1/52.0/.213; all three 68.8/77.5/.205. Caption: Craft-only, in-domain CALVIN. Prose’s approximately .9 LPIPS gain conflicts with table differences .084 or .070.
Go to primary source ↓e14PDF p. 12, Table 3b–f; pp. 12–14, Section 5.5 explanations; p. 7, Section 3.2 causal-attention paragraph
N=4/8 gives paper 68.8/65.0 and LPIPS .205/.195. T=2/93 gives paper 19.1/68.8. VDM random/frozen/LoRA/full gives paper 0.0/31.7/50.9/68.8. Table 3c ends at 72.3/80.9, unlike 68.8/77.5 elsewhere, without a reconciled configuration explanation. Page 13 says current-step-only attention; p. 7 says up-to-current-time.
Go to primary source ↓e15PDF p. 11, Section 5.5, Code transferability is enhanced by VDM; p. 12, Figure 7 and legend
4,000 CALVIN/Bridge trajectories are labeled up/down/left/right. Colors identify environment and direction. The VDM panel shows greater same-direction cross-environment mixing; no quantitative invariance metric or UMAP uncertainty is provided.
Go to primary source ↓e16PDF p. 21, Table 4; p. 20, Appendix A; p. 9, Section 5.1
dLDM/AR: AdamW; rates 1e-4/3e-4; weight decay .1/.05; momentum (.9,.99)/(.9,.98); batches 128/256; warmup 2k/5k; max iterations 100k/50k; no augmentations. Schedule is printed WarmipDecayLR; losses are L2 plus denoising versus CE. No hardware/runtime or complete sampling recipe is supplied.
Go to primary source ↓e17PDF p. 22, Algorithm 1, lines 2–7, 10–17, 20–26 and 29–36; p. 11, Section 5.5; pp. 6–7, Figure 4 and Section 3.2
Pseudocode returns undefined z, never calls ldm_qformer and omits z usage inside decoder. Main detaches z before decoder but passes rec to VDM without explicit detach; it returns MSE plus VDM loss without a warm-up switch. Figure/prose supply distinctions missing from pseudocode.
Go to primary source ↓e18PDF p. 20, Appendix B, Task analysis; p. 21, Figure 10
Text states folding is 55.3% and 5.2 hours; Figure 10 labels airplane 39.1% (2.7), boat 36.2% (2.5), summing to 75.3%. The discrepancy is unresolved.
Go to primary source ↓e19PDF p. 22, Appendix C; p. 23, Figure 11
Authors acknowledge accumulated reconstruction noise and lighting/texture/color drift. Figure 11 presents selected static frames, not complete temporal demonstrations.
Go to primary source ↓e20PDF p. 14, Section 7 Conclusion
Continued scaling remains future work; evaluated tasks are the craft-video benchmark and robotic manipulation environment.
Go to primary source ↓8.5 Primary sources
VideoWorld 2: Learning Transferable Knowledge from Real-world Videos ↗
PDF · 11,024 extracted words
Source fingerprint
84cd30bea522cb45732c5c274e5c7ecade4f64162d6635001b8eacb6840432a8