VideoWorld: Exploring Knowledge Learning from Unlabeled Videos
1. Paper overview
In one sentence: Predicting compact multi-step dynamics codes makes video-based task learning more effective, while robot execution still requires a separately supervised action decoder. e-motivatione-ldme-data-qualitye-go-resultse-robot-evale-robot-mapping
| At a glance | What to know |
|---|---|
| Research problem | Source description Can next-token prediction over visual demonstrations acquire rules and useful decisions? Video preserves board patterns and scene appearance but spends many tokens describing a small change. The paper asks whether compressing that change improves learning while retaining visual context. e-motivatione-generatore-ldm |
| Core mechanism | Source description A Latent Dynamics Model (LDM) learns discrete summaries of progressively longer visual changes and supplies extra prediction targets to an autoregressive video transformer. e-ldm |
| A key reported result | Video-GoBench 9×9 Go: 2317 ±25; 99.7%; 83.7%; 88.1%. Tournament Elo; legal rate; Action-Value; best-action accuracy. 300M model; 56,000 test boards from 1,000 matches after removing openings. Eight-agent tournament: 400 games per pair, 11,200 total; KataGo-9d anchored at Elo 2700. Video-only 300M: 1998 ±38, 99.6%, 59.7%, 58.9%. State-based 300M: 2308 ±21 Elo. KataGo-5d: 2253 ±20 Elo. Strong gain over video-only prediction. Action-Value is an oracle-value ratio, not win rate. The 5-dan description concerns calibrated agents, not a human professional tournament credential; reported Elo uncertainty is not training-seed variance. e-datae-evale-go-results |
| Reading caution | Reader analysis The generator avoids direct action supervision, but robot tasks use instructions and labeled IDM actions. Human Go moves are relabeled by KataGo: Table 4e improves from 34.6% to 71.5% Action-Value after relabeling. Training quality therefore depends on expert-generated decisions. e-robot-mappinge-robot-evale-data-quality |
Core contributions
- Source description
A Latent Dynamics Model (LDM) learns discrete summaries of progressively longer visual changes and supplies extra prediction targets to an autoregressive video transformer. e-ldm
- Source description
Video-GoBench provides rendered game data and legality, oracle-move and tournament evaluations; CALVIN and RLBench extend the test to simulated manipulation. e-datae-evale-robot-eval
Figure 3. Learn compact future-change targets first, then predict them together with images. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start at the lower left: demonstration clips enter the LDM, whose codes become training targets for the transformer above. Pink blocks are dynamics codes; green blocks represent image codes, with the upper sequence shifted for next-token prediction. On the right, follow the arrows from frames to causal features, then through masked query attention and quantization to reconstruction. Query q¹ sees the initial and first future feature; later queries see longer prefixes. Although the block is labeled “Causal VQ Encoder,” Section 3.2 explicitly leaves its feature maps unquantized until the query outputs pass through FSQ. Future clips provide training targets; inference predicts these codes from observed history. e-ldme-generatore-robot-mappinge-algorithm
What it supports. The architecture makes future change a short, explicit sequence rather than leaving it distributed across image tokens. The same transformer learns appearance and temporal summaries, but this diagram stops before the independently trained action mapper. Shared token prediction therefore does not establish a single end-to-end robot policy.
Where the evidence stops. Figure 3's mask agrees with the prose. Algorithm 1 instead prints f[:0] and a zero-based f[:(1+h)] prefix that do not implement the stated spans literally. The faithful diagram is retained; these pseudocode indexing issues remain unresolved.
2. Motivation
2.1 The problem and the proposed response
Can next-token prediction over visual demonstrations acquire rules and useful decisions? Video preserves board patterns and scene appearance but spends many tokens describing a small change. The paper asks whether compressing that change improves learning while retaining visual context. e-motivatione-generatore-ldm
2.2 What this reading follows
A Go move changes little of an image, yet it can transform the strategic meaning of the board. VideoWorld uses this mismatch to motivate two representations: image tokens describe what is visible, while latent dynamics codes summarize how it changes. The transformer learns to generate both. The figures below trace that training design into calibrated Go tournaments and simulated robot control, then examine which parts of the design matter. Read the headline performance alongside its conditions: expert-improved Go demonstrations, instruction-conditioned robot tasks, and an action-labeled inverse dynamics model. The evidence supports useful learned policies more directly than a general theory of visual reasoning. e-motivatione-ldme-data-qualitye-go-resultse-robot-evale-robot-mapping
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Not assigned |
| Architecture | Not assigned |
| Prediction paradigm | Not assigned |
| Quadrant | Not assigned |
This table preserves the labels recorded at reading time. The current major category is WAMs. View the current classification.
3.1 Evidence-based assessment
Insufficient evidence to decide
The recorded classification is unassigned, so no existing quadrant can be confirmed. Architecture evidence supports a video generator plus a separately trained inverse-dynamics action module. Joint prediction of frames and latent change codes does not make the complete control system one model jointly emitting executable actions. e-ldme-robot-mappinge-joint
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Turn a long pixel description into a short prediction target
Consider the two jobs assigned to the visual representation. Image tokens retain where stones or objects are and what the scene looks like. Dynamics codes summarize how that scene changes over successive future spans. The LDM learns the latter through reconstruction: a query summarizes an allowed feature prefix, FSQ restricts how much information its code can carry, and the decoder must recover future frames from the initial feature and codes. The transformer then learns to predict those learned codes alongside images. Analysis: this reorganizes supervision around task-relevant change without requiring a human action label for each generator target. It does not guarantee that every learned code is an action, nor that the bottleneck identifies a uniquely causal factor. The horizon and codebook sweeps test the usefulness of this representation empirically. e-ldme-horizone-codebook
Figure 6. Predicted images and executed robot actions occupy distinct branches of the pipeline. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Begin at the bottom: the yellow strip is labeled “Executed Action Frames” and receives control through inverse dynamics. The green strip above is “Synthesized Frames”; the caption identifies these as next-frame predictions during training. Do not merge the two into one executed trajectory. Above the transformer, decoded code visualizations show what the latent predictions represent. At the top, each UMAP panel corresponds to one of nine predicted codes, with colors denoting six task labels. Section 5.5 interprets stronger separation at later codes as longer-range task information. The instructional example is opening a drawer; the color legend identifies the broader task collection. e-probese-robot-mappinge-horizone-robot-eval
What it supports. The figure makes the action boundary explicit: visual predictions become robot controls only after inverse-dynamics decoding. Later-code UMAPs show more task separation, consistent with the authors' temporal interpretation. These are qualitative diagnostics; the task success tables, rather than the appearance of these example frames, establish measured execution performance.
Where the evidence stops. Figure 6 explicitly uses H=9, whereas Table 4b and Section 5.6 give a CALVIN default of H=10. The figure and caption are preserved without harmonizing that discrepancy. UMAP layout is a probe, not proof of causal planning or success on unseen environments.
5.2 Follow the predicted future across the action boundary
At deployment, the future clip used to train the LDM is unavailable. The transformer predicts its dynamics codes and next frame from the observed history instead. For robotics it also receives a task instruction. The IDM turns those predictions into displacement, orientation and gripper controls, which the simulator executes. The resulting observation then joins the history for another prediction cycle. Appendix A describes this as open-loop execution, but also explicitly supplies observation feedback after every action; the report preserves that distinction. The key architectural boundary is between a video-trained generator and a separately action-supervised mapper. Joint CALVIN/RLBench experiments share the generator training across both environments while training a different IDM for each. Thus neither unlabeled generator training nor shared visual tokens imply label-free execution or zero-shot embodiment transfer. e-robot-mappinge-robot-evale-joint
5.3 Separate policy competence from the stronger planning interpretation
The strongest evidence is behavioral: the 300M model improves Go tournament Elo and simulator task success over the corresponding video-only baseline. Mechanism evidence then adds useful qualifications. Longer code horizons improve results, and corrupting the first code damages performance more than corrupting a later one. Yet random code replacement also creates unusual inputs, so sensitivity alone does not prove a particular reasoning procedure. Likewise, UMAP separation is a visualization of representations rather than a controlled test of planning. The code-only ablation is especially informative: most of the Go gain remains without next-frame supervision, although its action mapper is also retrained. Analysis: these findings support compact dynamics targets as an effective learning device more strongly than they establish a general capacity for human-like forward reasoning. e-go-resultse-calvin-resultse-horizone-interventione-probese-targets
5.4 Training and inference
During training
Train LDM with reconstruction L2 loss and the transformer with next-token cross-entropy. The transformer starts randomly initialized; its image encoder is reconstruction-trained on the target dataset and frozen. Table A.1 lists 100,000 LDM and 1,000,000 transformer iterations, with no augmentation. e-training
The separately trained inverse dynamics model (IDM) uses action labels. For robotics, ResNet-18 encodes the predicted image and MLPs process latent codes into seven control outputs. Appendix A specifies mean squared error, AdamW at 1e-4, and one million IDM steps. e-robot-mapping
During inference
Generate H dynamics codes and the next image from history, with an instruction condition for robotics. Feed predictions to the IDM, execute its action in the simulator, append the resulting observation, and predict again. Appendix A calls execution open-loop but describes fresh feedback after each action. e-robot-mappinge-robot-eval
Analysis: this is an observation-feedback cycle at the action boundary. Imagined multi-step codes guide prediction; they are not an executed H-step action trajectory or explicit search over candidate plans. e-ldme-robot-mapping
5.5 Implementation flow
- Encode appearance
A customized MAGVIT-v2 encoder-decoder with finite scalar quantization (FSQ) converts frames into discrete tokens. A Llama-style transformer predicts tokens. For Go and CALVIN, spatial compression yields 4×4 tokens per frame. e-generatore-training
- Compress future change
For the current frame and H successors, a causal encoder without temporal downsampling produces unquantized features. Each query attends through its associated future step; FSQ quantizes the query output. A causal decoder reconstructs future frames from the initial feature and codes. Short clips receive replication padding. e-ldm
- Predict both representations
Interleave image tokens and dynamics codes in one autoregressive sequence. The transformer vocabulary combines distinct image and dynamics codebooks. These codes summarize future changes rather than directly specifying robot actuator labels. e-ldme-robot-mapping
6. Experiments & results
VideoWorld predicts compact multi-step dynamics codes alongside video frames, then converts predictions into task operations. On rendered 9×9 Go and simulated robot tasks, this representation substantially improves over video-only prediction. Expert-curated training data, language conditioning and a separately supervised robot action decoder qualify the broader claim of learning solely from unlabeled videos.
6.1 Read the original evidence
Table 1. Compact dynamics codes recover most of the gap between raw-video and state-based Go learning. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Separate the table into calibrated KataGo opponents, two 300M transformer baselines, and three VideoWorld sizes. The “w/o Search” check marks apply to the transformer agents, while the KataGo rows use search. Compare the two video-input 300M rows to assess the LDM contribution; use the state-input row as a different representation baseline. Legality and best-action accuracy are test-board measures, whereas Elo comes from the internal tournament. Action-Value compares chosen-move values with the KataGo-9d oracle on tournament positions; its percentage is not a win rate. The faded oracle row anchors the scoring system and is preserved as printed. e-go-resultse-evale-data
What it supports. VideoWorld 300M reaches 2317 ±25 Elo versus 1998 ±38 for video-only prediction, and 83.7% versus 59.7% Action-Value. Its 99.7% legality and 88.1% best-action accuracy should be read separately. The state-based model's 2308 ±21 Elo also shows how competitive a compact nonvisual representation remains on this board game.
Where the evidence stops. The experiment uses 9×9 Go and KataGo agents calibrated to human ranks. It does not certify professional human tournament performance or 19×19 strength. Preserve the printed Elo uncertainties; the paper does not present them as repeated-training-seed variation.
Table 2. VideoWorld improves simulated manipulation, and extra demonstrations bring it closer to the action-supervised oracle. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read across each row in the order Push, Open/Close, and Turn on/off; these are task-group success percentages. All listed models have 300M parameters. Start with the video-only Transformer row, then compare ordinary VideoWorld before examining data scaling. The dagger and double-dagger rows add 10k and 30k CALVIN trajectories, respectively, as the retained footnotes specify. Section 5.4 says the added data come from a supervised agent. “Video” in the input/output column describes the generator's training representation; it does not remove instruction conditioning or the separately labeled inverse-dynamics stage used to execute robot actions. e-calvin-resultse-robot-evale-robot-mapping
What it supports. The baseline-to-VideoWorld improvement is large: push success rises from 17.3% to 56.2%, drawer success from 24.1% to 75.4%, and light success from 19.2% to 72.1%. With 30k added trajectories, rates reach 72.7%, 91.0% and 93.8%, approaching the listed oracle while using a larger demonstration budget.
Where the evidence stops. These are simulated task-group rates, not long-horizon chain scores or physical-robot results. The reported ABCD→D split includes D in training. Added-data rows are not equal-data comparisons, and the table gives no uncertainty estimates.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Video-GoBench 9×9 Go 300M model; 56,000 test boards from 1,000 matches after removing openings. Eight-agent tournament: 400 games per pair, 11,200 total; KataGo-9d anchored at Elo 2700. | 2317 ±25; 99.7%; 83.7%; 88.1%. Tournament Elo; legal rate; Action-Value; best-action accuracy | Video-only 300M: 1998 ±38, 99.6%, 59.7%, 58.9%. State-based 300M: 2308 ±21 Elo. KataGo-5d: 2253 ±20 Elo. Strong gain over video-only prediction. Action-Value is an oracle-value ratio, not win rate. The 5-dan description concerns calibrated agents, not a human professional tournament credential; reported Elo uncertainty is not training-seed variance. e-datae-evale-go-results |
| CALVIN push, drawer and light manipulation 300M models; ABCD→D split, instruction conditioning and randomized object placement; Section 5.2 reports sampling 500 tasks. | VideoWorld: 56.2 / 75.4 / 72.1; with 30k extra trajectories: 72.7 / 91.0 / 93.8. Success rate (%), ordered Push / Open-Close / Turn on-off | Video-only: 17.3 / 24.1 / 19.2; action-supervised transformer oracle: 75.4 / 95.3 / 96.2. Extra agent-generated demonstrations narrow the oracle gap but change the data budget. These are individual simulated task groups, not five-task chain scores. No uncertainty is reported. e-robot-evale-calvin-results |
| Joint CALVIN and RLBench manipulation Joint LDM and transformer training on both environments, including 20k RLBench trajectories; environment-specific IDMs. | 56.0 / 74.8 / 74.5; 67.1 / 62.5. Success rate (%), CALVIN Push / Open-Close / Turn on-off; RLBench Microwave / Fridge | Oracle: 61.3 / 79.5 / 78.0; 72.1 / 69.0. Video-only: 6.5 / 13.0 / 15.6; 12.0 / 10.9. Shows learning across two training environments, not zero-shot transfer to an unseen one. No uncertainty is reported. e-jointe-robot-eval |
| Go dynamics horizon and code intervention 50M models; Table 4a horizon sweep and Table 4c inference-time random-code replacements. | H=1: 70.3 / 77.0; H=5: 73.9 / 80.9. Replacing code 1: 46.2 / 42.1; code 3: 72.1 / 80.6. Action-Value / best-action accuracy (%) | Video-only: 47.5 / 44.3; unmodified codes: 73.9 / 80.9. Longer summaries help; early-code corruption is especially damaging. Random replacement also introduces distribution shift, so it does not alone establish human-like planning. e-horizone-intervention |
| Go prediction-target ablation 50M models; code-only training removes next-frame cross-entropy and retrains an IDM using only codes. | Code-only: 73.0 / 78.6; code plus video: 73.9 / 80.9. Action-Value / best-action accuracy (%) | Video-only: 47.5 / 44.3. Most improvement accompanies code prediction. The smaller joint-prediction gain is not an isolated loss ablation because the downstream IDM also changes. e-targets |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 4(a–e). Temporal reach, bottleneck capacity and demonstration quality each affect performance. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read panels (a) and (b) as separate horizon sweeps for Go and CALVIN. H counts the future spans summarized by the LDM, not the number of actions executed without feedback. Panel (c) replaces predicted codes with random tokens: “None” means no intervention, and “All” replaces all codes. Panel (d) varies codebook capacity, with 64,000 entries best among the listed settings. Panel (e) isolates the importance of expert relabeling: the dagger identifies original human moves, while the next row uses KataGo-reannotated moves. All panels use the 50M model, so their baselines must not be substituted for the 300M main-table values. e-horizone-interventione-codebooke-data-qualitye-datae-robot-mapping
What it supports. In Go, moving from H=1 to H=5 raises Action-Value from 70.3% to 73.9%. Replacing the first code lowers it to 46.2%, showing substantial sensitivity to early predicted structure. Human-data relabeling raises Action-Value from 34.6% to 71.5%, a much larger change than this horizon increment and an important qualification to the unlabeled-learning framing.
Where the evidence stops. Random corruption tests sensitivity as well as temporal dependence. It does not uniquely identify planning. Panel (e) lists 6.8M human games, whereas Section 4.1 lists 7.8M while still claiming 10M total; the inconsistent counts cannot be reconciled from the supplied source.
Table 5. Most of the measured gain survives when the transformer predicts dynamics codes alone. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read downward within each column: “video” is the basic frame-prediction system, “code” predicts only LDM targets, and “code/video” predicts both. These are 50M models. The code-only row still receives frames as inputs; it removes supervision on generated next-frame tokens rather than removing vision from the system. Section 5.6 also retrains an IDM that receives only codes. Consequently, this comparison changes both the training targets and the action mapper's inputs. Use the large video-to-code jump to assess the value of the compact target, and treat the smaller code-to-joint jump as a combined-system improvement. e-targets
What it supports. Go Action-Value improves from 47.5% with video targets to 73.0% with code targets and 73.9% with both. For CALVIN, code-only achieves 47.2%, 70.0% and 65.1% success; joint prediction reaches 50.3%, 71.1% and 69.7%. Image prediction adds a consistent but smaller benefit in these reported settings.
Where the evidence stops. The comparison does not isolate next-frame cross-entropy while holding the action decoder fixed. The authors' explanation that frame prediction improves environmental understanding is a hypothesis; no seed variation or statistical uncertainty is shown.
7. Analysis & limitations
7.1 What the evidence leaves open
The generator avoids direct action supervision, but robot tasks use instructions and labeled IDM actions. Human Go moves are relabeled by KataGo: Table 4e improves from 34.6% to 71.5% Action-Value after relabeling. Training quality therefore depends on expert-generated decisions. e-robot-mappinge-robot-evale-data-quality
Evidence is restricted to 9×9 Go and simulated manipulation. UMAP clusters and selected tactics are suggestive probes, not independent proofs of reasoning. Appendix B finds no training-board overlap by move 30 in 400 games; this addresses exact-board reuse, not every form of memorization. e-datae-probese-overlape-conclusion
Source inconsistencies remain: Section 4.1 gives 7.8M human plus 3.2M self-play games while calling the total 10M; Table 4e instead gives 6.8M human. CALVIN uses H=10 in Table 4b/Section 5.6 but H=9 in Figure 6, while Section 5.1 lists frame length 10. e-datae-data-qualitye-horizone-probese-training
The authors report failed Go convergence beyond five-step compression and degradation with an oversized codebook. They identify video generation quality and real-world generalization as remaining challenges. e-horizone-codebooke-conclusion
7.2 Questions for discussion
- Would longer dynamics horizons still help after matching token budget and testing only nonoverlapping late-game boards?
- How much of the code-plus-video gain remains with a fixed code-only action decoder?
8. Reproducibility audit
8.1 Requirements and known gaps
Budget and configuration need reconciliation: Section 5.1 reports roughly four Go days and two CALVIN days on eight A100 GPUs, batches 256/32 and learning rate 3e-4. Table A.1 instead gives transformer rate 3e-5 and LDM rate 5.4e-5, with 30k warmup iterations. e-training
Algorithm 1 is not executable specification: line 6 returns f[:0], an empty Python slice, and zero-based line 11 uses f[:(1+h)], inconsistent with the prose query spanning the current frame through future step h. Figure 3 agrees with the prose mask. Implementation indexing requires resolution. e-algorithme-ldm
The text does not quantify the small labeled IDM dataset, fully specify Go move extraction or decoding settings, or name simulator software versions. Appendix A lists the current frame as an IDM input but details only predicted-frame and latent-code feature branches. e-robot-mappinge-traininge-generator
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Does temporal reach help beyond providing more output tokens?
Reader-proposed check, not performed: train 50M Go variants with H=1 and H=5 using the same expert-reannotated training subset, a 64,000-entry codebook, optimizer schedule and observation budget. Give the H=1 transformer masked padding slots so sequence length and training compute can be matched; document the separate LDM reconstruction budgets. Resolve Algorithm 1's indexing against the prose before either run. Evaluate legality, Action-Value and accuracy on fixed held-out boards, and separately on late-game boards absent from training, with repeated seeds. A reproducible H=5 advantage under these controls would support useful temporal reach; its disappearance would favor token-budget or supervision-volume explanations. e-horizone-codebooke-data-qualitye-evale-overlape-algorithme-ldm
Check 2: Does next-frame supervision improve codes with the decoder held fixed?
Reader-proposed check, not performed: compare code-only and code-plus-video 50M training with the same data and frozen LDM. Decode predicted codes from both variants through the same code-only IDM trained once on the same labeled examples. This removes the changing-IDM-input confound in Table 5. Measure Go Action-Value and accuracy, and CALVIN task-group success under identical instructions and initial-state seeds. Report labeled IDM sample count and decoder learning settings. If joint frame supervision improves performance through this fixed mapper across seeds, it supports better codes; if its advantage disappears, the original result may depend on predicted-frame inputs or decoder retraining. e-targetse-robot-mappinge-robot-evale-eval
8.3 Reading coverage
Visual audit: Read all six text chunks, including references and the embedded supplementary material. Visually inspected the title/version/byline on p. 1; Figures 1–9 and A.1; Tables 1–5 and A.1; and the method, evaluation, execution and pseudocode details on the declared pages. Inspected all six final original crops, retaining table footnotes and figure legends. Figure 3's attention mask was checked against Section 3.2 and Algorithm 1; the pseudocode discrepancy remains explicit. Figure 6's generated/executed branches and H=9 were checked against its caption and the H=10 setting elsewhere. Reference-only pp. 11–13 were read as text, not visually reviewed. No separate supplemental media or code was inspected; no experiment was run.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 14, 15. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract
- 1. Introduction
- 2. Related Work (2.1–2.3)
- 3. VideoWorld (3.1–3.2)
- 4. Video-GoBench (4.1–4.2)
- 5. Experiment (5.1–5.7)
- 6. Conclusion
- References
- Supplementary Material A. Implementation Details, Table A.1 and Algorithm 1
- Supplementary Material B. Details on Video-GoBench
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- Identity/version: the inspected title page identifies arXiv:2501.09781v2 [cs.CV], 5 March 2025. Title and all seven authors match the catalog. The catalog submission date is 16 January 2025; this report reviews the supplied v2, not an independently inspected first version. Changes between versions are not established.
- Acquisition caveat: Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout. This was addressed by inspecting original PDF pages 1–10 and 14–15, including all scientific figures and tables; reference-only pages 11–13 were read as text.
- Separate supplemental material availability has not been fully verified. The supplementary material embedded on PDF pages 14–15 was supplied and read; no separate supplement was supplied.
- Code, project pages, datasets and external videos were not inspected; no experiments were reproduced. The abstract claims an open-source release, but its availability was not independently verified.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e-identityPDF p. 1, title/byline, affiliation block and arXiv margin
The exact title and seven catalog authors are printed with Beijing Jiaotong University and ByteDance Seed; the margin identifies arXiv:2501.09781v2, 5 March 2025.
Go to primary source ↓e-motivationPDF pp. 1–2, Abstract and Section 1; p. 2, Figure 2 and caption
The paper asks about rules, reasoning and planning learned from video, motivating compact visual-change representations. Figure 2 contrasts state, video and video-with-LDM learning curves.
Go to primary source ↓e-generatorPDF p. 3, Section 3.1; p. 4, paragraph immediately below Figure 3
The source defines conditional frame generation and action mapping, using a customized MAGVIT-v2/FSQ tokenizer and Llama-style autoregressive transformer.
Go to primary source ↓e-ldmPDF pp. 4–5, Section 3.2, Latent Dynamics Model and Auto-regressive Transformer; p. 4, Figure 3
Queries summarize successively longer feature prefixes; features remain unquantized until FSQ on query outputs. The decoder reconstructs from the initial feature and codes; the transformer combines image and dynamics vocabularies. Figure 3 masks steps beyond each query span.
Go to primary source ↓e-robot-mappingPDF p. 5, Section 3.2, Mapping to Task Operation; p. 14, Appendix A, Robotics action execution and Inverse dynamics model
An independently trained, action-labeled IDM maps predictions to actions. The robot implementation uses ResNet-18, MLPs and pooling for XYZ displacement, Euler angles and gripper control. A simulator observation is appended after each action, despite the open-loop wording.
Go to primary source ↓e-trainingPDF p. 6, Section 5.1; p. 14, Table A.1 and Appendix A
Section 5.1 specifies frozen reconstruction-trained encoding, 4×4 spatial tokens, 64,000-code defaults, frame lengths, batches and eight-A100 timing. Its 3e-4 learning rate differs from Table A.1 transformer rate 3e-5; the appendix gives LDM 5.4e-5, losses, iterations, warmup and no augmentation.
Go to primary source ↓e-dataPDF p. 6, Section 4.1
Video-GoBench is rendered 9×9 Go, approximately 400M unique training boards, and 56,000 test boards after opening removal. The stated 10M games conflict with the listed 3.2M self-play plus 7.8M human components; human moves are relabeled by KataGo.
Go to primary source ↓e-evalPDF p. 6, Section 4.2, all four metric definitions
Legality and action accuracy use test boards; Action-Value divides the selected move value by the oracle value on tournament positions. BayesElo uses eight agents, 400 games per pair, a stated default confidence of 0.5 and a 2700 anchor for KataGo-9d.
Go to primary source ↓e-robot-evalPDF p. 6, Section 5.1; p. 7, Section 5.2
The CALVIN split is ABCD→D. Instructions condition generation; tasks are push, drawer and light for CALVIN and microwave/fridge for RLBench. Object positions are randomized and the source reports sampling 500 tasks.
Go to primary source ↓e-go-resultsPDF p. 7, Table 1, rows 1–8, all metric columns
VideoWorld 300M reports 2317 ±25 Elo, 99.7% legality, 83.7% Action-Value and 88.1% accuracy; video-only 300M reports 1998 ±38, 99.6%, 59.7% and 58.9%. State-based 300M is 2308 ±21 and KataGo-5d is 2253 ±20 Elo. VideoWorld rows are marked without search.
Go to primary source ↓e-calvin-resultsPDF p. 7, Table 2 and its footnotes; p. 8, Section 5.4, Results on CALVIN
VideoWorld has 56.2/75.4/72.1% success and reaches 72.7/91.0/93.8% with 30k extra agent-generated trajectories, compared with oracle 75.4/95.3/96.2%. All table models have 300M parameters; daggers denote added 10k/30k trajectories.
Go to primary source ↓e-jointPDF p. 7, Table 3; p. 8, Section 5.4, Generalization across multiple environments; p. 14, Appendix A, RLBench evaluation
Joint training uses CALVIN and 20k RLBench trajectories; LDM and transformer share combined data, but each environment trains its own IDM. Table 3 reports five task rates for oracle, video-only and VideoWorld.
Go to primary source ↓e-horizonPDF p. 8, Table 4a–b; p. 9, Section 5.6, Horizon length
At 50M, Go H=1 gives 70.3/77.0 and H=5 gives 73.9/80.9 Action-Value/accuracy; CALVIN H=10 gives 50.3/71.1/69.7% success. The authors report Go convergence failure beyond five steps.
Go to primary source ↓e-interventionPDF p. 8, Table 4c; p. 9, Section 5.5, LDM generates causally interrelated codes
Random replacement of the first Go code lowers Action-Value/accuracy from 73.9/80.9 to 46.2/42.1; replacing code 3 gives 72.1/80.6. The authors attribute stronger early-code effects to causal dependence.
Go to primary source ↓e-codebookPDF p. 8, Table 4d; p. 9, Section 5.6, Latent codebook size
The 64,000-entry codebook gives the best listed performance; 262,144 entries degrade Go and CALVIN. The text attributes overly large vocabularies to convergence difficulty.
Go to primary source ↓e-data-qualityPDF p. 8, Table 4e and dagger footnote; pp. 9–10, Section 5.6, Data quality
Table 4e lists 6.8M human and 3.2M KataGo records, totaling 10M. Original versus relabeled human data yield 34.6/34.8 versus 71.5/75.7% Action-Value/accuracy. Its human count differs from Section 4.1.
Go to primary source ↓e-targetsPDF p. 8, Table 5, all rows; p. 10, Section 5.6, Latent code prediction only
At 50M, code-only gives Go 73.0/78.6 and CALVIN 47.2/70.0/65.1, versus joint code/video 73.9/80.9 and 50.3/71.1/69.7. The source removes frame cross-entropy and retrains a code-only IDM.
Go to primary source ↓e-probesPDF p. 5, Figure 4 and caption; p. 6, Figure 5 and caption; p. 9, Figure 6, caption and Section 5.5
Training-code UMAPs and predicted-code visualizations probe temporal structure. Figure 6 shows nine predicted codes colored by task, green training predictions and yellow actual simulator control; the text explicitly sets H=9 for this figure.
Go to primary source ↓e-algorithmPDF p. 15, Algorithm 1, lines 6, 9–13 and 26–33; pp. 4–5, Section 3.2
Printed pseudocode returns f[:0] and uses range(H) with prefix f[:(1+h)], whereas prose describes query h accessing the current frame through future step h. Quantization precedes reconstruction and MSE training.
Go to primary source ↓e-overlapPDF p. 14, Figure A.1; p. 15, Appendix B, Board state count and Board state repetition rate
The appendix examines 400 games against KataGo-9d and reports training-board overlap reaching zero by move 30. The training distribution is concentrated in the first 100 moves.
Go to primary source ↓e-conclusionPDF p. 10, Section 5.7, Figures 7–9 and Section 6
The paper illustrates selected Go tactics and simulator task executions, and identifies visual generation quality and generalization as challenges for real-world use.
Go to primary source ↓8.5 Primary sources
VideoWorld: Exploring Knowledge Learning from Unlabeled Videos ↗
PDF · 10,217 extracted words
Source fingerprint
74aadabe364d6b57eb3eb8b589d13465d8bf25ccebf9589977f85d5bda3b5604