World Tokens: Enhancing Embodied Policies with Training-Time World Modeling
1. Paper overview
In one sentence: Training a video denoiser through the policy’s mandatory token interface improves manipulation while moving video-model cost out of deployment and into training. e-probleme-architecturee-deploye-ablationse-simplere-robote-latency
| At a glance | What to know |
|---|---|
| Research problem | Source description Image–text pretraining offers limited supervision for contact, occlusion, and displacement. Keeping a video model in the control loop adds latency; a parallel predictive loss can leave the policy free to bypass the supervised representation. World Tokens addresses both concerns through exclusive action conditioning on shared tokens. e-probleme-architecture |
| Core mechanism | Source description A query-based World Adapter provides shared conditioning to a future-video denoiser and a separate action expert; the video branch is removable after training. e-architecturee-deploy |
| A key reported result | R1 Pro fruit-to-basket manipulation: 76.0 pooled; 70.8 lemon, 83.3 strawberry, 75.0 mango, 75.0 banana. Closed-loop success (%). Physical right-arm control; lemon, strawberry, mango, banana; 24 trials each, 96 per policy, randomized layouts and policy order. Matched Qwen-GR00T: 59.4 pooled; 54.2/58.3/75.0/50.0. Pretrained pi_0.5: 100.0 pooled. A 16.6-percentage-point increase over the matched system, with no mango gain. Pretrained pi_0.5 is contextual, not a matched training comparison. e-robote-scale |
| Reading caution | Author claim Authors acknowledge considerable video-model training cost and a hand-designed Canny anchor. Broad cross-embodiment generality remains future work. e-limit |
Core contributions
- Source description
A query-based World Adapter provides shared conditioning to a future-video denoiser and a separate action expert; the video branch is removable after training. e-architecturee-deploy
- Source description
A Canny anchor supplies scene layout while encouraging the tokens to carry task-relevant visual information. Ablations favor combining video supervision, exclusive routing, learned queries, and edge conditioning. e-anchore-ablations
Figure 2. World tokens are the shared interface; the upper video branch exists only during training. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Begin at the lower left: observation and instruction enter the VLM, whose hidden states condition the World Adapter. Learned queries become the world tokens that feed the action head and, through a projection described in Section 3.4, condition the upper denoiser. Trace the separate upper-left route from observation through edge detection and the VAE to the structural anchor. The blue downward “Dynamic guidance” arrow denotes the training influence of video supervision, as clarified by the gradient paths in Section 3.5; it does not mean predicted future tokens become an online action input. The caption and equations specify that deployment retains the lower path. e-architecturee-objectivee-deploy
What it supports. Both objectives shape the same representation, and the action expert cannot read around it through the full VLM sequence. Removing the upper branch therefore removes video computation while preserving trained VLM and adapter weights. The tokens encode current context with predictive supervision; they are not themselves a forecast trajectory.
Where the evidence stops. The diagram omits loss formulas and the token projection. Read its schematic guidance arrow alongside Equations (3)–(5) and the gradient-path text. It does not establish predefined token semantics, explicit planning, or causal dynamics identification.
2. Motivation
2.1 The problem and the proposed response
Image–text pretraining offers limited supervision for contact, occlusion, and displacement. Keeping a video model in the control loop adds latency; a parallel predictive loss can leave the policy free to bypass the supervised representation. World Tokens addresses both concerns through exclusive action conditioning on shared tokens. e-probleme-architecture
2.2 What this reading follows
A future-prediction loss can improve a robot policy only if useful information reaches the representation that actually drives actions. World Tokens makes that connection architectural: a learned adapter compresses image-and-language features into tokens consumed by both an action expert and a video denoiser. The video branch disappears after training. This reading follows that shared interface, then asks whether the experiments distinguish its effects from extra model capacity, easier appearance reconstruction, and different evaluation budgets. The evidence spans simulated LIBERO control, real-data-to-simulation SIMPLER evaluation, physical fruit placement, and policy-only latency; those outcomes support different conclusions and should remain separate. e-probleme-architecturee-deploye-ablationse-simplere-robote-latency
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | WAMs |
| Architecture | Dual-system |
| Prediction paradigm | Other mechanisms |
| Quadrant | Outside quadrants |
3.1 Evidence-based assessment
Supports the recorded classification
Distinct VLM and action expert support Dual-system. Other mechanisms/Outside quadrants fits training-only video supervision: current-context tokens condition actions without deployed joint future/action prediction or inverse dynamics. WAM is the catalog’s broad umbrella. Latent prediction applies to video-VAE targets; no JEPA objective is used. The 2B label denotes the VLM backbone; the adapter adds approximately 0.5B. e-architecturee-objectivee-deploye-footprint
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Follow the two losses into one deployed representation
The essential object is the adapter output, not the generated video. The VLM first encodes the present observation and instruction; learned queries extract the fixed token set. Action flow matching trains those tokens to retain information useful for demonstrated commands. Video denoising trains the same tokens to explain demonstrated future latents, after projection into the denoiser’s conditioning space. The two gradients meet in both the adapter and VLM, while the action and video experts remain separate networks. The VAE is frozen so its latent targets remain stationary; the pretrained video denoiser is fine-tuned for robot data. At deployment, removing the denoiser does not undo those learned weights. Recomputing the tokens from each new observation provides feedback, and four flow steps generate the next action chunk without accessing future frames. e-architecturee-objectivee-deploye-setup
5.2 Interpret the bottleneck and edge anchor together
Reader interpretation: the adapter and edge anchor constrain two different shortcuts. Exclusive routing prevents the action expert from using the full VLM sequence instead of the video-supervised tokens. Appearance suppression prevents the video denoiser from relying as heavily on a full-color first frame. The intended division of labor leaves geometry in the anchor and task-relevant content in the shared tokens. Table 2 supports this account through controlled changes: restoring a VLM bypass or replacing the edge map with RGB lowers Long success. Yet the results do not directly measure which information each token contains. A stronger claim that the RGB condition causes appearance copying, or that the FFN creates gradient competition, remains an interpretation of the observed ordering. Independent training runs and targeted predictive controls would make that interpretation more discriminating. e-architecturee-anchore-ablations
5.3 Separate attention concentration, imagined motion, and robot success
Reader interpretation: the diagnostic and task results answer different questions. Figure 3 and Appendix D average final-block cross-attention over queries and heads, then examine an image patch distribution. Lower entropy shows concentration, and Figure A3 shows consistent separation across the displayed episode; dependent timesteps do not constitute independent demonstrations of generalization. Figure A2 instead shows a rollout from the video branch, which can illustrate learned scene evolution but is removed from deployed control. Physical execution evidence comes from Table 4, not from that synthetic sequence. The real-robot rollout panels are selected illustrations; the pooled trials establish the reported success rate. Together these observations are consistent with useful predictive supervision, while the component ablations provide the more direct test of architectural choices and none alone proves a general-purpose causal world model. e-attentione-video-rolloute-real-rolloutse-robote-ablationse-deploy
5.4 Training and inference
During training
Action flow matching regresses velocity from Gaussian noise toward demonstrations; video retains its native diffusion parameterization. Both losses update the VLM and adapter; video loss also updates its projection and denoiser. e-objectivee-deploy
The pretrained VLM and video denoiser are jointly fine-tuned. Adapter, projection, and DiT-B action expert start from scratch; the VAE stays frozen. World Tokens uses no embodied action pretraining. e-deploye-setupe-libero
During inference
Each invocation recomputes tokens from observations and language, then generates eight actions using four flow-integration steps. Future targets, edge processing, VAE, projection, and video denoiser are removed. Feedback comes through subsequent observations; no online imagined rollout or inverse-dynamics extraction is used. e-deploye-setup
5.5 Implementation flow
- Encode current context
Qwen3-VL-2B-Instruct encodes image patches and language into a variable-length sequence. World tokens summarize current context; they are neither predefined object slots nor predicted future states. e-architecturee-setup
- Resample through a mandatory interface
The adapter uses 256 learned queries and 12 blocks of cross-attention to VLM features, query self-attention, and feed-forward updates. Width is 2048 with eight heads. The action expert has no direct VLM bypass. e-architecturee-footprint
- Supervise with future video
A projection maps the tokens into Cosmos Predict2.5-2B cross-attention context. A frozen VAE encodes demonstrated futures and the edge anchor. The denoiser receives noisy future latents and both conditions; its output is not an action-policy input. e-architecturee-objectivee-setup
6. Experiments & results
World Tokens makes a video-supervised representation the mandatory interface between a vision-language model and an action expert. Its current-context tokens support future-video denoising during training; deployment retains only the VLM, adapter, and action expert. Controlled LIBERO and physical R1 Pro results support improved control, with modest adapter overhead but substantial training cost.
6.1 Read the original evidence
Table 1. World Tokens is competitive on LIBERO, with 98.2% average success and 97.0% on Long. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the four success columns before the latency column. Spatial, Object, Goal, and Long each contain ten tasks, and Avg. is their unweighted mean. Emb. PT records embodied action pretraining; a cross does not mean the VLM and video denoiser lack pretraining. World Tokens appears in the VLA block because of its deployed computation. Its 2B entry is the VLM backbone label, not a count of all deployed parameters: Appendix A describes an additional approximately 0.5B adapter. Compare the World Tokens row against both blocks, then use Appendix B for timing conditions rather than treating the latency column as a uniform benchmark. e-liberoe-setupe-deploye-footprinte-timing-conflict
What it supports. The row reports 99.6%, 98.8%, 97.4%, and 97.0% across the suites, averaging 98.2%. Cosmos Policy and DiT4DiT have higher averages, 98.5% and 98.6%. The useful conclusion is strong simulated control without embodied action pretraining or a deployed video branch, rather than universal leaderboard dominance.
Where the evidence stops. These are published-system comparisons with different pretraining and backbones. The main caption calls most latencies quoted, while Appendix B reports additional remeasurements; some entries also differ in hardware or action horizon. Small success gaps lack reported uncertainty.
Table 3. Real-robot training transfers to simulated evaluation, with uneven strengths across manipulation tasks. Original paper, p. 11 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the WidowX and Google blocks as separate evaluations of a policy trained on the BridgeV2/Fractal mixture. VM means visual matching in simulation, not physical deployment. WidowX columns cover spoon placement, carrot placement, block stacking, and eggplant placement; Google columns cover picking, moving near, drawers, and apple placement. The retained note matters: a dash is an unavailable result, and some Google averages omit Place. Use task columns when coverage differs. The Qwen-GR00T row here is a published 4B comparison, not the matched action-only model used to isolate the architecture elsewhere in the paper. e-simplere-setup
What it supports. World Tokens reports the highest listed averages, 71.5% on WidowX and 82.1% on Google Robot. Carrot placement reaches 85.0%, but WidowX stacking reaches only 32.0%, below GR00T-N1.5’s 57.0%. On Google, 65.0% Move Near trails StarVLA’s 82.6%; the method does not lead every task.
Where the evidence stops. Do not merge Google averages with different task coverage or interpret simulated VM success as a real-robot success rate. Published baselines differ in model size and training history, so this table establishes system-level competitiveness rather than a controlled component effect.
Table 4. The physical-robot comparison improves pooled success over the matched action-only baseline. Original paper, p. 12 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read World Tokens against Qwen-GR00T first. The setup uses the R1 Pro’s right arm for four fruit-to-basket tasks, with randomized object–basket positions and policy evaluation order. Each object receives 24 trials, so Avg. pools 96 trials per policy. The retained note explains why bold excludes the gray pretrained row: the relevant controlled comparison concerns systems without embodied action pretraining. Appendix A says the matched baseline shares training steps and batch size. The two pi_0.5 rows contextualize pretraining, but neither isolates the World Adapter or the video objective. e-robote-scalee-ablations
What it supports. World Tokens achieves 76.0% pooled success versus 59.4% for Qwen-GR00T, a reported-rate difference of 16.6 percentage points. Strawberry and banana improve by 25.0 points each; mango remains 75.0% for both. The pretrained pi_0.5 row reaches 100.0%, limiting any claim of absolute superiority.
Where the evidence stops. Four objects and 96 trials per policy define a narrow physical evaluation, and the table supplies no confidence intervals. The matched comparison changes both adapter and video training; LIBERO ablations, not these pooled robot outcomes alone, distinguish their contributions.
Figure A1. Latency comparisons hold integration count and compilation setting fixed within each pair. Original paper, p. 16 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the chart as two comparisons, not one four-way ranking. The left pair uses four integration steps and compiled execution; the right pair uses ten steps without compilation. Both use a single RTX 5090 D, two 224×224 camera views, batch size one, and an eight-action chunk. P50 is the steady-state median measured with CUDA events after warmup. It covers VLM encoding, token computation, and action integration, while excluding capture, communication, robot execution, and server overhead. The ten-step World Tokens run is a timing control for Fast-WAM; all reported task results use the four-step default. e-latencye-timing-conflicte-deploy
What it supports. At the deployed four-step setting, World Tokens takes 61.85 ms versus 56.32 ms for pi_0.5, within the paper’s 1.1× comparison. Under the ten-step uncompiled control, 119.72 ms versus 182 ms gives a Fast-WAM/World Tokens latency ratio of approximately 1.5 under that recipe.
Where the evidence stops. This measures policy computation, not end-to-end control delay or tail latency. Appendix B’s remeasurement account conflicts with the main table’s quoted-latency description; preserve that discrepancy. Comparing the left World Tokens bar directly with the right Fast-WAM bar confounds integration budget and compilation.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| LIBERO manipulation across four suites One policy trained on pooled Spatial/Object/Goal/Long demonstrations; 40 tasks, 50 evaluation episodes per task, 2,000 total. | 98.2 average; Spatial 99.6, Object 98.8, Goal 97.4, Long 97.0. Success rate (%) | Table 1 averages: StarVLA 97.8, World2Act 98.1, Cosmos Policy 98.5, DiT4DiT 98.6. Competitive simulated control without embodied action pretraining; not the highest overall average. Published methods differ in backbone and pretraining. e-setupe-libero |
| LIBERO Long component ablations Table 2 variants under the same LIBERO evaluation protocol. | Full method 97.0 Long / 98.2 average. Success rate (%) | Long: Qwen-GR00T 92.8; no world modeling 95.0; VLM bypass 94.1; parameter-matched FFN adapter 93.4; RGB anchor 91.5. Adapter-only improvement and additional video-supervision benefit are separable. Results favor exclusive query routing; training-seed uncertainty is unreported. e-ablations |
| SIMPLER visual-matching manipulation Train jointly on BridgeV2 and Fractal real-robot data; evaluate in simulation. WidowX uses 24 prescribed object-pose configurations per task, averaged over configurations and seeds. | WidowX 71.5 average; Google Robot 82.1 average. WidowX block stacking: 32.0. Success rate (%) | Published 4B Qwen-GR00T: 65.3 WidowX average; GR00T-N1.5: 57.0 stacking. Several Google averages omit Place. Highest listed averages, with task-coverage caveats. This published 4B baseline differs from the matched action-only baseline. Precise stacking remains weak. e-simpler |
| R1 Pro fruit-to-basket manipulation Physical right-arm control; lemon, strawberry, mango, banana; 24 trials each, 96 per policy, randomized layouts and policy order. | 76.0 pooled; 70.8 lemon, 83.3 strawberry, 75.0 mango, 75.0 banana. Closed-loop success (%) | Matched Qwen-GR00T: 59.4 pooled; 54.2/58.3/75.0/50.0. Pretrained pi_0.5: 100.0 pooled. A 16.6-percentage-point increase over the matched system, with no mango gain. Pretrained pi_0.5 is contextual, not a matched training comparison. e-robote-scale |
| Policy inference latency per eight-action chunk Batch one, two 224×224 views, 24 GB RTX 5090 D, CUDA-event steady-state median; excludes sensing, transport, execution, and server overhead. | 61.85 at four compiled flow steps; 119.72 at ten uncompiled steps for a timing-only control. Median milliseconds per chunk (lower is better) | pi_0.5: 56.32 at four compiled steps; Fast-WAM: 182 at ten uncompiled steps. VLA-level policy inference latency. Ten-step timing does not establish ten-step task success; it avoids overstating the architectural advantage over Fast-WAM. e-latencye-timing-conflict |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 2. The strongest separation appears on LIBERO Long when routing or the anchor is changed. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start from the bottom row, where all four design factors are enabled. Move upward to RGB anchor, FFN adapter, and VLM bypass: each changes one operative factor while retaining video supervision. The FFN is parameter matched. In the no-world-modeling row, the edge-anchor cross means that the anchor is inapplicable because the video branch is absent; it is not an independent second intervention. Qwen-GR00T is a broader system baseline without the adapter or video branch. Read Long alongside Avg.: the other suites are close to saturation and do not improve uniformly. The table supports component comparisons, not a single attribution for every baseline gap. e-ablationse-setupe-architecture
What it supports. Long success falls from 97.0% to 95.0% without video supervision, 94.1% with a VLM bypass, 93.4% with the FFN, and 91.5% with RGB anchoring. Two variants retaining video supervision underperform the no-video variant, supporting the importance of how predictive training reaches the action interface.
Where the evidence stops. The table gives point estimates without training-seed dispersion or confidence intervals. Routing and anchor ablations support the proposed mechanism in this setup, but do not directly measure gradient conflict or prove the authors’ explanation for the FFN’s behavior.
7. Analysis & limitations
7.1 What the evidence leaves open
Authors acknowledge considerable video-model training cost and a hand-designed Canny anchor. Broad cross-embodiment generality remains future work. e-limit
Success tables lack confidence intervals and training-seed dispersion. Attention entropy is descriptive: main-view means are 3.29 versus 5.22 bits over one episode per Long task, and adjacent steps are dependent. Sharper attention does not prove causal dynamics understanding. e-ablationse-robote-attention
Table 1 and Section 4.1 call non-pi_0.5 baseline latencies quoted; Appendix B describes re-timing Fast-WAM and DiT4DiT, with DiT4DiT on H100. This discrepancy remains unresolved. Cross-hardware entries and different horizons cannot establish controlled speedups. e-timing-conflict
R1 setup specifies cropped head-camera input, but rollout captions show head/wrist views; controller use of the wrist view is unclear. Figure A4 repeats lemon despite calling its tasks absent from the main figure. Selected examples cannot replace aggregate results. e-robote-real-rollouts
7.2 Questions for discussion
- Would the routing advantage persist across independent training seeds and equalized training FLOPs?
- Does temporally aligned future supervision help more than static reconstruction with the same video branch?
8. Reproducibility audit
8.1 Requirements and known gaps
Use AdamW: beta=(0.9,0.95), epsilon/weight decay 1e-8, cosine decay, linear warmup. Learning rates: VLM 1e-5, denoiser 2e-5, new modules 1e-4. H100 training: LIBERO 70K steps/8 GPUs/batch 128; SIMPLER 140K/16/256; R1 30K/8/128. Matched baselines share steps and batch, not necessarily FLOPs. e-optimizatione-scale
Reproduce grayscale Canny thresholds 60/140, Gaussian blur sigma=2.5, and RGB replication. Explicit video resolution, future-frame sampling interval, warmup duration, action normalization, R1 demonstration count, software versions, and total training wall time are omitted and require clarification. e-anchore-setupe-optimizatione-scale
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Test whether video supervision specifically benefits exclusive routing
Reader-proposed, not performed: train a two-by-two LIBERO comparison crossing video-loss weight 0 versus 0.5 with exclusive token conditioning versus a restored VLM bypass. Keep the 256-query adapter, backbone, initialization distribution, data, batch size, and training steps fixed; use five independent training seeds and common evaluation seeds. Report Long success with uncertainty across training runs, and record training FLOPs separately because a zero video loss changes compute. The decisive quantity is whether enabling video supervision improves exclusive routing more than bypass routing. If gains are equally large with bypass, or the interaction disappears across seeds, the paper’s strong routing-specific interpretation weakens. e-objectivee-architecturee-ablationse-scale
Check 2: Distinguish future dynamics from appearance reconstruction
Reader-proposed, not performed: cross correct eight-frame future targets versus eight repetitions of the current primary-view image with Canny versus RGB anchoring. Keep Cosmos initialization, trainable modules, loss weighting, action demonstrations, and optimization budget fixed. Use the specified Canny thresholds 60/140 and Gaussian sigma=2.5, then evaluate action-only deployment on held-out LIBERO evaluation seeds across multiple training runs. Report Long success and held-out video loss separately. A dynamics-specific account predicts an action benefit from temporally correct futures beyond static reconstruction, especially with the edge anchor. If repeated-current targets match that benefit, an appearance or auxiliary-regularization explanation remains viable even if video loss improves. e-architecturee-objectivee-anchore-ablationse-optimizatione-scalee-setup
8.3 Reading coverage
Visual audit: Visually inspected the title/author/version block, Figures 1–4 and A1–A4, Tables 1–4, method equations and training/deployment text, experimental protocols, and Appendix A–E evidence. Six final original crops were separately viewed for readable labels, complete bounds, and retained table notes. Figure 2’s conditioning arrows and schematic downward guidance were checked against its caption, Equations (1)–(5), and Section 3.5; guidance is training influence, not an online video-to-action input. Pages 15–17 support all retained hardware, optimization, footprint, and timing claims; pages 17–20 support the diagnostic and rollout boundaries. Reference-only pages 13–14 were read in full text but not rendered. Separate supplements and external code remain outside the supplied reading scope.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 15, 16, 17, 18, 19, 20. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract
- 1 Introduction
- 2 Related Work
- 3 Method (3.1–3.5)
- 4 Experiments (4.1–4.5)
- 5 Conclusion
- References
- Appendix A Implementation Details
- Appendix B Latency Measurement
- Appendix C World-Model Rollout
- Appendix D Attention Analysis
- Appendix E Additional Real-World Rollouts
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Identity: title and all six authors match the catalog after name-order normalization. The title page identifies arXiv:2608.09730v1 [cs.CV], 10 August 2026. No other revision or edition was supplied or compared.
- Acquisition omission: Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout. This limitation was addressed by inspecting the supplied PDF pages and all final crops.
- Separate supplemental material availability has not been fully verified.
- All five supplied text chunks were individually read, covering all 20 PDF pages. Visual inspection covered pages 1–12 and 15–20; the reference-only pages 13–14 were read as text.
- Code and external resources were not inspected; no training, inference, or robot experiments were reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e-identityPDF p. 1, title/author block and arXiv stamp
Exact catalog title; Qu Tang, Benhui Zhuang, Bo Yuan, Xue Yu, Longteng Guo, Junlan Feng. Bo Yuan is corresponding author. Affiliations: JIUTIAN Research and Zhongguancun Academy. Stamp: arXiv:2608.09730v1 [cs.CV], 10 Aug 2026.
Go to primary source ↓e-problemPDF pp. 1–2, Abstract and Section 1, paragraphs around Figure 1
Motivation concerns temporal supervision, online video cost, and a policy bypass around auxiliary predictive features.
Go to primary source ↓e-architecturePDF p. 4, Figure 2/caption and final paragraph; pp. 5–6, Sections 3.1–3.4, Eqs. (1)–(3)
Current images/language produce VLM features and query-resampled world tokens. Tokens exclusively condition actions and separately condition a training-only denoiser through projection. VAE is frozen; anchor is an encoded Canny image.
Go to primary source ↓e-objectivePDF p. 6, Section 3.5, Eqs. (4)–(5)
Joint objective is action denoising plus 0.5 times video denoising. Action interpolation runs from Gaussian noise to target with velocity y minus noise; video retains native diffusion.
Go to primary source ↓e-deployPDF p. 7, Section 3.5, Gradient paths, Initialization, Deployment
Both losses update VLM/adapter. VLM/denoiser are fine-tuned, other learned modules start from scratch, VAE stays frozen. Deployment removes video components and uses four flow-integration steps.
Go to primary source ↓e-setupPDF p. 8, Section 4.1, Implementation and Benchmarks
Qwen3-VL-2B-Instruct, 256 queries, DiT-B, Cosmos Predict2.5-2B, eight-action/eight-frame horizons. LIBERO pools 40 tasks from four suites, each with 500 demonstrations, and evaluates 50 episodes per task.
Go to primary source ↓e-liberoPDF p. 7, Table 1, World Tokens and comparator rows, success and Emb. PT columns
World Tokens: 99.6/98.8/97.4/97.0, mean 98.2, no embodied action pretraining. Averages: StarVLA 97.8, World2Act 98.1, Cosmos Policy 98.5, DiT4DiT 98.6; backbone/pretraining differ.
Go to primary source ↓e-ablationsPDF p. 9, Table 2/caption; p. 10, Section 4.3, Mechanistic ablations and Structural-anchor comparison
Long: 92.8 baseline, 95.0 no world modeling, 94.1 bypass, 93.4 FFN, 91.5 RGB, 97.0 full. FFN is parameter matched; edge anchor is inapplicable without video supervision. No uncertainty columns are supplied.
Go to primary source ↓e-attentionPDF pp. 9–10, Figure 3 and Section 4.3 Attention analysis; pp. 17–19, Appendix D and Figure A3
Final-block attention averages 256 queries/eight heads over 8×8 spatial grids. Main-view entropy: 3.29 versus 5.22 bits over one episode per Long task. Displayed episode separates at all 32 control steps; authors caution that steps are dependent and statistics descriptive.
Go to primary source ↓e-simplerPDF p. 8, Section 4.1 Benchmarks; pp. 10–11, Section 4.4; p. 11, Table 3/caption
BridgeV2/Fractal training precedes simulated VM evaluation. WidowX: 24 poses/task; World Tokens 74.0/85.0/32.0/95.0, mean 71.5. Google: 91.0/65.0/78.7/93.5, mean 82.1. Comparator averages may omit Place; dashes mean unavailable. Qwen-GR00T is a published 4B row here.
Go to primary source ↓e-robotPDF p. 8, Section 4.1 R1 Pro setup/Baselines; pp. 11–12, Section 4.5 and Table 4/caption
R1 right arm uses eight-dimensional joint/gripper actions and cropped head view. Four objects receive 24 trials each. World Tokens mean 76.0 versus matched Qwen-GR00T 59.4; per-object values and contextual pi_0.5 rows lack confidence intervals.
Go to primary source ↓e-limitPDF p. 12, Section 5 Conclusion, final paragraph
Authors identify considerable training-time cost and hand-designed anchor; joint training across larger robot datasets, embodiments, and tasks is future work.
Go to primary source ↓e-optimizationPDF p. 15, Appendix A, Optimization
AdamW beta=(0.9,0.95), epsilon=1e-8, weight decay=1e-8; cosine schedule and linear warmup. Rates: 1e-5 VLM, 2e-5 denoiser, 1e-4 new modules. Warmup duration/software versions are unspecified.
Go to primary source ↓e-anchorPDF p. 15, Appendix A, Structural anchor; p. 6, Section 3.4
At video-branch resolution: grayscale, Canny thresholds 60/140, Gaussian sigma=2.5, RGB replication, frozen VAE encoding. Explicit resolution and future-frame sampling interval are not given.
Go to primary source ↓e-scalePDF p. 15, Appendix A, Training scale; p. 8, Section 4.1
H100, 16 samples/device, no accumulation: LIBERO 70K steps/batch 128/eight GPUs; SIMPLER 140K/256/sixteen; R1 30K/128/eight. Matched baseline steps/batches identical. R1 demonstration count and training wall time are absent.
Go to primary source ↓e-footprintPDF p. 15, Appendix A, Deployment footprint; pp. 16–17, Appendix B, World Adapter overhead
Adapter: K=256, L=12, d=2048, eight heads, width-sized FFN, approximately 0.5B parameters. Appendix B attributes roughly 10 ms of 61.85 ms policy latency to it.
Go to primary source ↓e-latencyPDF pp. 15–17, Appendix B; p. 16, Figure A1
Batch-one/two 224×224 views/eight-action horizon, CUDA-event steady-state median on 24 GB RTX 5090 D. Compiled four-step World Tokens/pi_0.5: 61.85/56.32 ms; uncompiled ten-step World Tokens/Fast-WAM: 119.72/182 ms. Ten steps is timing-only. Sensing, communication, execution and server costs are excluded.
Go to primary source ↓e-timing-conflictPDF p. 7, Table 1 caption; p. 8, Section 4.1 Baselines and protocol; pp. 16–17, Appendix B Measurements/Comparability
Main text calls latencies other than World Tokens/pi_0.5 quoted. Appendix describes running baseline code/weights for pi_0.5, Fast-WAM, DiT4DiT, the last on H100. Other entries differ in hardware/horizons; OpenVLA’s quoted value is for one action despite per-chunk table framing.
Go to primary source ↓e-video-rolloutPDF pp. 17–18, Appendix C and Figure A2/caption
Sixteen displayed frames sample an autoregressive video-branch rollout from one observation. It is qualitative video evidence from a discarded branch, not an executed robot trajectory.
Go to primary source ↓e-real-rolloutsPDF p. 11, Figure 4/caption; pp. 19–20, Appendix E and Figure A4/caption
Figure 4 shows lemon/strawberry; A4 lemon/banana/mango with matched initial arrangements. Captions describe head/wrist initial views. A4 calls its three tasks absent from the main text although lemon repeats. Table 4 supplies aggregate outcomes.
Go to primary source ↓8.5 Primary sources
World Tokens: Enhancing Embodied Policies with Training-Time World Modeling ↗
PDF · 7,482 extracted words
Source fingerprint
04650e2cea439ca97d52654aee35a11d432ee7008a5955a033eb79364600f399