Awesome-WAMs
GitHub
PAPER REPORTENAll readings ↗

SG-WAM: Self-Guided World Modeling in Geometry-Aware Policy Space

English reading report: Method, equations, original figures, experiments and reproducibility.

Authors: Ruiteng Zhao; Zhengshen Zhang; Yue Su; Wenshuo Wang; Jiahui Li; Zhiyuan Yang; Francis E.H. Tay; Marcelo H. Ang Jr.; Haiyue Zhu

Affiliations: Advanced Robotics Centre, National University of Singapore; MMLab, The University of Hongkong; Nanyang Technological University; Singapore Institute of Manufacturing Technology, Agency for Science, Technology and Research (A*STAR)

Source: 2608.01397 ↗ · Project page ↗ · Catalog record

Reading: 98 / 558 · 6 original figures & tables · ~19 min ·

1. Paper overview

In one sentence: SG-WAM uses geometry-aligned policy tokens and action-conditioned future prediction to improve a compact manipulation policy, while keeping the predictive branches out of deployment. e03e04e05e07e12e15e18e20

At a glanceWhat to know
Research problem
Author claim

The authors argue that reconstructing future observations spends capacity on appearance, while externally defined latent targets can emphasize information different from that used by the action policy. Their goal is a predictive representation that remains action-relevant without discarding spatial relationships needed for manipulation. e02

Core mechanism
Source description

Online and EMA copies of the same policy construct current and future dynamics-token states. A separate action-conditioned predictor learns their transition, while the action expert consumes the shared policy context. e03e05e06

A key reported resultLIBERO-Plus zero-shot transfer: Overall 73.0; Camera 58.6; Robot 48.9; Language 81.4; Light 89.8; Background 86.1; Noise 80.7; Layout 74.2.

Success rate (%). The LIBERO-trained checkpoint is evaluated without adaptation under seven perturbation categories; Overall aggregates successful rollouts.

OpenVLA-OFT: 69.6 overall; VLA-JEPA: 62.9. The overall advantage over OpenVLA-OFT is 3.4 percentage points. Best listed overall score, not every column. Table 2 gives VLA-JEPA 74.6 on Layout, contradicting Section 4.2's best-layout claim. e10e12e13

Reading caution
Reader analysis

Tables report point estimates without confidence intervals or seed variation; simulation rollout counts and exact benchmark versions are omitted. Small physical trial counts limit precision. Attention maps are qualitative examples, not geometric-error measurements or causal proof of object-centric dynamics. e10e14e18e20e21

Core contributions

  • Source description

    Online and EMA copies of the same policy construct current and future dynamics-token states. A separate action-conditioned predictor learns their transition, while the action expert consumes the shared policy context. e03e05e06

  • Source description

    Frozen VGGT features supervise existing main-view image tokens. Geometry shapes the context of future prediction without becoming its target or adding teacher tokens to the policy sequence. e04

Figure 2. Predictive training shapes the context used by the action expert. Original paper, p. 3 ↗

Excerpt from the authors’ paper; cropped without altering the figure or table.

How to read it. Start at the lower inputs: main and wrist images, instruction L, and learnable dynamics tokens Q. The VLM's output sequence feeds the action expert on the right. Follow the upper pink branch separately: current dynamics states are projected and combined with intervening actions, while future inputs feed an EMA VLM and projector to create targets. The blue branch aligns main-view image states with frozen VGGT features. The double-headed loss markers show comparisons, not bidirectional gradient updates. Equations (7)–(8) and Appendix B specify that target gradients stop; the EMA snowflakes therefore mean exclusion from gradient optimization, not permanently unchanged weights. e03e04e05e06e07e08

What it supports. Future supervision and action imitation meet through shared online policy states. Geometry supervision shapes image tokens, while the predictor learns transitions of dynamics tokens. At deployment, the VLM, dynamics tokens and action expert remain; the geometry teacher, predictor and EMA target pathway are removed.

Where the evidence stops. The diagram abbreviates geometry projection and uses t+T for the future, whereas the equations use t+Δ. Appendix B specifies eight intervening steps. This figure does not describe inference-time planning or a measured rollout-speed improvement.

2. Motivation

2.1 The problem and the proposed response

Author claim

The authors argue that reconstructing future observations spends capacity on appearance, while externally defined latent targets can emphasize information different from that used by the action policy. Their goal is a predictive representation that remains action-relevant without discarding spatial relationships needed for manipulation. e02

2.2 What this reading follows

A robot policy can learn from future observations without generating a future image when it acts. SG-WAM makes this idea concrete by placing learnable dynamics tokens inside the same vision-language context that conditions action generation. During training, demonstrated actions help a predictor forecast those tokens' future states; an exponential moving average copy of the policy supplies the targets. A frozen geometry model separately grounds current image tokens. The important reading distinction is between these training signals and the deployed controller. The figures below trace that distinction, then test it against zero-shot transfer, physical execution and controlled ablations. e03e04e05e07e12e15e18e20

3. Research context

We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.

Catalog dimensionRecorded classification
Major categoryWAMs
ArchitectureDual-system
Prediction paradigmOther mechanisms
QuadrantOutside quadrants

3.1 Evidence-based assessment

Supports the recorded classification

Reader analysis

The recorded Dual-system assessment is consistent with a distinct VLM context encoder, auxiliary transition predictor and separate flow action expert. Their shared representation and joint training do not establish a single joint future/action generator. Other mechanisms and Outside quadrants fit action-conditioned forward prediction used as training supervision, with neither inverse-dynamics action extraction nor test-time future planning. Geometry supervision supports the spatial emphasis, but it is single-main-frame feature alignment rather than explicit 3D/4D reconstruction. e03e04e05e07e08

This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.

4. Problem formulation

4.1 Inputs and outputs

InputsOutputs
  • Current main-view and wrist-view RGB observations and a language instruction
  • Training additionally uses demonstrated action chunks and future observations
  • An eight-step continuous action chunk, with seven-dimensional actions
  • Training only: predicted future dynamics-token latents and projected geometry features

4.2 Equations and their role

z^t+ΔQ=Fψ(ztQ,etA),Lpred=1Nqd∥z^t+ΔQ−sg⁡(zˉt+ΔQ)∥F2\hat z^Q_{t+\Delta}=F_\psi(z^Q_t,e^A_t),\qquad \mathcal L_{\mathrm{pred}}=\frac{1}{N_qd}\left\|\hat z^Q_{t+\Delta}-\operatorname{sg}(\bar z^Q_{t+\Delta})\right\|_F^2
The predictor Fψ maps current projected dynamics states z and encoded intervening actions e to future states after Δ steps. The barred target comes from the EMA policy; sg stops its gradients. Nq counts dynamics tokens and d is latent width. Normalization averages squared error over slots and features, not a stated unit-vector normalization. e05
Atτ=τAt+(1−τ)ϵ,Lact=EAt,ϵ,τ[∥vω(Atτ,τ,Ht)−(At−ϵ)∥22]A_t^\tau=\tau A_t+(1-\tau)\epsilon,\qquad \mathcal L_{\mathrm{act}}=\mathbb E_{A_t,\epsilon,\tau}\left[\left\|v_\omega(A_t^\tau,\tau,H_t)-(A_t-\epsilon)\right\|_2^2\right]
At is the expert action chunk, ε is standard Gaussian noise, τ is uniform on [0,1], and Ht is the full policy context. The action expert vω learns the velocity from noise toward the demonstrated chunk. e07
L=Lact+λgeoLgeo+λpredLpred,λgeo=λpred=0.1\mathcal L=\mathcal L_{\mathrm{act}}+\lambda_{\mathrm{geo}}\mathcal L_{\mathrm{geo}}+\lambda_{\mathrm{pred}}\mathcal L_{\mathrm{pred}},\qquad \lambda_{\mathrm{geo}}=\lambda_{\mathrm{pred}}=0.1
The geometry term averages one minus cosine similarity between projected main-view tokens and frozen teacher features. The two weights control geometry alignment and future-state prediction relative to action learning. e04e08

5. Method in detail

5.1 Construct a future target inside the policy's representation family

Source description

Begin with the policy context, not the predicted future. Image and language embeddings are followed by continuous dynamics embeddings, and the VLM contextualizes the whole sequence. The resulting dynamics states have two roles: they remain available to the action expert, and a projector maps them into the latent prediction space. For the target, the same architecture processes a later observation with the same instruction, using an EMA copy of the backbone, projector and dynamics embeddings. The online predictor then learns to match that detached future representation given the demonstrated intervening actions. The EMA update stabilizes the target as the online representation changes; it does not make the target a permanently frozen external feature space. Training optimizes the online pathway while the target pathway follows through parameter averaging. e03e05e06

Figure 5. Current state slots query the ordered actions that connect two observations. Original paper, p. 11 ↗

Excerpt from the authors’ paper; cropped without altering the figure or table.

How to read it. Follow the top-left path from contextualized dynamics tokens through the projector to the current latent states. Self-attention lets those slots exchange information before they enter cross-attention as queries. The lower action branch supplies keys and values, as marked by K and V. Here the attention label Q means queries; the paper also uses Q for its learnable dynamics embeddings, so the two uses should not be conflated. Appendix B states that eight seven-dimensional actions are embedded individually with learned temporal positions. The sequence is retained rather than pooled, allowing state slots to attend selectively to different action steps. e05e06

What it supports. The predictor is a forward transition model: current state plus intervening actions produces a future latent estimate. It does not infer the actions from two states. Its output is compared with an EMA policy target, making the relationship to the downstream policy explicit at the representation level.

Where the evidence stops. The colored slots are schematic; their drawn count is not the configured eight-token count. The source leaves latent width and predictor depth/head counts unspecified. Attention connectivity alone does not prove that the model uses action order effectively.

5.2 Separate geometric grounding from the learned transition

Source description

Geometry enters through a different target and a different set of tokens. A frozen VGGT-1B receives only the current main-view frame. Its final patch features are pooled onto an 8×8 grid and aligned by cosine distance with projected main-view policy image states. No VGGT tokens are appended to the VLM. The dynamics tokens then form their representations in that geometrically supervised visual context. In parallel, the predictor learns the action-conditioned change between current and future dynamics states, while flow matching trains the action expert on complete policy context. All three losses shape the online model in one stage. At deployment, auxiliary branches disappear, but the learned dynamics tokens remain. Thus the deployed policy retains the representations learned under geometric and predictive supervision without computing either auxiliary target. e03e04e05e07e08

5.3 Read the evidence as policy learning, then ask what mechanism is isolated

Reader analysis

Reader analysis: organize the experiments into three different questions. The published benchmark tables ask whether the trained policy performs competitively; their heterogeneous model sizes and imported baselines limit causal attribution. The physical table asks whether gains appear in executed manipulation, but its small sample and restricted OOD conditions constrain generalization. The ablations ask which ingredients help within SG-WAM. Removing either auxiliary objective lowers success, and replacing actions with zeros while preserving the branch provides more specific evidence that action information matters. None of these observations alone measures physical fidelity of the predicted latent states or proves that EMA policy targets outperform every external representation. The strongest supported conclusion is that this particular combination improves action-policy learning in the supplied experiments, with its predictive computation confined to training. e10e14e15e18e20e05

5.4 Training and inference

During training

Source description

Joint single-stage optimization combines action flow matching, geometry alignment and latent MSE. Both auxiliary loss weights are 0.1. The EMA momentum is 0.999; its pathway runs in evaluation mode without gradients. VGGT stays frozen, while online modules are trained jointly. e05e08

Source description

Simulation training uses all four LIBERO suites for 40,000 steps, batch 96, cosine scheduling and 5,000 warmup steps. Peak learning rates are 1e-5 for the VLM, 2.5e-5 for training-only modules and 1e-4 for the action head. Real-world models train jointly on three tasks for 40 epochs with the same batch and learning rates. Absence of embodied pretraining does not mean absence of pretrained VLM or geometry weights. e09e16e04

During inference

Source description

Current images and instruction pass through the online VLM with dynamics tokens. Integrating the action expert's learned velocity field from noise yields an action chunk. VGGT, SGWP and the EMA pathway are removed. The source does not specify the numerical solver, integration-step count or how many actions execute before the next observation. e03e07e08

Reader analysis

Executed robot success therefore tests a policy shaped by predictive training. It does not demonstrate inference-time imagined rollouts, action search or inverse-dynamics recovery from a predicted video. e03e07

5.5 Implementation flow

  1. Build policy context

    Qwen3.5-0.8B processes image and language embeddings followed by eight continuous learnable dynamics tokens. Their valid attention positions and rotary indices extend the original sequence. The action expert receives all final-layer visual, language and dynamics states. e03e06e05

  2. Ground the current image tokens

    Frozen VGGT-1B receives only the main frame, resized to 518×518 with antialiased bicubic interpolation. Its final aggregator yields a 37×37 patch grid, pooled to 8×8. A token-wise projector aligns policy main-view states with these targets using cosine distance. e04

  3. Forecast a demonstrated transition

    Project dynamics states into a compact latent space. Self-attention exchanges information among slots; cross-attention uses those states as queries and eight temporally embedded action tokens as keys/values. An EMA VLM, projector and dynamics-token set process the future observation and unchanged instruction to supply detached targets. e05

6. Experiments & results

SG-WAM trains a compact manipulation policy to predict future states of its own dynamics tokens, conditioned on demonstrated actions. A slowly updated copy of the policy supplies future targets, while a frozen geometry teacher shapes current image tokens. Deployment retains the online VLM, dynamics tokens and flow-matching action expert; future prediction is training supervision. The strongest transfer result is 73.0% on LIBERO-Plus, subject to heterogeneous published baseline protocols.

6.1 Read the original evidence

Table 2. The strongest reported overall transfer score coexists with weaker individual conditions. Original paper, p. 5 ↗

Excerpt from the authors’ paper; cropped without altering the figure or table.

How to read it. Read the bottom row across the seven perturbations before checking Overall. Camera, Robot, Language, Light, Background, Noise and Layout are distinct test conditions; they are not stages of one episode. Bold marks the best reported score in a column and underlining marks second-best. The SG-WAM checkpoint is trained jointly on the four original LIBERO suites and transferred without fine-tuning. Appendix E defines Overall through successful rollouts, so do not reconstruct it as an unweighted mean of the seven displayed percentages. The Params column describes model size, not a matched training budget, and Section 4.1 says simulation baseline values come from their respective papers. e09e10e12e13e22

What it supports. SG-WAM reports 73.0% overall, exceeding OpenVLA-OFT's 69.6% by 3.4 percentage points. It leads the displayed Camera, Language, Light and Noise columns. Its Robot, Background and Layout scores are below the strongest corresponding baselines, showing that the overall advantage is not uniform.

Where the evidence stops. Section 4.2 claims best Layout performance, but the table gives VLA-JEPA 74.6 versus SG-WAM 74.2. Also, Robot is described as embodiment shift in prose but Robot Initial States in Figure 22. Neither inconsistency is resolved by the supplied source.

Table 3. Physical execution improves, with narrow task and perturbation coverage. Original paper, p. 7 ↗

Excerpt from the authors’ paper; cropped without altering the figure or table.

How to read it. Read the grouped headers carefully. Pick and Place and Towel Folding each have an in-distribution column and three OOD columns, while Toolbox Organization has only an ID column. The platform is a UR5e with main and wrist RGB cameras. Each task contributes 100 demonstrations; the methods are evaluated for 20 trials on each of the first two tasks and 10 on Toolbox Organization. Appendix F says the three tasks are trained jointly and the same final real-world checkpoint serves ID and OOD evaluation. Novel Object means an unseen towel or changed cube colors with updated instructions, rather than unrestricted category transfer. e14e15e16

What it supports. SG-WAM leads every displayed cell: ID success is 75% for Pick and Place, 45% for Towel Folding and 50% for Toolbox Organization. Nevertheless, its OOD performance falls to 40% on novel-color cubes and 25% on an unseen towel, so relative gains do not imply consistently reliable execution.

Where the evidence stops. No Toolbox Organization OOD score is supplied. Initial scenes are reproduced only as closely as possible across methods, and the small trial counts come without confidence intervals. These results do not establish direct zero-shot transfer from the simulation checkpoint.

6.2 Results and evaluation conditions

Task & protocolReported resultComparison & interpretation
LIBERO four-suite manipulation

One policy trained jointly on Spatial, Object, Goal and Long; corresponding standard benchmark evaluation.

Spatial 99.4; Object 99.8; Goal 98.6; Long 96.2; average 98.5.

Success rate (%)

Spatial Forcing also averages 98.5 with 7B parameters and embodied pretraining; SG-WAM reports 0.9B without it.

Ties the strongest listed average. Baseline values are imported from their papers, so this is not a matched-data or matched-compute experiment. e10e11

LIBERO-Plus zero-shot transfer

The LIBERO-trained checkpoint is evaluated without adaptation under seven perturbation categories; Overall aggregates successful rollouts.

Overall 73.0; Camera 58.6; Robot 48.9; Language 81.4; Light 89.8; Background 86.1; Noise 80.7; Layout 74.2.

Success rate (%)

OpenVLA-OFT: 69.6 overall; VLA-JEPA: 62.9. The overall advantage over OpenVLA-OFT is 3.4 percentage points.

Best listed overall score, not every column. Table 2 gives VLA-JEPA 74.6 on Layout, contradicting Section 4.2's best-layout claim. e10e12e13

UR5e Pick and Place

100 demonstrations; 20 evaluation trials per method; barrier-obstructed cube transfer from drawer to bowl. OOD uses the same real-world checkpoint.

ID 75; Background 55; Light Change 60; Novel Object 40.

Task success rate (%)

VLA-JEPA: 35/20/25/20; VPP: 30/15/10/10, in the same condition order.

Physical execution improves in each tested condition; novel-object testing here changes cube color and instruction, not object category. e14e15e16

UR5e Towel Folding and Toolbox Organization

100 demonstrations per task; 20 towel trials and 10 toolbox trials per method. Towel requires two folds; toolbox requires a screwdriver, two gears and closure.

Towel: ID 45, Background 25, Light Change 35, Novel Object 25. Toolbox: ID 50.

Task success rate (%)

Towel VPP: 35/15/15/10; VLA-JEPA: 20/10/10/15. Toolbox ID: VPP 30, VLA-JEPA 20.

No toolbox OOD result is tabulated. Table 6 reports SG-WAM towel stages at 75/45 and toolbox stages at 80/60/50/50; these are not stated as conditional per-stage probabilities. e14e15e16e17

LIBERO objective ablation

Four geometry/world-modeling combinations evaluated on all four suites.

Neither 95.3; geometry only 96.6; world modeling only 97.6; both 98.5.

Average success rate (%)

Removing world modeling costs 1.9 percentage points; removing geometry costs 0.9. Long drops from 96.2 to 92.2 without world modeling.

Both objectives help the final policy; this does not directly measure latent prediction fidelity. e18

LIBERO dynamics-token count

Token count varied with other settings held fixed.

1 token: 96.1; 4: 97.5; 8: 98.5; 16: 97.2.

Average success rate (%)

Long rises from 90.2 with one token to 96.2 with eight, then falls to 92.4 with sixteen.

Supports the selected eight-token configuration, without establishing universal capacity scaling. e19

LIBERO intervening-action ablation

Replace demonstrated actions with zeros before the same encoder; preserve sequence length, temporal embeddings and other training settings.

Full model: average 98.5, Long 96.2. Null actions: average 97.6, Long 94.6.

Success rate (%)

Actual actions improve average by 0.9 and Long by 1.6 percentage points.

Isolates the utility of action information more directly than deleting the predictor; uncertainty across runs is not reported. e20

6.3 Ablations and diagnostic examples

Read component removals and qualitative examples within their stated evaluation conditions.

Table 4. Both training objectives contribute to policy success. Original paper, p. 7 ↗

Excerpt from the authors’ paper; cropped without altering the figure or table.

How to read it. Use Geo. and WM. as the two switches. Geo. means the frozen-teacher alignment of main-view image states; WM. means self-guided latent world modeling. The top row removes both, the middle rows retain one objective each, and the bottom row is the complete method. Read Avg. to assess overall policy performance, then Long to inspect the suite most affected by removing world modeling. These cells are task success percentages, not latent MSE values. In particular, turning off an auxiliary objective should not be confused with removing a prediction module at deployment: the full trained policy already deploys without its predictor. e03e04e08e18e20

What it supports. The full combination reaches 98.5% average success, compared with 95.3% for neither component. Removing world modeling costs 1.9 percentage points overall and 4.0 on Long; removing geometry costs 0.9 overall. This supports complementary benefits for the policy within the reported training setup.

Where the evidence stops. The table supplies no seed variation or confidence intervals. The gains establish behavioral usefulness in this setting, not geometric reconstruction accuracy or uniquely action-causal latent representations. The action-information control in Table 7 addresses a narrower mechanism.

Table 7. Keeping the predictor but withholding actual actions tests the information it uses. Original paper, p. 17 ↗

Excerpt from the authors’ paper; cropped without altering the figure or table.

How to read it. This comparison changes the predictor's action input rather than deleting its action branch. The Table 7 caption and Equations (17)–(18) state that the demonstrated intervening actions are replaced with an all-zero sequence before the same action encoder. Sequence length, temporal positional embeddings, architecture, objectives and other training settings are retained. Consequently, Null-Action Sequence still contains a learnable conditioning pathway and temporal structure; it simply lacks information about the actual robot commands. Compare the two rows within each suite. Because both are policies evaluated through task completion, the result measures the downstream effect of that training information. e05e20

What it supports. Actual actions improve every displayed suite: the full model reaches 98.5% average versus 97.6% for null actions, and 96.2% versus 94.6% on Long. The narrower control supports a contribution from action content beyond the retained pathway's capacity and temporal embeddings.

Where the evidence stops. Zero inputs are not equivalent to deleting the action encoder, whose learned parameters and positional embeddings can still produce nonzero features. The paper does not report action-order shuffling, repeated-seed uncertainty or direct prediction-error measurements for this control.

7. Analysis & limitations

7.1 What the evidence leaves open

Reader analysis

Tables report point estimates without confidence intervals or seed variation; simulation rollout counts and exact benchmark versions are omitted. Small physical trial counts limit precision. Attention maps are qualitative examples, not geometric-error measurements or causal proof of object-centric dynamics. e10e14e18e20e21

Reader analysis

The source labels the LIBERO-Plus Robot condition as embodiment shift in prose but Robot Initial States in Figure 22. Its exact coverage remains ambiguous. Larger backbones and broader cross-embodiment datasets are future work, not demonstrated extensions. e22e23

7.2 Questions for discussion

  1. Does action order matter beyond the information retained by an unordered action summary?
  2. Would geometry alignment still help under OOD shifts when future prediction and compute are held fixed?

8. Reproducibility audit

8.1 Requirements and known gaps

Source description

Required inputs include pretrained Qwen3.5-0.8B and VGGT-1B, LIBERO data, future-frame/action pairing, and the three-task real demonstration set. The physical system uses a UR5e, Azure Kinect and RealSense D405; an RTX 3090 workstation is specified for inference/control, not training. e03e04e09e14e16e17

Reader analysis

Missing implementation details include latent width, predictor depth/head counts, optimizer specifics, action-coordinate semantics and normalization, terminal future-pair handling, flow integration settings, software versions and training hardware/time. These require resolution before claiming faithful reproduction. The supplied text provides a project link, not verified executable artifacts. e01e05e07e09e16

8.2 Proposed reproduction checks

The following checks are proposals motivated by the paper. They have not been run as part of this reading.

Check 1: Distinguish action content from action order

Reader-proposed check, not executed: reproduce true-action and zero-action training, then add a variant that shuffles the eight actions within each chunk before adding the unchanged temporal embeddings. Keep the demonstration split, future-frame pairing, architecture, loss weights and training schedule fixed, with paired initialization seeds. Evaluate overall and Long success on identical initial states and report rollout counts and uncertainty across training runs. If true actions reproducibly outperform zeros but match shuffled actions, the evidence supports useful action content without establishing sensitivity to its temporal order. A reproducible advantage over both controls would strengthen the ordered-transition explanation. e05e09e20

Check 2: Test whether spatial correspondence explains the geometry gain

Reader-proposed check, not executed: compare aligned VGGT targets, spatially permuted targets and no geometry loss while holding the EMA predictor, action supervision and compute budget fixed. Permute teacher positions within each 8×8 grid to preserve its feature distribution while breaking correspondence to main-view policy tokens. Match seeds and evaluation initial states, and report success on LIBERO and the LIBERO-Plus Camera, Background and Layout conditions. If aligned targets consistently beat both controls, spatial correspondence becomes a stronger explanation for the gain. If permuted targets perform similarly, generic regularization remains a plausible alternative. Include the source's null-action control when assessing interactions. e04e08e09e12e18e20

8.3 Reading coverage

Visual audit: Visually inspected the title/author page; all scientific body pages; Appendix A–G method, preprocessing, training, hardware, result and ablation pages; and Appendix H–J rollout pages. Figures 1–27 and Tables 1–7 were inspected in their page context. All six final original crops were individually viewed, including the corrected framework crop. Figure 2 loss arrows and EMA symbols were checked against Eqs. (4), (7)–(8) and Appendix B; Figure 5 query/key/value directions agree with its caption and Appendix B. References on pages 8–10 were read as text and are not a basis for independently verified claims about other papers. Separate supplements, external code and linked pages remain outside this pass. Illustrative rollout strips are not treated as additional success-rate measurements.

PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22. Appendix coverage: reviewed.

Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.

Text reading scope & known omissions
  • Abstract
  • 1 Introduction
  • 2 Related Work
  • 3 Method: 3.1–3.5
  • 4 Experiments: 4.1–4.5
  • 5 Conclusion
  • References
  • Appendix A: Details of Learnable Dynamics Tokens
  • Appendix B: Details of Self-Guided World Predictor
  • Appendix C: Implementation Details of Geometric Supervision
  • Appendix D: Training Objectives
  • Appendix E: Simulation Experiments
  • Appendix F: Real-World Experiments
  • Appendix G: Ablation Study
  • Appendix H: Real-World OOD Rollouts Visualization
  • Appendix I: LIBERO Visualization
  • Appendix J: LIBERO-Plus Visualization

Outside the original text pass

  • Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
  • Separate supplemental material availability has not been fully verified.
  • Identity/version: the title page identifies arXiv:2608.01397v1 [cs.RO], 2 August 2026. The exact title and all nine authors agree with the catalog after normalizing name order and spacing; Marcelo H. Ang Jr. is verified from the title page. No different revision or edition was supplied or substituted.
  • All seven supplied text chunks were read individually, covering all 22 pages. Text extraction does not reconstruct figure images; this was addressed by inspecting the retained PDF's scientific pages and all final crops. References on pages 8–10 were read as text.
  • Separate supplemental material availability has not been fully verified; no separate supplement was supplied.
  • Code, project website and external cited works were not inspected. No training, inference experiment or reproduction was run.

The visual audit above records the subsequent illustrated pass.

8.4 Traceable evidence

e01PDF p. 1, title/author block, arXiv margin and project linkInspect

Exact title and nine authors are printed on arXiv:2608.01397v1 [cs.RO], dated 2 Aug 2026. Four institutional affiliations and a project URL are present.

Go to primary source ↓
e02PDF pp. 1–2, Section 1 and Figure 1Inspect

Motivates policy-derived future targets to reduce perceptual burden and potential target-policy mismatch while preserving geometry.

Go to primary source ↓
e03PDF p. 3, Figure 2 and Sections 3.1–3.2, Eqs. (1)–(3); p. 4, opening paragraphsInspect

Qwen3.5-0.8B contextualizes views, instruction and dynamics tokens; full states condition a separate action expert. Geometry, predictor and EMA pathways are auxiliary and removed at inference.

Go to primary source ↓
e04PDF p. 4, Section 3.3, Eq. (4); p. 12, Appendix C, Eq. (13); p. 13, Eq. (15)Inspect

Frozen VGGT-1B processes one 518×518 main frame; 1,369 final-layer patch tokens form a 37×37 grid pooled to 8×8. Existing main-view policy tokens are projected and cosine-aligned; VGGT is not the future-target encoder.

Go to primary source ↓
e05PDF p. 4, Section 3.3, Eqs. (5)–(8); p. 11, Appendix B and Figure 5; p. 12, Eqs. (11)–(14)Inspect

Eight seven-dimensional actions become ordered MLP tokens with temporal embeddings. Dynamics states supply cross-attention queries; actions supply keys/values. Future targets use an EMA backbone, projector and token set with momentum 0.999 and no gradients. MSE averages over token and feature dimensions.

Go to primary source ↓
e06PDF p. 11, Appendix AInspect

Eight continuous dynamics embeddings are appended after image/language embeddings, extending valid attention positions and rotary indices. They participate in all transformer layers and remain at inference.

Go to primary source ↓
e07PDF p. 4, Section 3.4, Eq. (9)Inspect

Conditional flow matching interpolates Gaussian noise and expert actions and learns velocity At−epsilon conditioned on full policy states. Inference integrates that field; solver and integration-step count are not specified.

Go to primary source ↓
e08PDF p. 4, Eq. (10); p. 5, Section 3.5 continuation; p. 13, Appendix D, Eq. (16)Inspect

Single-stage joint optimization uses action, geometry and prediction losses, with both auxiliary weights 0.1. Online modules receive gradients; VGGT stays frozen and target policy modules update through EMA.

Go to primary source ↓
e09PDF p. 13, Appendix E, training paragraphInspect

Pretrained Qwen3.5-0.8B initialization; four-suite joint training for 40k steps, batch 96, cosine schedule with 5k warmup, peak VLM/train-only/action-head learning rates 1e-5/2.5e-5/1e-4.

Go to primary source ↓
e10PDF p. 5, Section 4.1; pp. 13–14, Appendix EInspect

LIBERO evaluation follows joint four-suite training; LIBERO-Plus is zero-shot with no fine-tuning. Overall score counts successful rollouts. Simulation baselines are taken from their respective papers; exact rollout counts, seeds and software versions are not provided.

Go to primary source ↓
e11PDF p. 5, Table 1, SG-WAM and Spatial Forcing rowsInspect

SG-WAM: 0.9B, no embodied pretraining, 99.4/99.8/98.6/96.2 and 98.5 average. Spatial Forcing: 7B with embodied pretraining, 98.5 average.

Go to primary source ↓
e12PDF p. 5, Table 2, SG-WAM, OpenVLA-OFT and VLA-JEPA rowsInspect

SG-WAM Camera/Robot/Language/Light/Background/Noise/Layout/Overall: 58.6/48.9/81.4/89.8/86.1/80.7/74.2/73.0. OpenVLA-OFT overall 69.6; VLA-JEPA overall 62.9 and Layout 74.6.

Go to primary source ↓
e13PDF p. 6, Section 4.2, best-layout statement; p. 5, Table 2, Layout columnInspect

The prose calls SG-WAM best on layout shifts, whereas the table gives VLA-JEPA 74.6 and SG-WAM 74.2. The table supports an overall lead, not a layout lead.

Go to primary source ↓
e14PDF p. 6, Section 4.3 and Figure 3; p. 14, Appendix F, task descriptionsInspect

UR5e with Azure Kinect main and RealSense D405 wrist RGB. Each of three tasks has 100 demonstrations; each method receives 20 pick/towel trials and 10 toolbox trials. Initial scenes are randomized and approximately matched across methods.

Go to primary source ↓
e15PDF p. 7, Table 3, all task/condition columnsInspect

SG-WAM Pick ID/background/light/novel: 75/55/60/40; towel: 45/25/35/25; toolbox ID: 50. VLA-JEPA: 35/20/25/20, 20/10/10/15, 20. VPP: 30/15/10/10, 35/15/15/10, 30. Toolbox OOD columns are absent.

Go to primary source ↓
e16PDF p. 15, Appendix F, OOD definitions and training paragraphInspect

OOD replaces the table covering or lighting, uses an unseen towel or novel-color cubes with updated instructions. Models train on three tasks for 40 epochs, batch 96, simulation learning rates; same final checkpoint serves ID/OOD without additional adaptation.

Go to primary source ↓
e17PDF p. 16, Appendix F, hardware sentence and Table 6Inspect

RTX 3090 workstation is used for inference/control. SG-WAM towel stage rates are 75/45; toolbox stage rates are 80/60/50/50. The table does not define these as conditional probabilities.

Go to primary source ↓
e18PDF p. 7, Table 4; p. 6, Section 4.5Inspect

Average success for neither/geometry-only/world-modeling-only/both is 95.3/96.6/97.6/98.5. Long is 91.0/92.2/94.4/96.2. No uncertainty estimates accompany these comparisons.

Go to primary source ↓
e19PDF p. 7, Table 5 and token-count discussion; p. 6, Section 4.5 continuationInspect

One/four/eight/sixteen dynamics tokens yield 96.1/97.5/98.5/97.2 average and 90.2/95.2/96.2/92.4 Long, with other settings held fixed.

Go to primary source ↓
e20PDF p. 17, Appendix G, Table 7 and Eqs. (17)–(18); p. 18, continuationInspect

Zeros replace actual actions before the unchanged encoder, retaining temporal embeddings and training settings. Null/full average is 97.6/98.5 and Long 94.6/96.2; Spatial 98.4/99.4, Object 99.2/99.8, Goal 98.0/98.6.

Go to primary source ↓
e21PDF p. 7, Figure 4 and Section 4.5; p. 17, Figure 10 and Appendix GInspect

Selected and all-eight-token attention maps compare geometry on/off for one illustrated scene. Authors associate geometry with attention to interaction regions; no quantitative geometric accuracy or attention color scale is supplied.

Go to primary source ↓
e22PDF p. 6, Section 4.2; p. 13, Appendix E; p. 21, Figure 22 captionInspect

The prose calls the Robot perturbation an embodiment shift, while Figure 22 labels it Robot Initial States; the supplied source does not resolve that difference.

Go to primary source ↓
e23PDF p. 7, Section 5, final paragraphInspect

Scaling to larger policy backbones and broader cross-embodiment datasets is proposed as future work.

Go to primary source ↓

8.5 Primary sources

Scroll across the image to inspect details. Press Esc to close.