Awesome-WAMs
GitHub
PAPER REPORTENAll readings ↗

GeoSem-WAM: Geometry- and Semantic-Aware World Action Models

English reading report: Method, equations, original figures, experiments and reproducibility.

Authors: Fulong Ma; Daojie Peng; Wenjun Yue; Jiahang Cao; Bintao Wang; Qiang Zhang; Jun Ma

Affiliations: HKUST(GZ); HKU; USTC; OC; SDU; X-Humaniod

Source: 2606.03188 ↗ · Catalog record

Reading: 187 / 558 · 6 original figures & tables · ~19 min ·

1. Paper overview

In one sentence: Training a video/action model to predict future geometry and semantics improves reported manipulation success while allowing those dense prediction heads to be removed at deployment. e2e3e4e5e7e9e13

At a glanceWhat to know
Research problem
Author claim

RGB prediction can encourage appearance modeling without sufficiently encoding object identity or spatial layout. The authors ask whether richer predictive supervision improves action-relevant representations while retaining deployment without future video generation. Their broader explanation of why WAMs work remains a motivation rather than a general causal result established here. e2

Core mechanism
Source description

DPT-style heads supervise future geometry and semantic segmentation from intermediate video-expert tokens, alongside RGB and action flow losses. e3e5

A key reported resultFranka manipulation and visual/geometric generalization: Easy-Pick/B1/B2/D: 100/96/94/92; Multi-Pick/Multi-Goal/Pick-Pour: 98/96/92; average 95.4.

Success rate (%). Four core tasks, two background variants and a height test involving a 4 cm platform change; 50 independent trials per scenario.

Fast-WAM: 100/86/86/80/92/90/88; average 88.9. Scenario gains are 0/10/8/12/6/6/4 points. Their mean, about 6.57 points, explains printed +6.6 despite rounded averages differing by 6.5. Easy-Pick ties; the height-test column gains most. e11e12

Reading caution
Author claim

The authors identify scarce pixel annotations, potentially noisy pseudo-labels and gradient conflicts among heterogeneous losses. Foundation-model features and gradient surgery are future proposals, not evaluated components. e13

Core contributions

  • Source description

    DPT-style heads supervise future geometry and semantic segmentation from intermediate video-expert tokens, alongside RGB and action flow losses. e3e5

  • Source description

    Simulation comparisons, a four-condition ablation, qualitative representation probes and physical Franka experiments evaluate the auxiliary signals. e7e8e9e10e12

Figure 1. Predict rich future targets during training; retain the observation-to-action route for deployment. Original paper, p. 3 ↗

Excerpt from the authors’ paper; cropped without altering the figure or table.

How to read it. Start with the dashed rectangle: the initial RGB frame and instruction supply context, while noisy actions enter the action branch and leave as clean actions. The arrow labeled KV Cache points from Video DiT toward Action DiT; the text describes shared attention between these experts. Above the deployment region, future video passes through the VAE/video path, and DPT outputs future depth and semantic sequences. Those dense outputs provide training supervision and are removed at deployment. The figure simplifies the training path: Eqs. (1)–(6) explicitly add noise to video latents and actions, even though video corruption is not drawn here. e3e4e5

What it supports. The architecture supports a representation-learning interpretation of world modeling: future visual and structural targets shape a context representation used by an action expert. It does not require decoding a future video before selecting actions. Geometry and semantic annotations supervise training rather than serving as required observation inputs.

Where the evidence stops. Section 3.1 calls inference a single forward pass, but Section 3.2 says actions are iteratively denoised. The figure's KV-cache arrow does not resolve the denoising schedule or cache-update policy. No measured latency accompanies this diagram.

2. Motivation

2.1 The problem and the proposed response

Author claim

RGB prediction can encourage appearance modeling without sufficiently encoding object identity or spatial layout. The authors ask whether richer predictive supervision improves action-relevant representations while retaining deployment without future video generation. Their broader explanation of why WAMs work remains a motivation rather than a general causal result established here. e2

2.2 What this reading follows

Consider a robot choosing where to grasp an object: matching future image appearance need not force its internal features to distinguish object identity from spatial layout. GeoSem-WAM adds future depth and semantic supervision to address that gap. Its video and action experts learn together, but deployment retains action generation without future video rollout. Read the architecture first, then the dense-head mechanism, qualitative probes and success tables. The evidence favors structured supervision within this setup; it leaves annotation quality, optimization details and inference timing unresolved. Several printed simulation averages also require reconciliation before precise comparative gains can be trusted. e2e3e4e5e7e9e13

3. Research context

We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.

Catalog dimensionRecorded classification
Major categoryWAMs
ArchitectureDual-system
Prediction paradigmOther mechanisms
QuadrantOutside quadrants

3.1 Evidence-based assessment

Supports the recorded classification

Reader analysis

Separate video/action DiT experts support Dual-system; training-only predictive supervision supports Other mechanisms and Outside quadrants. This is neither explicit inverse dynamics nor inference-time joint future/action rollout. The 3D multiview label needs qualification: depth supervision and multiple illustrated views do not establish 3D reconstruction or cross-view consistency. Efficient inference is an architectural motivation without measured latency. e3e4e5e14

This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.

4. Problem formulation

4.1 Inputs and outputs

InputsOutputs
  • Current RGB observation, language instruction and Gaussian action noise at deployment (e3, e4).
  • Training targets: future video, demonstrated actions, geometry and semantic labels; dense annotations are supervision, not policy inputs (e3–e5).
  • A denoised action chunk for execution (e4).
  • Training-only future RGB flow and dense geometry/semantic predictions (e4, e5).

4.2 Equations and their role

zt:t+Kσ=(1−σ)zt:t+Kgt+σϵz,v^z=fθrgb(zt:t+Kσ,σ,c),vz=ϵz−zt:t+Kgtz^\sigma_{t:t+K}=(1-\sigma)z^{\mathrm{gt}}_{t:t+K}+\sigma\epsilon_z,\quad \hat v_z=f^{\mathrm{rgb}}_\theta(z^\sigma_{t:t+K},\sigma,c),\quad v_z=\epsilon_z-z^{\mathrm{gt}}_{t:t+K}
Eqs. (1)–(2): z with superscript gt denotes target video latents; sigma is a noise level in [0,1], epsilon_z is Gaussian noise, and c includes current vision and language. The video DiT predicts flow v_z; Eq. (3) penalizes squared error. e4
at:t+H−1σ=(1−σ)at:t+H−1gt+σϵa,v^a=fϕact(at:t+H−1σ,σ,zt),Lact=∥v^a−(ϵa−at:t+H−1gt)∥22a^\sigma_{t:t+H-1}=(1-\sigma)a^{\mathrm{gt}}_{t:t+H-1}+\sigma\epsilon_a,\quad \hat v_a=f^{\mathrm{act}}_\phi(a^\sigma_{t:t+H-1},\sigma,z_t),\quad \mathcal L_{\mathrm{act}}=\|\hat v_a-(\epsilon_a-a^{\mathrm{gt}}_{t:t+H-1})\|_2^2
Eqs. (4)–(6): a with superscript gt is the demonstrated H-step action chunk, epsilon_a is Gaussian noise, and z_t is the latent world representation. This flow-regression loss does not guarantee one-step action sampling. e4
Lgeo=1K∑τ∥o^τgeo−oτgeo∥1,Lsem=1K∑τCE(o^τsem,oτsem),L=λrgbLrgb+λgeoLgeo+λsemLsem+λactLact\mathcal L_{\mathrm{geo}}=\frac{1}{K}\sum_\tau\|\hat o^{\mathrm{geo}}_\tau-o^{\mathrm{geo}}_\tau\|_1,\quad \mathcal L_{\mathrm{sem}}=\frac{1}{K}\sum_\tau\mathrm{CE}(\hat o^{\mathrm{sem}}_\tau,o^{\mathrm{sem}}_\tau),\quad \mathcal L=\lambda_{\mathrm{rgb}}\mathcal L_{\mathrm{rgb}}+\lambda_{\mathrm{geo}}\mathcal L_{\mathrm{geo}}+\lambda_{\mathrm{sem}}\mathcal L_{\mathrm{sem}}+\lambda_{\mathrm{act}}\mathcal L_{\mathrm{act}}
Eqs. (7)–(9): tau ranges from t+1 through t+K. Hatted o terms are predicted geometry or semantic logits; unhatted terms are targets. CE is pixel-wise cross-entropy. Lambda coefficients balance the four objectives; numerical settings are absent. e5

5. Method in detail

5.1 Use future structure as a training target

Reader analysis

The essential separation is between information available to the policy and targets used to teach it. GeoSem-WAM takes current visual observations and language, while demonstrated future video, actions and dense labels supply learning signals. The video DiT is therefore not merely an image generator attached to a robot: its intermediate tokens must also support geometry reconstruction and semantic discrimination. The DPT pathway reassembles several token levels and fuses them before producing dense outputs. Reader interpretation: requiring multiple descriptions of the same future scene could discourage a representation that captures only appearance shortcuts. The actual evidence for that interpretation is limited to the qualitative probes and policy ablation; the architecture itself does not show that every useful geometric or semantic property is learned. e3e4e5e8e10

Figure 2. Intermediate video features receive dense spatial and semantic supervision through a shared decoding path. Original paper, p. 5 ↗

Excerpt from the authors’ paper; cropped without altering the figure or table.

How to read it. Follow the arrows from the VAE encoder into the bottom Transformer block and upward through Video DiT. At several depths, horizontal branches send features into 3D Reassemble modules. On the right, the 3D Fusion cascade flows downward, combining these multi-level representations before splitting into Geometry and Semantic outputs. This direction matters: the two outputs are decoded from fused intermediate features rather than supplied as perception inputs. Section 3.2 describes projection, reassembly, progressive fusion and upsampling, extending DPT-style operations to video. Eqs. (7) and (8) attach L1 geometry error and pixel-wise semantic cross-entropy at future prediction steps. e3e5e10

What it supports. The same video backbone is asked to support both spatial reconstruction and object-level labeling. The proposed benefit is a latent representation useful to the action expert after these heads are removed. Table 3 provides the relevant policy comparison; this diagram alone establishes the supervision pathway, not its effectiveness.

Where the evidence stops. The source does not specify token-tap indices, channel sizes, dense-label production or loss coefficients. The figure's four illustrated feature levels are not a complete implementation configuration. Geometry prediction also does not by itself establish explicit 3D scene reconstruction.

5.2 Separate context computation from action denoising

Reader analysis

During training, both video latents and action chunks are corrupted with Gaussian noise and their experts predict flow targets. At deployment, the paper removes future video generation and the DPT heads, but it explicitly retains noisy-action initialization and iterative denoising in Section 3.2. This makes 'no future rollout' a narrower statement than 'no iteration.' Figure 1's video-to-action KV-cache arrow suggests how an observation-derived context could support an action expert efficiently. Reader interpretation: one context computation followed by repeated action updates could reconcile the overview's single-forward-pass language with the detailed action description. That remains an interpretation because neither a denoising schedule nor cache-update algorithm is supplied. The resulting chunk is intended for control, but the paper leaves chunk execution frequency and feedback timing unspecified. e3e4

5.3 Read success gains alongside their controls

Reader analysis

The ablation supplies the strongest direct evidence that the added objectives matter: RGB-only, geometry, semantics and joint supervision are compared on LIBERO. Both single additions improve the displayed average and the combination scores highest, but absent seed dispersion and training-budget matching limit a causal conclusion. The Franka table then tests a different question: whether the trained policy succeeds under selected background, height and task-complexity changes. Its larger gains are informative, yet they do not quantify the same generalization distribution as RoboTwin randomization. Reader interpretation: confidence should come from agreement across these distinct observations, not from merging their averages. Simulation reporting also needs care: the Fast-WAM LIBERO suite values and RoboTwin clean/random values do not reproduce all printed aggregates, so exact margins remain qualified. e6e7e9e10e11e12e15

5.4 Training and inference

During training

Reader analysis

Four weighted objectives are jointly optimized. Pretrained encoder frozen/trainable status, loss weights, optimizer schedule, action-expert size and prediction horizons are not specified. e3e4e5

Source description

LIBERO provides 500 demonstrations per suite across 10 tasks. Appendix A describes paired RGB, depth and semantic observations. Franka uses 50 teleoperated trajectories per core task, idle-frame removal and action smoothing, with Fast-WAM-style fine-tuning on two NVIDIA H800 GPUs. e6e11e14

During inference

Source description

Condition on the first observation, omit future video rollout and discard DPT heads. Section 3.2 initializes actions from Gaussian noise and iteratively denoises them. Physical deployment uses an RTX 4090. e3e4e11

Reader analysis

Section 3.1 instead describes a single forward pass. One video-context pass with repeated action steps is a plausible reconciliation, but the schedule is unspecified. Chunk execution frequency and feedback timing are also absent. e3e4

5.5 Implementation flow

  1. Couple video and action experts

    The Wan2.2-5B video DiT supplies the backbone, with its pretrained VAE and native T5 encoder reused. Language conditions tokens through cross-attention. A differently sized action DiT shares attention with the video branch in a Mixture-of-Transformer architecture. Figure 1 depicts a video-to-action KV cache. e3

  2. Learn predictive latent features

    Ground-truth video latents and action chunks are mixed with Gaussian noise. Both experts regress noise-minus-target flows using squared error. The action flow is conditioned on the current latent world representation, without an inference-time search over imagined outcomes. e4

  3. Decode dense structure

    Tokens from several video Transformer blocks enter 3D reassemble modules. Fusion combines and upsamples these features before geometry and semantic heads. Future geometry uses L1 reconstruction; semantic logits use pixel-wise cross-entropy. e5

6. Experiments & results

GeoSem-WAM adds future depth and semantic prediction losses to a video/action model, then removes those prediction heads for robot deployment. Its central evidence is improved manipulation success with structured training supervision. Simulation gains are small and some averages are internally inconsistent; physical background and height tests show larger gains. Architecture and result evidence: e3–e15.

6.1 Read the original evidence

Table 1. LIBERO reports a high mean success rate, with an unresolved Fast-WAM averaging discrepancy. Original paper, p. 6 ↗

Excerpt from the authors’ paper; cropped without altering the figure or table.

How to read it. Read each row across the four task suites before looking at the average column. Section 4.2 specifies 50 trials per task under different random seeds; Section 4.1 describes 10 tasks and 500 demonstrations per suite. The bottom GeoSem-WAM row is strongest on some columns, but the Long column favors LingBot-VA. Fast-WAM is the mechanism-relevant baseline because the paper adds structured supervision to its training approach. Compare it with GeoSem-WAM both suite by suite and in the printed average column. This distinction is essential here because the baseline's displayed average does not equal the arithmetic mean of its suite entries. e6e7e10

What it supports. GeoSem-WAM reports 99.0, 100.0, 98.2 and 97.0 across the suites, averaging 98.55%. Its printed mean exceeds Fast-WAM's reported 97.60% by 0.95 percentage points and LingBot-VA's 98.50% by 0.05 points. LingBot-VA nevertheless leads on Long, 98.5% versus 97.0%.

Where the evidence stops. Fast-WAM's printed suite values average 97.35%, not the reported 97.60%; the source does not reconcile them. Preserve both rather than silently correcting the table. No confidence intervals or fully harmonized baseline training protocol establish the significance of these small average differences.

Table 4. Physical generalization tests show larger improvements than the simulation averages. Original paper, p. 8 ↗

Excerpt from the authors’ paper; cropped without altering the figure or table.

How to read it. Start with the Easy-Pick column, where both policies reach 100%. Then read the B1 and B2 background variants and the D column associated with height generalization. Figure 4 shows the two mats and a standard/elevated comparison with a 4 cm difference. Finish with Multi-Pick, Multi-Goal and Pick-Pour, which increase distractors, target assignments or sequential manipulation demands. The caption specifies 50 independent trials per scenario; the training description separately reports 50 demonstrations per core task. Keep those counts distinct. Finally, inspect the Improvement row against the two policy rows rather than treating it as an independently measured metric. e11e12

What it supports. GeoSem-WAM improves the background variants by 10 and 8 percentage points and the height-test column by 12 points. Multi-Pick, Multi-Goal and Pick-Pour gain 6, 6 and 4 points. The seven gains average about 6.57 points, which rounds to the printed 6.6; the reported policy averages are 95.4% and 88.9%.

Where the evidence stops. Figure 4 labels the standard panel Easy-Pick-D and the elevated panel Depth Setting, so the exact table-to-height-condition mapping remains imperfectly explained. The table gives no confidence intervals, and its trial count does not establish broad robustness outside these selected laboratory variations.

6.2 Results and evaluation conditions

Task & protocolReported resultComparison & interpretation
LIBERO four-suite manipulation

Spatial, Object, Goal and Long; 50 trials per task under different random seeds.

99.0 / 100.0 / 98.2 / 97.0; reported average 98.55.

Success rate (%)

Fast-WAM: reported 97.60 average; LingBot-VA: 98.50 average and 98.5 Long.

Reported averages imply +0.95 percentage points over Fast-WAM, but its printed suite scores average 97.35, not 97.60. GeoSem-WAM is not best on Long. No uncertainty establishes its 0.05-point average lead over LingBot-VA. e6e7

RoboTwin 2.0 bimanual manipulation

50 tasks in clean/randomized settings; neither GeoSem-WAM nor Fast-WAM uses embodied pretraining according to Table 2.

Clean 92.94; random 92.14; reported overall 92.52.

Success rate (%)

Fast-WAM: 91.88 / 91.78 / reported 91.80; LingBot-VA: 92.90 / 91.50 / 92.20.

Reported aggregates differ by +0.72 points. Equal clean/random means would be 92.54 and 91.83, leaving an unresolved discrepancy. Table 5 shows regressions too: Move Stapler Pad is 73/63 versus Fast-WAM 77/64. e9e15

LIBERO structured-supervision ablation

RGB-only baseline; add geometry, semantics, or both.

97.6 baseline; 98.2 geometry; 98.1 semantics; 98.6 both.

Average success rate (%)

Both exceeds either single auxiliary objective.

Printed rates imply +0.6, +0.5 and +1.0 percentage points. The displayed +0.61%, +0.51% and +1.02% are consistent with relative gains. Seed dispersion and a compute-matched control are absent. e10

Franka manipulation and visual/geometric generalization

Four core tasks, two background variants and a height test involving a 4 cm platform change; 50 independent trials per scenario.

Easy-Pick/B1/B2/D: 100/96/94/92; Multi-Pick/Multi-Goal/Pick-Pour: 98/96/92; average 95.4.

Success rate (%)

Fast-WAM: 100/86/86/80/92/90/88; average 88.9.

Scenario gains are 0/10/8/12/6/6/4 points. Their mean, about 6.57 points, explains printed +6.6 despite rounded averages differing by 6.5. Easy-Pick ties; the height-test column gains most. e11e12

6.3 Ablations and diagnostic examples

Read component removals and qualitative examples within their stated evaluation conditions.

Figure 3. Qualitative probes connect structured supervision to semantic separation and depth information in latent tokens. Original paper, p. 6 ↗

Excerpt from the authors’ paper; cropped without altering the figure or table.

How to read it. Read the two scatter panels as a qualitative comparison of intermediate video-token embeddings colored by semantic class. The right scatter shows more visibly separated groups, which the caption and discussion associate with GeoSem-WAM's improved semantic organization. In panel (c), read columns (1) through (4) as RGB observation, Fast-WAM probe depth, GeoSem-WAM probe depth and ground-truth depth. Compare object boundaries and the robot silhouette across the latter three columns. The described depth experiment freezes the backbone and trains a simple probe using intermediate tokens, so these images inspect information recoverable from the representation rather than direct robot execution. e8e7e10

What it supports. The shown GeoSem-WAM depth predictions more closely resemble the ground-truth shapes than the blurrier baseline examples. Together with the embedding plots, this is qualitative support for the authors' representation hypothesis. The policy-success tables are separate evidence for whether these representations accompany better manipulation.

Where the evidence stops. No class-name legend, depth color scale, projection specification or quantitative probe score accompanies the figure. These omissions belong to the source, not the crop. Selected examples cannot establish average depth accuracy or prove that representation changes cause action gains.

Table 3. Both auxiliary objectives improve the displayed success rate; their combination scores highest. Original paper, p. 7 ↗

Excerpt from the authors’ paper; cropped without altering the figure or table.

How to read it. Read the checkmarks first. Every row retains RGB supervision, while the second and third rows add geometry or semantics individually and the final row enables both. This is the paper's clearest comparison addressing whether each structured target contributes. The Average SR column contains the success rates; the final delta column is labeled percent. Subtract the displayed success rates from the RGB-only baseline to obtain percentage-point gains. Separately, divide those gains by the baseline to understand why the printed delta values are approximately 0.61, 0.51 and 1.02. The paper does not explicitly define that conversion, so this interpretation is arithmetic analysis. e10e7

What it supports. Geometry alone raises displayed success from 97.6% to 98.2%, semantics alone to 98.1%, and both to 98.6%. These are gains of 0.6, 0.5 and 1.0 percentage points. The joint setting performs best, supporting complementary usefulness within this reported experiment without establishing statistical interaction or universal benefit.

Where the evidence stops. The delta column is consistent with relative percentages, not percentage-point changes. No training-budget control or seed variance is supplied. The joint score is rounded to one decimal here, while Table 1 reports 98.55%; their different precision should remain visible.

7. Analysis & limitations

7.1 What the evidence leaves open

Author claim

The authors identify scarce pixel annotations, potentially noisy pseudo-labels and gradient conflicts among heterogeneous losses. Foundation-model features and gradient surgery are future proposals, not evaluated components. e13

Reader analysis

Figure 3 suggests more separable semantic embeddings and sharper frozen-backbone depth probes, but gives no depth-error or clustering metric. Success gains do not establish explicit 3D consistency, general reasoning, or that future imagination is unnecessary for every WAM. e2e8

Reader analysis

No latency, control-frequency or training-cost benchmark quantifies efficiency. Baseline protocols are incompletely documented; Tables 1–4 lack confidence intervals. Some Table 5 entries differ from Table 2, including Motus clean success (88.52 versus 88.66). e3e7e9e10e12e15

7.2 Questions for discussion

  1. Would equal-budget training and repeated seeds preserve the joint-supervision advantage?
  2. How sensitive are generalization gains to noisy dense labels?
  3. Can raw trial counts reconcile the simulation aggregates?

8. Reproducibility audit

8.1 Requirements and known gaps

Reader analysis

Required components include Wan initialization, shared-attention action expert, DPT token taps, synchronized labels and action representation. Missing details include normalization, denoising steps, attention/cache rules, training schedules, software versions and exact train/test partitions. e3e4e5e14

Reader analysis

Physical replication needs the stated data, hardware and scene conditions, plus unspecified idle thresholds, smoothing parameters and label provenance. Appendix B illustrates Pick-Pour from two viewpoints; it supplies neither a controller specification nor additional independent trials. e11e14e16

8.2 Proposed reproduction checks

The following checks are proposals motivated by the paper. They have not been run as part of this reading.

Check 1: Test whether aligned dense targets explain the ablation gain

Reader-proposed experiment, not performed: reproduce the four Table 3 conditions using identical demonstrations, splits, initialization, optimization steps and evaluation seeds, logging actual training compute. Add a control with the same auxiliary heads and loss weights but geometry/semantic labels shuffled across training examples. Remove every auxiliary head for evaluation. Report per-task success and uncertainty, plus held-out depth-probe error using identical probes. If correctly aligned labels consistently outperform RGB-only and shuffled-label controls, that supports the structured-information explanation. If shuffled labels match the gain or repeated-seed intervals overlap substantially, the claimed mechanism needs a weaker interpretation. Obtain the missing loss weights and label recipe before calling this a faithful reproduction. e3e5e8e10e13e14

Check 2: Resolve the deployment schedule and test the removal claim

Reader-proposed experiment, not performed: once the actual attention mask, cache policy and denoising schedule are available, instrument a trained checkpoint on the stated RTX 4090. Hold observations, instructions, initial action noise and action-denoising steps fixed. Compare the same checkpoint with future-video/DPT computations retained versus disabled, retaining identical action conditioning. Record video/action forward-call counts, action outputs, latency, peak memory and success on the same Franka conditions. If removal changes actions under matched conditioning, it would reveal an undocumented dependency; if outputs agree and latency falls, the training-only interpretation is supported. Count action updates separately to resolve the single-forward-pass wording. Predefine which standard/elevated setup corresponds to Easy-Pick-D. e3e4e11e12

8.3 Reading coverage

Visual audit: Inspected the title/authors/version page, all method and equation pages, all six original figures and all five tables, including the complete task table and both appendices. All six final crops were individually viewed and retain their original labels and table headers. Pages 4–5 establish equations and training details; page 7 establishes hardware and trial protocols; pages 12–14 establish appendix observations and episode scope. Cross-checked Figure 1 against Sections 3.1–3.2 and Figure 2 against Eqs. (7)–(9). Preserved the single-pass versus iterative-action ambiguity, simulation aggregate discrepancies and height-panel naming ambiguity. Figure 6's first-person Depth/Sem legend appears interchanged; no modality-specific conclusion relies on that legend. Reference pages 9–11 were read as text only. Separate supplemental material and code were not inspected.

PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 12, 13, 14. Appendix coverage: reviewed.

Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.

Text reading scope & known omissions
  • Title, authors, abstract and version (p. 1)
  • 1 Introduction (pp. 1–2)
  • 2 Related Works, 2.1–2.3 (pp. 2–3)
  • 3 Methodology, 3.1–3.2, Eqs. (1)–(9) (pp. 3–5)
  • 4 Experiments, 4.1–4.4 (pp. 5–8)
  • 5 Conclusion (p. 8)
  • Acknowledgments and References (pp. 9–11)
  • Appendix A: Simulation Environments and Multi-Modal Observations (pp. 12–13)
  • Appendix B: Example Episodes of Real World Experiments on Franka (pp. 13–14)

Outside the original text pass

  • Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
  • Separate supplemental material availability has not been fully verified.
  • Identity/version: the inspected title and all seven authors match the catalog; the source is arXiv:2606.03188v1, dated 2 June 2026 (e1). No revision or edition difference was observed; no other version was inspected.
  • All four supplied text chunks were read. Text extraction does not reconstruct figures; this was addressed by inspecting all six original figures and all five tables in the PDF.
  • Separate supplemental material availability has not been fully verified; no separate supplement was supplied.
  • Code and external resources were not inspected, and no experiments were reproduced.
  • Reference pages 9–11 were read as text; visual inspection covered pages 1–8 and 12–14.

The visual audit above records the subsequent illustrated pass.

8.4 Traceable evidence

e1PDF p. 1, title block, affiliations, author footnotes and arXiv margin stampInspect

Title: GeoSem-WAM: Geometry- and Semantic-Aware World Action Models. Authors: Fulong Ma, Daojie Peng, Wenjun Yue, Jiahang Cao, Bintao Wang, Qiang Zhang, Jun Ma. Version: arXiv:2606.03188v1 [cs.RO], 2 Jun 2026. Fulong Ma and Daojie Peng contribute equally; Jun Ma is corresponding author. Affiliations are printed as HKUST(GZ), HKU, USTC, OC, SDU and X-Humaniod.

Go to primary source ↓
e2PDF pp. 1–2, Abstract and Section 1, motivation and contributionsInspect

The authors motivate geometric/semantic prediction beyond RGB and attribute WAM benefits primarily to learned representations while avoiding explicit future generation at deployment.

Go to primary source ↓
e3PDF p. 3, Figure 1 and Section 3.1; p. 4, Sections 3.1–3.2 architecture paragraphInspect

Wan2.2-5B video DiT, native T5 and VAE are reused with a differently sized action DiT and shared attention. Figure 1 labels a video-to-action KV cache and dashed inference region. Section 3.1 describes a single forward pass from the first observation and excludes geometry/semantic annotations as inputs; DPT heads are discarded.

Go to primary source ↓
e4PDF p. 4, Section 3.2, Eqs. (1)–(6) and Action Modeling inference paragraphInspect

Video/actions use linear Gaussian corruption and noise-minus-target flow regression. Action flow is conditioned on z_t; inference initializes Gaussian actions and iteratively denoises them without future frames.

Go to primary source ↓
e5PDF pp. 4–5, Dense Structured World Supervision; p. 5, Figure 2 and Eqs. (7)–(9)Inspect

DPT heads aggregate multi-level tokens through 3D reassembly/fusion. Geometry uses future-step L1 loss and semantics pixel-wise cross-entropy, alongside weighted RGB/action losses. Numerical balancing weights are not given.

Go to primary source ↓
e6PDF p. 5, Sections 4.1–4.2, LIBERO protocolInspect

Four suites each provide 500 demonstrations over 10 tasks. Each task is evaluated for 50 trials under different random seeds; suite and overall success rates are reported.

Go to primary source ↓
e7PDF p. 6, Table 1, GeoSem-WAM, Fast-WAM and LingBot-VA rowsInspect

GeoSem-WAM: 99.0/100.0/98.2/97.0, average 98.55. Fast-WAM: 97.2/100.0/97.0/95.2, printed average 97.60. LingBot-VA: 98.5/99.6/97.2/98.5, average 98.50. No uncertainty columns appear.

Go to primary source ↓
e8PDF p. 6, Figure 3(a–c), caption and Section 4.2 semantic/geometric analysisInspect

Colored intermediate-token embeddings illustrate semantic clustering. A simple depth probe is trained on frozen-backbone intermediate tokens; panel c columns are RGB, Fast-WAM depth, GeoSem-WAM depth and ground truth. No numeric probe metric is reported.

Go to primary source ↓
e9PDF pp. 5–6, Section 4.2 RoboTwin 2.0; p. 7, Table 2Inspect

Evaluation covers 50 tasks in clean/random settings. GeoSem-WAM reports 92.94/92.14/92.52 and Fast-WAM 91.88/91.78/91.80; both are marked without embodied pretraining. LingBot-VA reports 92.90/91.50/92.20 with embodied pretraining. Motus clean success is 88.66.

Go to primary source ↓
e10PDF pp. 6–7, Section 4.3; p. 7, Table 3, all rows and delta columnInspect

RGB-only, geometry addition, semantic addition and both report 97.6, 98.2, 98.1 and 98.6 average success. Printed delta percentages are +0.61, +0.51 and +1.02. No uncertainty or matched-budget specification is supplied.

Go to primary source ↓
e11PDF p. 7, Section 4.4; p. 8, Figure 4 and captionInspect

Franka uses four core tasks, 50 teleoperated trajectories per task, idle removal/action smoothing, two H800 GPUs for fine-tuning and RTX 4090 deployment. Generalization varies two mat backgrounds and standard/elevated setups differing by 4 cm. Figure 4 labels the standard panel Easy-Pick-D and the elevated panel Depth Setting, leaving table mapping imperfectly explained.

Go to primary source ↓
e12PDF p. 8, Table 4, caption, both model rows and Improvement rowInspect

Seven scenarios each have 50 independent trials. Fast-WAM: 100/86/86/80/92/90/88, average 88.9. GeoSem-WAM: 100/96/94/92/98/96/92, average 95.4. Printed improvement: 0/+10/+8/+12/+6/+6/+4, average +6.6.

Go to primary source ↓
e13PDF p. 8, Section 5 Conclusion, limitations and future workInspect

Dense annotations may be scarce and pseudo-labels noisy; heterogeneous losses complicate training and risk gradient conflicts. Foundation-model features and gradient surgery are proposed future directions.

Go to primary source ↓
e14PDF pp. 12–13, Appendix A, benchmark descriptions and Figure 5Inspect

Appendix A documents RGB/depth/semantic observations and clean/random RoboTwin settings. Figure 5 shows multiple LIBERO camera observations and dense labels. No explicit cross-view reconstruction objective or implementation schedule is supplied.

Go to primary source ↓
e15PDF p. 12, Table 5, Move Stapler Pad and Average rowsInspect

Move Stapler Pad scores are 73/63 for GeoSem-WAM and 77/64 for Fast-WAM. Average entries repeat GeoSem-WAM 92.94/92.14 and Fast-WAM 91.88/91.78. Motus is 88.52/87.02, differing in clean score from Table 2.

Go to primary source ↓
e16PDF p. 13, Appendix B; p. 14, Figure 6 and captionInspect

Pick-Pour is shown from third-person and first-person views with dense modality images. The first-person legend appears to interchange Depth/Sem labels relative to the third-person legend. No additional quantitative evaluation or controller specification is supplied.

Go to primary source ↓

8.5 Primary sources

Scroll across the image to inspect details. Press Esc to close.