Making Foresight Actionable: Repurposing Representation Alignment in World Action Models
1. Paper overview
In one sentence: AGRA aligns the video features read by a separate action decoder with frozen semantic targets, improving manipulation while retaining multi-depth predictive guidance. e-gape-aligne-architecturee-reale-pass-ablation
| At a glance | What to know |
|---|---|
| Research problem | Source description Plausible predicted motion can coexist with failed physical grasps. The baseline's action attention misses the hand-object contact region, and perturbing background video tokens changes its predicted actions. The paper asks whether reconstruction-trained features expose the spatial structure needed for continuous control, rather than treating visual plausibility as sufficient evidence of an effective policy. e-gape-pca |
| Core mechanism | Source description Diagnoses the video-action interface using complementary attention maps and hidden-state interventions, separating where the decoder attends from which perturbations change its output. e-gape-sensitivity |
| A key reported result | Pick-and-Place (ID): AGRA-DinoL8: 80% Physical task success rate. IRON-R01-1.11; ten seen objects × five positions, 50 trials; instructed-goal completion. WAM: 34%; +46 percentage points. A substantial execution gain under the stated physical protocol; uncertainty across training runs is not supplied. e-reale-config |
| Reading caution | Source description Physical evaluation covers only two task families. The authors acknowledge that richer semantic tasks could change encoder rankings and that AGRA only partially addresses the video-action mismatch. e-limits |
Core contributions
- Source description
Diagnoses the video-action interface using complementary attention maps and hidden-state interventions, separating where the decoder attends from which perturbations change its output. e-gape-sensitivity
- Source description
Adds spatially matched semantic alignment to the intermediate world-model features while retaining video and action flow-matching objectives and multi-depth guidance. e-aligne-architecture
- Author claim
Demonstrates physical ID/OOD improvements, simulation comparisons and training-pass ablations; the proposed explanation is improved action grounding. e-reale-sime-pass-ablation
Figure 2. A plausible future leaves the baseline decoder distracted by irrelevant regions. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Begin with the left sequence: the predicted hand approaches the banana, while the lower row shows where the action head reads video features. The red box marks the interaction region, but attention also collects around the other hand and background. On the right, distinguish ordinary images from the Zero and Mean Intervention maps. These maps measure how much the predicted action changes when a spatial hidden-state token is replaced; brighter colors indicate larger change after min-max normalization. Section 3.2 specifies averaging attention over action tokens and heads and displaying intervention sensitivity on the last latent frame. e-gap
What it supports. The examples motivate two different diagnostics. Attention identifies accessed regions; token replacement tests whether the output depends on those representations. Both reveal distraction in this baseline example, supporting the paper's decision to regularize the world-action interface rather than judging control readiness solely from the generated scene.
Where the evidence stops. These are selected, normalized visualizations without an absolute heatmap scale. Token replacement changes an internal representation, not the physical scene, and the figure alone does not measure how often the failure occurs.
2. Motivation
2.1 The problem and the proposed response
Plausible predicted motion can coexist with failed physical grasps. The baseline's action attention misses the hand-object contact region, and perturbing background video tokens changes its predicted actions. The paper asks whether reconstruction-trained features expose the spatial structure needed for continuous control, rather than treating visual plausibility as sufficient evidence of an effective policy. e-gape-pca
2.2 What this reading follows
A video model can depict a hand approaching the right object while the robot policy still misses its grasp. AGRA studies the interface between those two abilities. Its intervention is a training loss: make selected video-model features resemble spatially coherent DINOv2 features, then let a separate action decoder read a hierarchy of video features. The resulting physical improvements are large, but the paper's most instructive evidence is the combination of execution tests, token interventions and training-pass ablations. Read them together: successful videos, localized attention and successful actions are different observations, and none should silently substitute for the others. e-gape-aligne-architecturee-reale-pass-ablation
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | WAMs |
| Architecture | Dual-system |
| Prediction paradigm | IDM |
| Quadrant | Q4 · Dual-system × IDM |
3.1 Evidence-based assessment
Supports the recorded classification
Distinct video and action DiTs with an explicit feature bridge support Dual-system. Mapping implicit future representations to actions supports the catalog's IDM interpretation broadly, although this is not an explicit inverse model consuming fully generated before/after images. Joint flow-matching training alone does not make it One Model. Alignment is a training regularizer; inference still uses predictive world features. e-architecturee-conditioninge-align
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 1. Separate predicting a scene from reading it for action
The policy starts from an observation, an instruction and robot state. Its video model has learned to predict future visual latents, but an action decoder must still extract the information needed for a grasp. AGRA's diagnosis concerns that extraction step. A plausible banana-reaching video is compatible with the action head reading the wrong hand or background. The attention visualization measures where the decoder places weight, while token interventions measure how perturbing a representation changes its action output. Those are complementary observations. As a reader's interpretation, the useful lesson is to test the interface directly: visual quality cannot substitute for physical evaluation, and an attractive attention map cannot substitute for an intervention. The paper therefore connects these diagnostics to real execution tests, while its qualitative rollout figures remain selected examples. e-architecturee-gape-attentione-sensitivitye-reale-execution
Figure 7. Semantic alignment supervises an intermediate feature field while the action decoder keeps its multi-layer video bridge. Original paper, p. 13 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow the central blue branch downward from instruction, observation and noise. Horizontal arrows carry video features into the pink action branch, which also receives robot state and action noise. The leftward branch sends selected video features through an MLP toward the foundation-encoder target; its direction represents feature alignment, rather than executed actions. Section 3.3 establishes that the visual encoder is frozen. Although the drawing uses adjacent block labels, Appendix C.1 specifies bridge layers 0, 4, 8, 12, 15, 19, 23 and 27, with alignment at layer 8. e-architecturee-aligne-confige-conditioninge-real-variants
What it supports. The design separates semantic regularization from predictive-feature access. Only a selected layer is directly aligned by default, while the action head continues reading several depths. This distinction explains why repeatedly feeding the aligned layer to every action block is a materially different ablation, rather than the default AGRA architecture.
Where the evidence stops. The drawing shows a predicted-future output, but Appendix A.2 specifies one high-noise video pass at deployment, followed by action denoising. It also omits the two training passes; use Appendix A.3 and B.2.2 for their roles.
5.2 2. Align the high-noise features that deployment will consume
Training has two uses for the video model. One pass samples a video noise level and learns the full velocity-prediction objective. A separate pass fixes the future latents at noise level one and supplies features to the action head, matching deployment. The default AGRA variant applies semantic alignment to this second pass. Its targets come from reference video frames: a frozen DINOv2 encoder produces patches, interpolation matches the video-token grid, and a projector puts video features into the same feature space. Negative cosine similarity adds a weak semantic constraint alongside the video and action losses. This clarifies an easily missed point in the architecture drawing: DINOv2 is a training target, not a sensor for unknown future frames during execution. At deployment, one video pass supplies guidance for four action-denoising steps. e-aligne-conditioninge-confige-architecturee-training
5.3 3. Keep a hierarchy after aligning one layer
Aligning layer 8 does not mean discarding the rest of Cosmos. The action head reads eight depths through its bridge. BridgeL8 deliberately removes that diversity by repeating the same aligned feature at every cross-attention block, and its poor physical performance shows that alignment alone is insufficient in this construction. The authors interpret shallower alignment as organizing semantic structure while leaving deeper layers useful for geometry and motion. The layer sweep and appendix attention visualizations support that interpretation within the tested model, but do not establish a universal division of labor. The strongest next comparison keeps the bridge intact and moves only the loss between VideoPass and ActionCondPass. Their simulation results favor aligning the consumed features, while the shared-pass appearance exception warns against treating train-inference consistency as a guarantee of dominance. e-confige-real-variantse-layere-extra-analysise-pass-ablation
5.4 Training and inference
During training
First adapt Cosmos with video-only training for approximately 4,000–5,000 steps. Then jointly train video and action branches: video denoising uses sampled noise levels, while action conditioning uses a separate video forward pass at noise level one. Action gradients can update the video backbone. e-traininge-conditioninge-pass-ablation
Physical pretraining combines 40,000 factory robot trajectories (92 hours) and 37,000 EgoDex trajectories (42 hours) for 60,000 joint steps. Laboratory adaptation uses 560 Pick-and-Place and 180 steamer demonstrations for 2,000 steps, freezing Cosmos and updating only the action head. e-datae-training
The batch size is 256 with a cosine schedule and 5% warm-up. Joint physical-training learning rates are 10^-5 for Cosmos and 10^-4 for the action head; video/action/alignment loss weights are 1/1/0.01. Training uses 32 GPUs, each with 140GB memory. e-training
During inference
Condition Cosmos on the current observation and instruction, supplying pure noise for future latents. Extract multi-layer features from one pass at noise level one, then run four action flow-matching steps. Physical action chunks have horizon 48; simulation uses 16. The report does not infer a replanning cadence from these horizons. e-architecturee-confige-training
The predicted-future panels illustrate the video model's capability. Deployment consumes its high-noise hidden states, so the architecture figure does not imply a complete future-video rollout before every control chunk. e-architecturee-config
5.5 Implementation flow
- Build the predictive branch
Cosmos-Predict-2.5-2B processes 17-frame, 192×336 clips compressed to five latent frames, with the first frame conditioning the future. The 28-layer video DiT predicts flow velocities and exposes its intermediate token features. e-architecturee-config
- Bridge multiple depths into control
Features from layers 0, 4, 8, 12, 15, 19, 23 and 27 pass through normalization and a four-layer transformer projection module. Each guides one cross-attention block in the eight-block, approximately 500M-parameter action DiT; action/state tokens supply queries and video features supply keys and values. e-confige-architecture
- Construct alignment targets
For each latent frame, take its first corresponding RGB frame, extract frozen DINOv2 patch features, interpolate to the video-token grid and concatenate across time. Project the selected video hidden states into this semantic feature space; default alignment uses Cosmos layer 8. e-aligne-config
- Regularize the features consumed by actions
Minimize average negative cosine similarity to semantic targets. The default applies alignment to the separate high-noise action-conditioning pass. DINOv2 provides training targets; it does not replace the multi-layer bridge or generate control commands. e-aligne-conditioning
6. Experiments & results
AGRA improves a video-conditioned robot policy by aligning intermediate video features with frozen DINOv2 features during training. A multi-layer bridge preserves predictive information for action decoding. Physical Pick-and-Place success rises from 34% to 80%; controlled simulation gains are smaller. The evidence supports a useful interface regularizer, with limited task coverage and incomplete causal characterization.
6.1 Read the original evidence
Figure 4. Execution gains coexist with improved attention localization, but the variants do not have a universal ranking. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the legend before comparing bars: hatched variants exclude EgoDex, while the solid purple-blue AGRA-DinoL8 is the default. Panel (a) is physical ID Pick-and-Place. Panel (b) separates semantic selection, unseen object instances and the steamer task under attribute changes; these are not repeated measurements of one task. The success axes use fractions, while labels above bars are percentages. Panel (c) measures attention on a manually annotated subset: higher mask mass and lower centroid distance are favorable. Appendix C.2.2 supplies 50 ID, 40 semantic, 50 instance and 40 attribute trials and defines success by completing the instruction. e-reale-real-variantse-attentione-human
What it supports. Default AGRA improves ID success from 34% to 80%. Its attention-in-mask ratio increases from 0.238 to 0.267 and centroid error decreases from 3.57 to 3.26 grid units. Layer-15 alignment performs poorly on ID, and the repeated-layer bridge reaches zero on the steamer attribute task, supporting the importance of alignment placement and retained feature diversity.
Where the evidence stops. The plot supplies integer percentage labels, not unrounded counts or uncertainty. Preserve 82 and 48 as plotted. Multi-layer alignment also reaches 82 on attributes versus the default's 80, so layer 8 alone does not dominate every condition.
Figure 9. AGRA leads the reported simulation totals, under a protocol distinct from the physical experiments. Original paper, p. 16 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use the bottom legend to follow one method across the two task categories and the overall summary. Appendix B.1 defines 18 Pick & Place and six articulated tasks, evaluated over 50 episodes per task. This figure uses the full-data regime: 24,000 trajectories and 80,000 training steps. The salmon AGRA bars show both category performance and the aggregate; the neighboring baseline bars compare policy families with different pretrained representations. Notice the vertical axis begins at 20%, so bar-height ratios exaggerate relative performance differences. Read the numeric labels and percentage-point differences instead. e-sime-sim-protocol
What it supports. AGRA reports 68.8% on rearrangement, 59.0% on articulated tasks and 66.4% overall. Its reported overall margin is 11.0 percentage points over LDA-1B and 18.8 points over GR00T N1.6. These results broaden evaluation to simulated task diversity, while the alignment-specific causal comparison belongs to the separate few-shot ablation.
Where the evidence stops. Figure 9 contains an unexplained aggregation discrepancy: UWM's overall 29.5% is below both category values, 30.1% and 38.4%. Preserve the published bars without reconciling them. The comparison also lacks uncertainty and does not isolate alignment from pretraining differences.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Pick-and-Place (ID) IRON-R01-1.11; ten seen objects × five positions, 50 trials; instructed-goal completion. | AGRA-DinoL8: 80% Physical task success rate | WAM: 34%; +46 percentage points. A substantial execution gain under the stated physical protocol; uncertainty across training runs is not supplied. e-reale-config |
| Pick-and-Place (semantic generalization) 40 physical trials; target-object/container instructions are repeated after swapping target positions. | AGRA: 82% as labeled in Figure 4. Physical task success rate | WAM: 55%; plotted difference 27 percentage points. Tests instruction-conditioned selection alongside grasping. Integer plot labels do not justify reconstructing exact success counts. e-real |
| Pick-and-Place (unseen instances) Ten novel objects × five positions, 50 physical trials. | AGRA: 62% Physical task success rate | WAM: 30%; +32 percentage points. Evidence for transfer to the selected unseen objects, not unrestricted object generalization. e-real |
| Open-Steamer-Transfer-Bun (attribute generalization) 40 physical trials; four plate settings and one tablecloth setting, each with eight varied buns. | AGRA: 80% Physical task success rate | WAM: 48% as plotted; difference 32 percentage points. Requires removing the lid and transferring the bun; differs in task as well as shift type from the other regimes. e-reale-data |
| Action-grounding diagnostics Manually annotated interaction-mask subset; model attention and hidden-state interventions. | AGRA: 0.267; 3.26 grid units; mean-intervention R = 10.31. Attention mass ratio; centroid error; matched sensitivity ratio | WAM: 0.238; 3.57 grid units; R = 8.41. Shuffle R is 1.01 versus 0.99. Supports improved localization and relative action sensitivity, with little separation for shuffle; these are not execution-success metrics. e-attentione-sensitivity |
| RoboCasa GR1 tabletop (full data) 24 simulation tasks, 50 episodes each; 24,000 training trajectories and 80,000 steps. | AGRA: 66.4% Overall simulated success rate | LDA-1B: 55.4%; FLARE: 55.0%; GR00T N1.6: 47.6%. Reported margins are 11.0, 11.4 and 18.8 percentage points; this multi-method comparison does not isolate alignment alone. e-sim-protocole-sim |
| RoboCasa GR1 tabletop (few-shot alignment-pass ablation) 2,400 trajectories, 40,000 steps; separate ID and OOD suites. | ActionCondPass: 61.75 / 56.55 / 45.31 / 66.28. Success percentage, ID / unseen appearance / object types / combinations | WAM: 58.41 / 53.77 / 43.18 / 59.57; SharedDenoisingPass leads appearance at 57.77. Default alignment improves all columns over WAM, but does not beat every variant on every shift. e-sim-protocole-pass-ablation |
| Real-world feature-interface ablation Figure 4 physical ID and steamer attribute protocols. | Default layer-8 alignment: 80% ID; repeated-layer BridgeL8: 0% steamer attribute. Task success rate | ID: layer-15 alignment 22%, multi-layer alignment 72%, SigLIP alignment 60%, BridgeL8 28%. Alignment location and multi-depth access matter here; the experiment does not prove a universal optimal depth. e-real-variantse-real |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 1. Relative sensitivity shifts toward interaction tokens, with almost no separation under shuffle. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Each row changes how video hidden states are perturbed; the columns compare the baseline and aligned policy. Section 4.2.1 defines an interaction-token set and a background-token set, computes action deviations for each, and divides the former by the latter. Thus a value above one means the interaction perturbation changes the predicted action more than the background perturbation. Read across each row rather than comparing raw values between intervention types, because zeroing, replacing by a mean, shuffling and swapping with complement tokens impose different changes. This diagnostic measures the model's action output, not physical trial completion. e-sensitivitye-gap
What it supports. AGRA's mean-intervention ratio is 10.31 versus 8.41 for WAM; zero replacement gives 2.82 versus 1.95, and swapping gives 10.19 versus 8.36. All reported ratios rise, but shuffle moves only from 0.99 to 1.01. That weak separation limits any claim that action grounding improves equally under all interventions.
Where the evidence stops. A larger ratio can result from a larger numerator, a smaller denominator, or both. The table does not report absolute deviations, confidence intervals, or enough matching details to establish reduced background sensitivity by itself.
Table 2. Aligning the action-conditioning pass leads three columns, with a clear unseen-appearance exception. Original paper, p. 17 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Begin with WAM as the jointly trained baseline without alignment. SharedDenoisingPass removes the separate fixed-noise conditioning pass, so the action head consumes features from the randomly sampled video-denoising timestep during training. VideoPass retains the fixed-noise action branch but attaches alignment to the random-timestep video pass. ActionCondPass aligns the fixed-noise features actually read by the action head. This last pair most directly tests placement of the semantic loss. The four columns are different evaluation distributions, and Appendix B.1 places this table in the 2,400-trajectory, 40,000-step regime rather than the full-data comparison. e-pass-ablatione-conditioninge-sim-protocol
What it supports. ActionCondPass improves ID from 58.41 to 61.75 and unseen combinations from 59.57 to 66.28 versus WAM. It also leads unseen object types. However, SharedDenoisingPass scores 57.77 on unseen appearance versus 56.55 for ActionCondPass. The table supports targeted alignment as generally effective under this setup, not universally best.
Where the evidence stops. SharedDenoisingPass changes the action-conditioning distribution as well as the computation schedule, so it is not a pure alignment-location control. No seed variation or uncertainty is supplied; the apparent margins require repeated runs before a statistical claim.
7. Analysis & limitations
7.1 What the evidence leaves open
Physical evaluation covers only two task families. The authors acknowledge that richer semantic tasks could change encoder rankings and that AGRA only partially addresses the video-action mismatch. e-limits
Attention shifts and relative intervention ratios support the proposed mechanism but do not establish that reduced absolute background dependence causes the success gain. Confidence intervals, training-seed variation and diagnostic subset size are absent. e-attentione-sensitivitye-reale-pass-ablation
Human data helps AGRA more strongly in the plotted OOD settings, but its effect is not uniformly positive for WAM: ID falls from 42% without EgoDex to 34% with it. The simulation discussion also identifies post-grasp slippage beyond visual localization. e-humane-layer
Figure 9 reports UWM overall success of 29.5%, below both displayed category rates, 30.1% and 38.4%. The aggregation is unexplained; comparative totals are retained as reported rather than reconstructed from category means. e-sime-sim-protocol
7.2 Questions for discussion
- Would matched absolute background sensitivity predict execution robustness better than the ratio alone?
- Does layer 8 remain preferable when the task demands richer semantics or more complex contact dynamics?
8. Reproducibility audit
8.1 Requirements and known gaps
A faithful reconstruction needs the pretrained video model, DINOv2, dual-DiT bridge, robot/human demonstrations and the staged training schedule. Preserve physical 48-step versus simulated 16-step horizons, including every-third-frame physical video sampling. e-confige-datae-training
The supplied implementation description does not identify the exact DINOv2 checkpoint, GPU model, software versions, action-noise Beta parameters, human-action handling, or physical control/replanning cadence. These remain implementation questions, not defaults inferred from related systems. e-architecturee-confige-datae-training
Proposed checks should compare alignment placements with matched initialization/data and test interaction versus background interventions using both absolute deviations and ratios. They are experiments to perform, not reproduced results. e-pass-ablatione-sensitivity
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Move only the alignment loss, then test each shift separately
Reader-proposed experiment: reproduce the few-shot RoboCasa regime with the same 2,400 trajectories, video-adapted initialization, 40,000 training steps, bridge layers, optimizer schedule and action sampler. Compare WAM, AGRA-VideoPass and AGRA-ActionCondPass; keep the separate fixed-noise action-conditioning pass in all three, so only the semantic loss and its location differ. Use several matched random seeds and the same evaluation episodes. Report ID and each OOD suite separately with uncertainty, training compute and failure categories. The discriminating observation is whether ActionCondPass repeatedly improves unseen combinations and object types over VideoPass. Failure to repeat that ordering would weaken the claim that aligning the exact consumed feature distribution is the operative advantage. Add SharedDenoisingPass only as a separate conditioning-distribution comparison. e-sim-protocole-pass-ablatione-conditioninge-traininge-config
Check 2: Decompose the sensitivity ratio and connect it to physical failures
Reader-proposed experiment: evaluate WAM and AGRA on the same held-out scenes with independently annotated interaction masks. Hold the observation, instruction, robot state and sampling noise fixed while intervening. Match interaction and background token counts, stratify perturbation magnitudes, and repeat the paper's mean, zero, shuffle and complement-swap interventions. Report both absolute action deviations and their ratio, including denominator distributions and uncertainty. Then use paired physical trials that change background appearance while preserving object placement and instruction, scoring localization, grasp and completion separately. The grounding explanation gains support if lower absolute background sensitivity accompanies better paired execution. If the ratio rises only because interaction perturbations have larger effects, the stronger claim of reduced background dependence is not established. e-gape-sensitivitye-attentione-reale-layer
8.3 Reading coverage
Visual audit: Visually inspected the title/version/author page, all 16 original figures, both tables, method equations, training and inference descriptions, physical evaluation protocol and simulation appendices. Pages 18–19 and 21 supply implementation, data, hardware and temporal-pairing details. All six final crops were viewed, including the corrected Figure 4 crop retaining panel (b). Architecture arrows were checked against Sections 3.1–3.3 and Appendix A: cross-attention carries video features into the action branch, while the encoder/projector branch defines training alignment. The diagram's adjacent-block labels are schematic; Appendix C.1 specifies the actual sampled layers. Figure 7 does not depict the full two-pass training schedule or the single-pass deployment schedule. Figure 9's UWM aggregation discrepancy and Table 2's unseen-appearance exception are disclosed. References on pages 9–12 were read as text; their page images and external supplemental videos are outside this visual pass.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Title, authors, affiliations, arXiv version and abstract (p. 1)
- Sections 1–2: Introduction and Related Work (pp. 2–3)
- Sections 3.1–3.3: baseline, diagnosis and alignment objective (pp. 3–5)
- Sections 4.1–4.2.2: setup, results and further analysis (pp. 6–8)
- Section 5: Conclusion and Limitation (p. 8)
- References (pp. 9–12)
- Appendix A.1–A.3: architecture, sampling and optimization (pp. 13–15)
- Appendix B.1–B.2.2: simulation protocol, comparisons and ablations (pp. 15–18)
- Appendix C.1–C.3: implementation, collection, evaluation and training (pp. 18–19, 21)
- Appendix D.1–D.3: additional execution, layer and encoder analysis (pp. 20–23)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout. This acquisition limitation was addressed by visually inspecting all scientific figure/table pages and the method/evaluation pages.
- Separate supplemental material availability has not been fully verified.
- The supplied PDF is arXiv:2606.12217v1, dated 10 June 2026. Its title and all six authors agree with the catalog; no other revision was supplied or compared.
- All seven supplied text chunks were read individually. Reference pages 9–12 were read as text; their page images were not inspected.
- Project-page content, code, datasets and external demonstration videos were not inspected. No experiments were run and no implementation or release availability is asserted.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e-identityPDF p. 1, title block, affiliation line and arXiv margin
The title and six authors match the catalog. The supplied artifact identifies itself as arXiv:2606.12217v1 [cs.CV], 10 June 2026. Affiliations are The University of Hong Kong and XPENG Robotics.
Go to primary source ↓e-gapPDF p. 4, Section 3.2 and Figure 2
Attention is averaged over action tokens and heads. Zero/mean replacement of spatial video tokens is evaluated through Euclidean action deviation; min-max-normalized heatmaps show the last latent frame and reveal background sensitivity in the illustrated baseline.
Go to primary source ↓e-pcaPDF pp. 4–5, Figure 3 and Section 3.3 introduction
Per-model PCA fitted across multiple samples maps three components to RGB. The authors interpret DINOv2's smoother spatial organization as a useful target for Cosmos features.
Go to primary source ↓e-alignPDF p. 5, Section 3.3, Eqs. (3)–(7)
Semantic targets use the first RGB frame corresponding to each latent frame, frozen DINOv2 features, spatial interpolation and temporal concatenation. Projected video features are aligned by average negative cosine similarity; alignment supplements the baseline objective.
Go to primary source ↓e-architecturePDF p. 3, Section 3.1, Eqs. (1)–(2); pp. 13–14, Figure 7 and Appendix A.1–A.2, Eqs. (8)–(17)
Separate video and action DiTs communicate through projected multi-layer video features used as cross-attention keys/values. Both use flow matching. Inference evaluates the video model once at noise level one, followed by iterative action denoising.
Go to primary source ↓e-conditioningPDF p. 15, Appendix A.3, Eqs. (18)–(20); pp. 17–18, Appendix B.2.2, ActionCondPass definition and targeted application
Video training uses a sampled-noise pass; action conditioning uses a separate pass at fixed noise level one. Default ActionCondPass alignment targets the hidden states actually consumed by the action head.
Go to primary source ↓e-configPDF p. 6, Section 4.1; p. 18, Appendix C.1
Cosmos-Predict-2.5-2B has 28 layers; the action DiT has eight blocks and approximately 500M parameters. Clips contain 17 frames at 192×336, encoded into five latent frames. Layers 0, 4, 8, 12, 15, 19, 23, 27 supply the bridge. DINOv2 input is 448×448; inference uses four action denoising steps.
Go to primary source ↓e-realPDF p. 6, Figure 4(a)–(b), WAM and AGRA-DinoL8 bars; p. 19, Appendix C.2.2
Plotted WAM/AGRA success percentages are 34/80 ID, 55/82 semantic, 30/62 instance-level and 48/80 attribute. Protocols use 50, 40, 50 and 40 physical trials respectively, with success requiring completion of the instructed goal. Integer labels do not provide unrounded trial counts.
Go to primary source ↓e-real-variantsPDF p. 6, Figure 4(a)–(b) and compared variants; pp. 7–8, Section 4.2.2(a)–(c)
ID bars show DinoL8 80, DinoL15 22, DinoL4/8/12 72, SiglipL8 60 and BridgeL8 28. BridgeL8 reaches zero in the attribute steamer task; multi-layer alignment reaches 82 there versus default 80. BridgeL8 reuses aligned layer 8 for every action cross-attention block.
Go to primary source ↓e-humanPDF p. 6, Figure 4, hatched no-EgoDex versus solid bars; p. 8, Section 4.2.2(d)
Without/with EgoDex, AGRA's plotted ID, semantic, instance and attribute percentages are 74/80, 55/82, 28/62 and 32/80. WAM values are 42/34, 50/55, 30/30 and 30/48. Human-data effects differ by model and evaluation setting.
Go to primary source ↓e-attentionPDF p. 6, Figure 4(c) and Section 4.2.1; p. 7, continuation and Figure 5
On a manually annotated evaluation subset, attention mass inside hand-object masks rises from 0.238 to 0.267 and centroid error falls from 3.57 to 3.26 grid units. The subset size and uncertainty are not reported here.
Go to primary source ↓e-sensitivityPDF p. 7, Table 1 and Section 4.2.1, causal intervention paragraph
Matched Sensitivity Ratio divides action deviation from interaction-token perturbations by background-token perturbations. WAM/AGRA values are Mean 8.41/10.31, Zero 1.95/2.82, Shuffle 0.99/1.01 and Swap with complement 8.36/10.19.
Go to primary source ↓e-sim-protocolPDF pp. 15–16, Appendix B.1
RoboCasa GR1 evaluation has 24 tasks, comprising 18 rearrangement and six articulated tasks, with 50 episodes per task and 29 joint-space DoF. Full data means 24,000 trajectories/80,000 training steps; few-shot means 2,400 trajectories/40,000 steps. OOD suites contain 18 appearance, 32 object-type and 14 combination tasks.
Go to primary source ↓e-simPDF p. 16, Figure 9 and Appendix B.2.1
Full-data overall success is AGRA 66.4%, LDA-1B 55.4%, FLARE 55.0%, DiT4DiT 50.8%, GR00T N1.6 47.6%, Diffusion Policy 40.9% and UWM 29.5%. AGRA scores 68.8% on Pick & Place and 59.0% on articulated tasks. The vertical axis begins at 20%. UWM's displayed category values are 30.1% and 38.4%.
Go to primary source ↓e-pass-ablationPDF p. 17, Table 2 and Appendix B.2.2; p. 18, continuation
In ID/appearance/object-type/combination order, WAM scores 58.41/53.77/43.18/59.57, VideoPass 60.41/55.33/44.56/60.14 and ActionCondPass 61.75/56.55/45.31/66.28. SharedDenoisingPass scores 58.24/57.77/41.87/58.57, leading the appearance column. Definitions distinguish changing action-conditioning distribution from changing alignment placement.
Go to primary source ↓e-layerPDF p. 16, Figures 10–11; pp. 17–18, Appendix B.2.2
The simulation layer sweep peaks at layer 8. The displayed single-layer bridges trail the multi-layer bridge. The authors attribute smaller simulation improvements partly to post-grasp slippage that visual improvements alone cannot solve.
Go to primary source ↓e-dataPDF pp. 18–19, Appendix C.2.1
Pretraining uses 40,000 factory robot trajectories (92 hours) and 37,000 EgoDex human trajectories (42 hours). Laboratory data comprise 560 Pick-and-Place trajectories (1.5 hours) and 180 steamer-task trajectories.
Go to primary source ↓e-trainingPDF p. 19, Appendix C.3; p. 21, continuation of C.3
Training starts with approximately 4,000–5,000 video-only adaptation steps. Joint real-world pretraining lasts 60,000 steps, followed by 2,000 action-head-only fine-tuning steps with Cosmos frozen. Batch size is 256, warm-up 5%, and training uses 32 GPUs with 140GB each. Loss weights are 1/1/0.01; real-world learning rates are 1e-5 for Cosmos and 1e-4 for the action head. Action horizons are 48 physical and 16 simulation steps; physical video targets sample every third frame. GPU model, software versions and precise execution/replanning cadence are not supplied.
Go to primary source ↓e-executionPDF pp. 7, 20–21, Figures 6, 12–13 and Appendix D.1
Selected physical execution sequences illustrate successful AGRA rollouts and baseline spatially biased grasps. These are qualitative cases rather than an independent estimate of success frequency.
Go to primary source ↓e-extra-analysisPDF pp. 21–23, Appendix D.2–D.3 and Figures 14–16
The authors interpret text-to-video attention at layer 8 as clearer object grounding, action attention across bridge depths as progressively local, and DINOv2 PCA structure as more spatially coherent than SigLIP.
Go to primary source ↓e-limitsPDF p. 8, Section 5, Limitation
The authors acknowledge limited physical task diversity, possible benefits of alternative encoders on richer semantic tasks, and incomplete resolution and quantification of the video-action mismatch.
Go to primary source ↓8.5 Primary sources
Making Foresight Actionable: Repurposing Representation Alignment in World Action Models ↗
PDF · 10,910 extracted words
Source fingerprint
8fe740c3e5496a1174636e496410b954e5b63928181f5906f40c959123c72389