Artificial Foveated Perception for Mitigating Shortcut Learning in Robotic Foundation Models

English reading report: Method, equations, original figures, experiments and reproducibility.

Authors: Xiatao Sun; Yuan Zhuang; Mateo Sanchez Lopez Negrete; Matei-Victor Coldea; Chen Liang; Haoyang Zhang; Che Liu; Ziyao Zeng; Shawn Li; Qian Wang; Fei Miao; Daniel Rakita

Affiliations: Yale University; University of Connecticut; Peking University; Imperial College London; Digients

Source: 2607.10655 ↗ · Catalog record

Reading: 126 / 558 · 6 original figures & tables · ~20 min ·

1. Paper overview

In one sentence: AFP transfers task-relevance masks into a policy’s attention during fine-tuning, improving reported distractor robustness without an inference-time mask predictor, at the cost of extra annotation and training machinery. e01 e02 e03 e05 e06 e08 e10 e11 e13 e18

At a glance	What to know
Research problem	Reader analysis Action supervision can reward background correlations that predict demonstrations without identifying the objects and robot interactions needed for success. The authors frame this as shortcut learning and propose explicit visual grounding. Their experiments primarily perturb distractors while preserving task objectives, testing a narrower distribution shift than general robustness to arbitrary new environments. e02 e09 e12
Core mechanism	Source description A reusable task-conditioned mask predictor combines lightweight visual features, region queries and language modulation; its masks supervise existing policies without changing their inference architecture. e03 e04 e08
A key reported result	Physical Drawer Assembly under OOD distractors: AFP: 0.67 OOD and 0.83 ID. Success rate (higher is better). π0.5 on an i2RT YAM Arm; 300 demonstrations, 50k fine-tuning steps, 30 physical rollouts per method and setting; unused furniture parts added at evaluation. Direct fine-tuning: 0.30 OOD and 0.73 ID. Executed robot success improves; the rounded OOD rates differ by 37 percentage points. No confidence interval or seed variation is supplied. e12 e13
Reading caution	Reader analysis The authors identify annotation cost and bias, ambiguous relevance, heavy occlusion and mask-distribution shift as failure risks. Shared dataset families and the authors’ experimental scenes appear in mask training, but disjoint task/scene/trajectory splits are not documented sufficiently to assess overlap. e05 e09 e14 e19

Core contributions

Source description
A reusable task-conditioned mask predictor combines lightweight visual features, region queries and language modulation; its masks supervise existing policies without changing their inference architecture. e03 e04 e08
Source description
Attention-distribution matching is paired with an asymmetric PCGrad-style update that removes only the auxiliary gradient component conflicting with action learning. Experiments connect improved attention alignment to higher manipulation success. e06 e07 e10 e11 e13

AFP diagram: RGB frames feed MobileNetV3 and a dense feature path; learned region queries cross-attend to encoded features, receive temporal modeling and CLIP/FiLM language conditioning, and combine with dense features to produce a mask used for auxiliary policy training. — **Figure 2.** Dense visual detail and language-conditioned queries meet in a relevance mask that supervises policy training. Original paper, p. 3 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.

2. Motivation

2.1 The problem and the proposed response

Reader analysis

Action supervision can reward background correlations that predict demonstrations without identifying the objects and robot interactions needed for success. The authors frame this as shortcut learning and propose explicit visual grounding. Their experiments primarily perturb distractors while preserving task objectives, testing a narrower distribution shift than general robustness to arbitrary new environments. e02 e09 e12

2.2 What this reading follows

A robot can learn the right movements while looking at the wrong evidence. AFP addresses that possibility by teaching a separate model to identify task-relevant visual regions, then using its masks to supervise the robot policy’s attention. The deployed policy still sees the original images. Read this paper through three distinct outcomes: agreement with expert relevance masks, successful manipulation under added distractors, and progress per training step. The figures support improvements in all three, but they do not establish that attention is a causal explanation of action or that fewer updates necessarily mean less total training time. This edition examines the supplied arXiv v1. e01 e02 e03 e05 e06 e08 e10 e11 e13 e18

3. Research context

We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.

Catalog dimension	Recorded classification
Major category	Foundational work
Architecture	Not applicable
Prediction paradigm	Not applicable
Quadrant	Not applicable

This table preserves the labels recorded at reading time. The current major category is VLA. View the current classification.

3.1 Evidence-based assessment

Supports the recorded classification

Reader analysis

The recorded foundational-work classification fits a perception and training method usable across VLA/WAM policies. AFP predicts relevance masks, not joint futures/actions or inverse dynamics. Its auxiliary supervision does not define a new world/action architecture, supporting the recorded Not applicable architecture, prediction paradigm and quadrant. Compatibility with Motus does not make AFP itself a One Model WAM. e03 e04 e08 e11

This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.

4. Problem formulation

4.1 Inputs and outputs

Inputs	Outputs
RGB observation frames or sequences Natural-language task instruction Fine-tuning action demonstrations and AFP mask targets	AFP: continuous per-frame task-relevance masks Fine-tuned underlying policy: robot actions using the original observation stream

4.2 Equations and their role

\mathcal{L}_{\mathrm{AFP}}=\frac{1}{B}\sum_{b=1}^{B}\left\|\frac{\bar{a}_b}{\sum_i\bar{a}_b^{(i)}}-\frac{w_b}{\sum_i w_b^{(i)}}\right\|_2^2

Equation (5): B is batch size, b indexes examples, i indexes image tokens, averaged policy attention is the vector \(\bar a_b\), and the pooled AFP mask is \(w_b\). Normalization compares spatial distributions rather than absolute attention mass. e06

g_{\mathrm{AFP}}^{\mathrm{PC}}=\begin{cases}g_{\mathrm{AFP}}-\dfrac{g_{\mathrm{AFP}}^\top g_{\mathrm{act}}}{\|g_{\mathrm{act}}\|_2^2}g_{\mathrm{act}},&g_{\mathrm{AFP}}^\top g_{\mathrm{act}}<0,\\g_{\mathrm{AFP}},&\text{otherwise},\end{cases}\qquad g=g_{\mathrm{act}}+\lambda g_{\mathrm{AFP}}^{\mathrm{PC}}

Equation (6): the two gradients differentiate action loss and AFP loss with respect to policy parameters. A negative inner product triggers removal of the conflicting auxiliary component. The final update direction weights that projected component by \(\lambda\), whose numerical setting is not supplied. e07 e19

5. Method in detail

5.1 Build a relevance field before supervising a policy

Reader analysis

AFP starts with a labeling decision: which regions matter for the instruction? Appendix C’s tool uses positive and negative key-frame prompts to initialize SAM 2 masks, then propagates merged instances through the trajectory with MatAnyone. The output is continuous, allowing uncertain boundaries, partial occlusion and interaction regions to receive graded relevance. AFP learns from these image–instruction–mask examples. Its feature path preserves spatial detail, while learned region queries are decoded from visual memory and then modulated by language. This matters because object presence alone does not determine task relevance. A reader’s interpretation is that the annotation is a reusable prior on useful evidence, rather than a guarantee of causal sufficiency: the paper itself warns that ambiguous relevance and unseen mask distributions can defeat that prior. e03 e04 e05 e14 e16

5.2 Move attention without sacrificing the action objective

Reader analysis

During policy fine-tuning, the mask becomes a target over image tokens. Averaging attention across trainable layers, heads and query positions produces a comparable vector, and Equation (5) normalizes both vectors before measuring squared distance. Thus the objective specifies where image attention should go, not its absolute mass. A second issue appears when decreasing the grounding loss would increase action loss. Equation (6) tests the gradient inner product and, only when it is negative, projects the auxiliary component away from the action-gradient direction. The action gradient is left intact. As a local geometric interpretation, this removes that opposing component; it is not a guarantee of future task success. Figure 6 motivates the intervention empirically, while Section 2.4 confirms that the learned policy later executes without AFP. e06 e07 e08 e15

5.3 Require evidence to travel from grounding to behavior

Reader analysis

The first empirical link is diagnostic: attention becomes more similar to expert masks. Because this resembles the supervised training target, that finding alone cannot establish better manipulation. Table 1 supplies a second link by evaluating success under added simulation distractors, and Table 2 extends the comparison to physical rollouts with task-specific perturbations. The comparisons preserve the policy architecture and observation stream, helping locate the change in fine-tuning. Nevertheless, they bundle mask supervision with the projected update, and the supplied paper does not isolate controlled mask degradation despite promising mask-quality ablations. Figure 9 adds a third, separate claim about progress per update. A careful reading therefore accepts the reported behavioral improvements while keeping causal mask relevance, the contribution of gradient projection, and net training-time savings open to targeted reproduction checks. e01 e08 e09 e10 e11 e12 e13 e15 e18

5.4 Training and inference

During training

Source description

AFP is supervised on 380,574 annotated images from Open-X Embodiment, DROID, MimicGen, LIBERO and the authors’ scenes. Annotators supply key-frame positive/negative prompts; SAM 2 initializes instance masks and MatAnyone propagates continuous alpha masks. The paper does not specify AFP’s exact supervised loss or dataset split. e05 e16 e19

Source description

Policy fine-tuning uses action demonstrations plus AFP attention supervision. Simulation uses 500 trajectories per task and 20k updates; physical tasks use 300 episodes and 50k updates. Only CLIP is explicitly described as frozen; policy-specific trainable-module choices are not enumerated. e03 e09 e12 e19

During inference

Source description

AFP runs on fine-tuning data to generate supervision targets. After training, the original policy receives the original observation stream and executes actions without AFP calls. Section 2.4 specifies this evaluated integration more narrowly than Figure 1’s broad ‘Training / Inference’ label. No new action decoder, planner or feedback controller is specified. e01 e08

5.5 Implementation flow

Represent visual detail and candidate regions
MobileNetV3-Large supplies multiscale features, projected into shared-dimensional tokens for a deformable transformer encoder. Learned queries attend to encoded visual memory through a deformable decoder. VMFormer feature and query temporal modules encourage consistency across frames. e03
Condition relevance on language
A frozen CLIP text encoder produces a normalized instruction embedding. A gated residual FiLM adapter changes decoder queries using language-derived channel scales and shifts. Query–dense-feature inner products followed by a sigmoid produce the primary saliency query’s mask. This predicts relevance, not future observations or actions. e03 e04
Transfer grounding into the policy
Pool the AFP mask to image tokens. Average policy attention across trainable layers, heads and query positions; normalize both vectors and minimize their squared distance. The action objective remains primary through conflict-aware projection of the AFP gradient. e06 e07

6. Experiments & results

Artificial Foveated Perception (AFP) learns language-conditioned relevance masks and uses them to supervise a robot policy’s visual attention during fine-tuning. A protected auxiliary gradient improves reported distractor robustness across four simulation backbones and one physical robot policy. AFP is absent from the deployed control loop; annotation requirements and incomplete experimental specifications limit reproducibility.

Source and visual limitations

Reader analysis

The supplied v1 contains six suitable selected visuals, but no located controlled mask-quality ablation despite the abstract’s promise. The available mechanism ablation concerns gradient projection. Figure 9 reports success against training steps, with no wall-clock or compute table; the edition preserves that narrower efficiency evidence. e01 e15 e17 e18

6.1 Read the original evidence

Coffee-task RGB frames, expert task masks, direct-fine-tuning attention and AFP auxiliary-loss attention in ID and OOD rows; an adjacent table compares Soft-IoU and EMD for pi0.5. — **Figure 3.** The attention diagnostic shows much closer agreement with expert relevance masks after AFP supervision. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.

Complete two-block simulation table covering eight MimicGen tasks and four backbones, each with direct-fine-tuning and AFP rows and separate ID/OOD success-rate columns. — **Table 1.** Every listed simulation backbone improves its OOD success rate with AFP across the eight tasks. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.

Physical success table for pi0.5 with direct fine-tuning and AFP across Drawer, Lamp and Table Assembly, Cloth Folding and Closet Storage, with ID and OOD columns for each task. — **Table 2.** The physical experiment reports improved success for all five tasks in both settings. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.

Eight pi0.5 ID-success learning curves compare direct fine-tuning in blue with AFP in red up to 40k steps. An adjacent horizontal bar chart lists average episode lengths, from Square at 157 to Coffee Preparation at 678 steps. — **Figure 9.** Learning curves suggest earlier ID success on several tasks; the horizontal axis measures updates rather than elapsed time. Original paper, p. 16 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.

6.2 Results and evaluation conditions

Task & protocol	Reported result	Comparison & interpretation
Physical Drawer Assembly under OOD distractors π0.5 on an i2RT YAM Arm; 300 demonstrations, 50k fine-tuning steps, 30 physical rollouts per method and setting; unused furniture parts added at evaluation.	AFP: 0.67 OOD and 0.83 ID. Success rate (higher is better)	Direct fine-tuning: 0.30 OOD and 0.73 ID. Executed robot success improves; the rounded OOD rates differ by 37 percentage points. No confidence interval or seed variation is supplied. e12 e13
MimicGen Kitchen D1 under OOD distractors π0.5; 500 demonstration trajectories, 20k fine-tuning steps; randomized placement and materials for 3–5 unrelated RoboSuite mesh objects.	AFP: 0.59 OOD and 0.89 ID. Success rate (higher is better)	Direct fine-tuning: 0.22 OOD and 0.82 ID. A 37-percentage-point OOD increase still leaves substantial failure. Table 1 shows OOD gains for every listed task/backbone pair; success-rollout denominators and seed aggregation are not explicit. e09 e11
Attention grounding across eight MimicGen tasks π0.5; expert masks on 10 ID and 10 OOD rollouts per task-model pair; reported metrics average across eight tasks.	AFP: Soft-IoU 0.934 ID / 0.916 OOD; EMD 0.035 ID / 0.038 OOD. Soft-IoU ↑; Earth Mover’s Distance (EMD) ↓	Direct: Soft-IoU 0.170 / 0.159; EMD 4.389 / 4.086, respectively. Attention matches expert relevance much more closely. These metrics measure grounding, not task success or causal necessity; EMD units and implementation details are unspecified. e09 e10
Auxiliary-gradient conflict diagnostic Appendix B’s 20k-step fine-tuning diagnostic; task, backbone and repeat count are not identified in the figure or accompanying discussion.	Projection triggered 11,152 times over 20k steps; the projected run remains stable in the plotted range. Projection-trigger count and total-loss trajectory	Without projection, loss rises beyond the plotted vertical range after the onset marker. Supports managing gradient conflict in the illustrated run, without establishing stability for every configuration. The projected curve trends downward but is not literally monotonic despite the appendix wording. e15
ID convergence across eight MimicGen tasks Figure 9 compares π0.5 with and without AFP over training-step curves extending to 40k.	AFP curves reach higher success earlier on Kitchen and Coffee Preparation; episode lengths are 622 and 678 steps, respectively. Success versus training steps; average episode length	Direct fine-tuning converges more slowly in those panels; Coffee and Stack Three curves are much closer. Qualitative evidence for optimization efficiency. No exact speedup, wall-clock timing, hardware cost or reduced-demonstration comparison is reported. e18

6.3 Ablations and diagnostic examples

Read component removals and qualitative examples within their stated evaluation conditions.

Left: total loss versus training step, with a red projected-gradient curve remaining low and a blue unprojected curve rising beyond the axis range after an onset marker. Right: action/AFP gradient cosine similarity repeatedly crosses below zero. — **Figure 6.** Conflicting auxiliary gradients motivate projecting the AFP update before combining it with action learning. Original paper, p. 14 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.

7. Analysis & limitations

7.1 What the evidence leaves open

Reader analysis

The authors identify annotation cost and bias, ambiguous relevance, heavy occlusion and mask-distribution shift as failure risks. Shared dataset families and the authors’ experimental scenes appear in mask training, but disjoint task/scene/trajectory splits are not documented sufficiently to assess overlap. e05 e09 e14 e19

Reader analysis

Improved attention and success are consistent with reduced shortcut reliance, but attention alignment alone does not establish causal feature use. The abstract promises mask-quality ablations; the supplied experiments instead provide grounding comparisons, projection diagnostics and convergence curves, without a located controlled mask-quality study. e01 e10 e11 e13 e15 e17 e18

7.2 Questions for discussion

Would area-matched but task-irrelevant masks provide comparable regularization benefits?
Does AFP still improve control on tasks and scenes excluded from mask-predictor training?
How much end-to-end training time remains saved after mask preparation and gradient projection costs?

8. Reproducibility audit

8.1 Requirements and known gaps

Source description

Reproduction requires the mixed mask dataset, annotation pipeline, AFP checkpoint or training recipe, policy checkpoints and demonstration splits. Missing settings include mask-training loss, image resolution, query count, optimizer, learning rate, batch size, auxiliary weight, policy attention extraction/pooling details, hardware and software versions. The primary saliency query’s selection rule is also unspecified. e04 e05 e16 e19

Reader analysis

Reader-proposed checks: compare relevant masks against area-matched shuffled masks under identical optimization, and separately compare projected versus unprojected updates while measuring both grounding and held-out success. Include mask preparation and per-update cost in timing; these controls test mechanism and net efficiency without assuming the published gains reproduce. e06 e07 e09 e15 e18

8.2 Proposed reproduction checks

The following checks are proposals motivated by the paper. They have not been run as part of this reading.

Check 1: Separate task relevance from generic attention regularization

Proposed, not run: fine-tune π0.5 on Coffee D1 and Kitchen D1 using the reported 500-trajectory, 20k-update budget. Compare direct fine-tuning, AFP masks, and spatially shuffled masks matched for value distribution and area. Keep policy initialization, demonstrations, auxiliary weight and projection identical between auxiliary conditions; explicitly partition mask training by scene and trajectory. Across repeated seeds, measure held-out ID/OOD success alongside Soft-IoU using common distractor configurations. If shuffled masks match AFP’s success gains despite poorer task alignment, generic regularization remains a plausible explanation. A selective AFP advantage would support the relevance mechanism more directly than attention agreement alone. e05 e06 e07 e09 e10 e11 e14 e19

Check 2: Test projection’s behavioral benefit and full computational cost

Proposed, not run: hold a π0.5 Kitchen D1 mask set, training data and initialization fixed, then compare projected and unprojected auxiliary updates over a declared weight sweep, with direct fine-tuning as a reference. Record action loss, grounding loss, gradient cosine, projection frequency and unclipped total-loss traces; periodically evaluate identical held-out rollouts. Measure elapsed training time including attention extraction, gradient handling and mask preparation, reporting AFP pretraining cost separately. Projection should reduce conflicting-update instability without degrading held-out success. Failure to do so would weaken the stability mechanism; fewer updates without a lower end-to-end time would not reproduce a practical time-saving claim. e05 e06 e07 e09 e11 e15 e18 e19

8.3 Reading coverage

Visual audit: Inspected the original title/author/version block, Figures 1–9, Tables 1–2, equations, method/training text, physical-evaluation details and Appendices A–E on these pages. Each of the six final crops was separately viewed at its returned resolution. Figure 2 arrows were checked against Sections 2.2–2.4; the projection inequality against Equation (6). Figure 1’s broad integration label, Figure 6’s clipped curve and nonmonotonic fluctuations, and Figure 9’s training-step axis are disclosed. All pages supporting retained method, numerical, evaluation and reproduction claims are included. Reference-only pages 9–12 were read in the complete text but not rendered; no separate supplement or code was inspected.

PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 13, 14, 15, 16. Appendix coverage: reviewed.

Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.

Text reading scope & known omissions

Title, author block and abstract (p. 1–2)
1 Introduction (p. 2)
2 Method; 2.1 Overview; 2.2 Model Architecture (pp. 2–4)
2.3 Model Training; 2.4 Auxiliary Training Loss During Fine-tuning (p. 4)
3 Evaluation; 3.1 Empirical Diagnosis; 3.2 Simulation Experiments (pp. 5–7)
3.3 Real-World Experiments; 4 Discussion and Limitations (pp. 7–8)
References (pp. 9–12)
A Related Works; A.1 Robotic Foundation Models; A.2 Shortcut Learning (p. 13)
B Auxiliary Loss Projection for Stability (pp. 13–14)
C Labeling Tool (pp. 14–15)
D Additional Empirical Findings on Shortcut Learning (p. 15)
E Training Efficiency (p. 16)

Outside the original text pass

Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
Separate supplemental material availability has not been fully verified.
Identity notes: the inspected title page identifies arXiv:2607.10655v1 [cs.RO], 12 July 2026. Its exact title and all twelve authors match the catalog after converting surname-first catalog names to title-page order. No revision or edition discrepancy was observed; no other version was supplied or compared.
Acquisition omission preserved: text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout. This reading supplements all five complete text chunks with actual PDF-page and final-crop inspection.
Acquisition omission preserved: separate supplemental material availability has not been fully verified. No separate supplement was supplied.
All supplied text was read, including references and Appendices A–E. Visual inspection covered pp. 1–8 and 13–16; reference-only pp. 9–12 were read as text.
Code, released data and annotation software were not inspected or executed. No experiments were reproduced; release statements remain author claims.

The visual audit above records the subsequent illustrated pass.

8.4 Traceable evidence

e01PDF p. 1, title/author block, arXiv margin, abstract and Figure 1; p. 2, abstract continuationInspect

Exact supplied title; twelve credited authors from Xiatao Sun through Daniel Rakita; five named affiliations; arXiv:2607.10655v1 [cs.RO], 12 Jul 2026. Figure 1 uses a broad Training / Inference label. The abstract claims ablations over mask quality and grounding-loss design.

Go to primary source ↓

e02PDF p. 2, Section 1, paragraphs 2–4Inspect

The authors motivate auxiliary visual grounding by action supervision permitting spurious background, lighting and scene-context correlations during fine-tuning.

Go to primary source ↓

e03PDF p. 3, Figure 2 and Sections 2.1–2.2, Eqs. (1)–(2)Inspect

RGB frames and task language feed a relevance-mask predictor with MobileNetV3-Large multiscale features, deformable encoder/decoder, learned region queries, VMFormer temporal modules and frozen CLIP text encoding.

Go to primary source ↓

e04PDF p. 4, Section 2.2 continuation, Eqs. (3)–(4)Inspect

FiLM supplies residual, gated channel modulation from normalized text embeddings. Dense features and language-conditioned queries are scored by inner product; a sigmoid of the primary saliency query score produces the mask. The primary-query selection procedure is not specified here.

Go to primary source ↓

e05PDF p. 4, Section 2.3, both paragraphsInspect

Supervised AFP mask training uses 380,574 images from Open-X Embodiment, DROID, MimicGen, LIBERO and the authors’ scenes. Targets are continuous relevance masks. The authors state that they release the annotation tool and labeled dataset.

Go to primary source ↓

e06PDF p. 4, Section 2.4, attention averaging and Eq. (5)Inspect

Image-token attention is averaged over trainable layers, heads and query positions. The AFP loss is batch-mean squared L2 distance between sum-normalized averaged attention and pooled mask vectors.

Go to primary source ↓

e07PDF p. 4, Section 2.4, Eq. (6) and final update expressionInspect

When the AFP/action gradient inner product is negative, the AFP gradient’s action-opposing component is subtracted. The action gradient remains unchanged and the update combines it with lambda times the projected auxiliary gradient.

Go to primary source ↓

e08PDF p. 4, Section 2.4, final paragraph; p. 3, Figure 2 captionInspect

AFP masks are supervision targets produced on fine-tuning data. Deployed inference uses original observations without an AFP call, and the core policy architecture is unchanged.

Go to primary source ↓

e09PDF p. 5, Sections 3.1–3.2; p. 6, Figure 4Inspect

Eight MimicGen tasks use 500 demonstration trajectories each and 20k fine-tuning steps. OOD scenes add 3–5 unrelated RoboSuite meshes with random placement and materials. Grounding uses 10 ID and 10 OOD rollouts per task-model pair and expert masks, averaged across tasks. Section 3.2 reuses the protocol without separately specifying Table 1 success denominators.

Go to primary source ↓

e10PDF p. 5, Figure 3 metric table and Section 3.1 final paragraphsInspect

For pi0.5, direct/AFP Soft-IoU is 0.170/0.934 ID and 0.159/0.916 OOD; EMD is 4.389/0.035 ID and 4.086/0.038 OOD. The picture shows Coffee; the text specifies metrics averaged across eight tasks. EMD units and exact implementation are not provided.

Go to primary source ↓

e11PDF p. 6, Table 1, both task blocks, especially pi0.5 rows / Kitchen D1 columns; Section 3.2Inspect

SmolVLA, OpenVLA, pi0.5 and Motus all improve every listed OOD task with AFP. Kitchen pi0.5 direct/AFP rates are 0.82/0.89 ID and 0.22/0.59 OOD. Stack 3 OOD is 0.61/0.88; Threading OOD is 0.46/0.77. No uncertainty or seed aggregation is reported in the table.

Go to primary source ↓

e12PDF p. 7, Figure 5, Table 2 caption and Section 3.3; p. 8, Section 3.3 paragraphs 1–2Inspect

Physical evaluation uses pi0.5 and an i2RT YAM Arm, 300 demonstration episodes per task, 50k fine-tuning steps and 30 rollouts per task/method/setting. Assembly distractors are unused furniture parts, folding distractors are unseen socks, and closet OOD opens the otherwise closed right cabinet door.

Go to primary source ↓

e13PDF p. 7, Table 2, all five task columns and both method rowsInspect

Direct/AFP ID and OOD success: Drawer 0.73/0.83 and 0.30/0.67; Lamp 0.57/0.67 and 0.20/0.53; Table 0.40/0.50 and 0.13/0.37; Cloth 0.67/0.73 and 0.47/0.57; Closet 0.63/0.70 and 0.23/0.53. Rates are rounded to two decimals; uncertainty is not supplied.

Go to primary source ↓

e14PDF p. 8, Section 4, Limitations paragraphInspect

The authors identify annotation cost and bias, ambiguous task relevance, heavy occlusion and mask-dataset distribution shift; suggested future directions include more annotation and self-supervised grounding.

Go to primary source ↓

e15PDF pp. 13–14, Appendix B; p. 14, Figure 6, both panels and captionInspect

Figure 6 shows stable projected-loss training versus a rising, vertically clipped unprojected curve, and frequent negative gradient cosine similarity. The caption reports 11,152 projection triggers over 20k steps. Task, backbone and repeats are unnamed. Appendix wording calls the projected decrease monotonic, although the plotted trace has small upward fluctuations.

Go to primary source ↓

e16PDF pp. 14–15, Appendix C and Figure 7Inspect

A local annotation interface accepts positive/negative key-frame prompts and multiple instance layers. SAM 2 initializes masks; merged masks initialize MatAnyone propagation, producing continuous alpha masks throughout each sequence.

Go to primary source ↓

e17PDF p. 15, Appendix D and Figure 8, metric tableInspect

The additional-backbone grounding comparison repeats the main protocol. SmolVLA OOD Soft-IoU rises from 0.105 to 0.763, OpenVLA from 0.133 to 0.849, and Motus from 0.173 to 0.861. This is a backbone comparison, not a controlled mask-quality manipulation.

Go to primary source ↓

e18PDF p. 16, Appendix E and Figure 9, convergence panels and episode-length barsInspect

ID pi0.5 success curves extend to 40k training steps, with earlier gains visible on Kitchen and Coffee Preparation and close Coffee/Stack Three curves. Episode lengths include Kitchen 622, Coffee Preparation 678, and Square 157 steps. The source relates convergence to task complexity but supplies no wall-clock/hardware accounting or demonstration-count sweep.

Go to primary source ↓

e19PDF pp. 3–4, Sections 2.2–2.4; pp. 5–8, Section 3; pp. 13–16, Appendices B–EInspect

These method, evaluation and appendix sections give component names, dataset size and update budgets but do not enumerate optimizer, learning rate, batch size, auxiliary-weight value, mask-training loss, resolution, query count, exact dataset partition, attention pooling implementation, hardware or software versions.

Go to primary source ↓

8.5 Primary sources

Artificial Foveated Perception for Mitigating Shortcut Learning in Robotic Foundation Models ↗

PDF · 7,739 extracted words

Source fingerprint

3d36c6233603a6a88c8c1eb4cc1b26c4b585eabcf0aef544b54f577ac19701df