Retrieve, Don't Retrain: Extending Vision Language Action Models to New Tasks at Test Time
1. Paper overview
In one sentence: ReCAP adds tasks to a frozen world-action policy through retrieved trajectories, while learned residuals adapt their motions to a compatible target embodiment. e-adaptatione-architecturee-pusht-resultse-robotwin-resultse-real-resultse-factoriale-limits
| At a glance | What to know |
|---|---|
| Research problem | Source description Can a robot acquire new tasks by indexing cheaper demonstrations instead of collecting target-robot data and fine-tuning for every task? Training assumes paired target/query and pool embodiments with semantically similar motions and shared state/action coordinates, including end-effector pose and gripper signals where applicable. Deployment supplies new pool demonstrations and current target observations, without target-task training or weight updates. e-adaptation |
| Core mechanism | Source description A frozen-policy adaptation protocol externalizes task coverage into an extensible pool; adaptation to a previously unseen embodiment still requires training. e-adaptation |
| A key reported result | RoboTwin: five held-out manipulation tasks: 31.5 unseen; 43.5 seen Mean success rate (%). Aloha-Agilex target, UR5 pool; five paired training tasks and five held-out evaluation tasks. Unseen: Retrieval Only 26.0, co-training 10.0, target-only Cosmos 4.0. Best seen baseline: Cosmos 32.5. Reader-calculated improvement over retrieval replay is 5.5 percentage points. Held-out per-task success ranges from 5.0 to 47.5, so transfer remains uneven. e-robotwin-setupe-robotwin-results |
| Reading caution | Source description Shared end-effector coordinates are required; raw video must first become state-action trajectories. Execution-speed or temporal-scale mismatch can undermine residual correction, especially for longer chunks. Scalable embodiment-invariant retrieval remains open. e-limits |
Core contributions
- Source description
A frozen-policy adaptation protocol externalizes task coverage into an extensible pool; adaptation to a previously unseen embodiment still requires training. e-adaptation
- Author claim
A residual over retrieved actions and a joint future-image objective are combined within the Cosmos video-generation backbone. The authors interpret retrieval as coarse progression and prediction as a visual consistency signal. e-architecture
Figure 2. A retrieved motion becomes context for a jointly generated target action and future image. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start at the lower-left current observation and follow the curved arrow into the retrieval space, then the returning arrow toward the pool chunk. The blue query context and green retrieved context enter the same ReCAP block. Read the outputs above it as target-embodiment actions and a future target observation. Section 4.1 supplies the implementation detail missing from this overview: current and retrieved inputs become clean latent frames along the temporal axis, language enters through cross-attention, and future query outputs are denoised. Equation (3) further specifies that the action output adds a learned residual to the retrieved pool action, rather than simply replaying that action. e-architecturee-adaptatione-inference
What it supports. The architecture supports the catalog’s joint-prediction classification because actions and future images belong to one denoised video sequence. Retrieval supplies an external motion reference, while the shared model learns its target-side correction. The picture alone does not show an independent world-model planning loop.
Where the evidence stops. The noisy squares sit above the retrieved-action label in this schematic. Section 4.1 explicitly assigns clean conditioning to the retrieved chunk and noise to future query outputs; use that formulation rather than inferring corruption or an attention mask from horizontal placement.
2. Motivation
2.1 The problem and the proposed response
Can a robot acquire new tasks by indexing cheaper demonstrations instead of collecting target-robot data and fine-tuning for every task? Training assumes paired target/query and pool embodiments with semantically similar motions and shared state/action coordinates, including end-effector pose and gripper signals where applicable. Deployment supplies new pool demonstrations and current target observations, without target-task training or weight updates. e-adaptation
2.2 What this reading follows
A demonstration from another embodiment can tell a robot how a task progresses, even when copying its motions would produce the wrong contacts. ReCAP builds that distinction into a Cosmos Policy: retrieve an observation/action chunk, predict a correction to its actions, and jointly predict the target scene’s future image. Paired training teaches this transfer once; deployment expands a trajectory pool while keeping policy weights fixed. The evidence spans controlled pushing, simulated dual-arm manipulation and a small physical-robot study. Read the comparisons alongside the residual/state ablation: they show useful transfer, but also its dependence on curated demonstrations, shared action coordinates and retrieval quality. e-adaptatione-architecturee-pusht-resultse-robotwin-resultse-real-resultse-factoriale-limits
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | WAMs |
| Architecture | One Model |
| Prediction paradigm | Joint prediction |
| Quadrant | Q1 · One Model × Joint prediction |
3.1 Evidence-based assessment
Supports the recorded classification
The main ReCAP implementation supports One Model × Joint prediction: query actions and future images are denoised in one shared video sequence, beyond merely sharing a loss. Retrieval is external memory, not a second learned dynamics-to-policy pipeline. The action-only π0.5 comparison does not inherit this WAM classification. Memory and action alignment are supported; a distinct long-horizon scaling evaluation is not supplied. e-architecturee-backbonee-inference
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Separate transferable task progression from embodiment-specific execution
Imagine transferring a disc-pusher demonstration to a triangle pusher. The sequence can indicate how to approach and move the T-block, but the contact geometry changes what action actually works. ReCAP expresses this distinction through a residual: the retrieved action provides a reference and the learned output corrects it in shared coordinates. Current target observations remain available, so the policy does not blindly follow a fixed source trajectory. The same principle motivates the UR5-to-Aloha and human-hand-to-robot settings. Reader interpretation: the shared representation is the enabling assumption behind the apparent simplicity of adding actions. If embodiments have incompatible action spaces, the residual cannot by itself establish a correspondence; the paper identifies an action translator or embodiment-agnostic interface as future work. e-adaptatione-architecturee-pusht-setupe-robotwin-setupe-real-setupe-limits
5.2 Read the future-image objective as a testable interaction
The model jointly denoises an action chunk and a future query observation, using clean current and retrieved latent context. The authors argue that future pixels provide a dense signal for adapting contacts and other embodiment differences. Table 3 tests this account more directly than the backbone comparison: switching on next-state supervision barely changes the absolute-action setting, while the residual setting improves more strongly. Without retrieval, the same objective reduces reported unseen success. Reader interpretation: this supports a context-dependent learning benefit, not a general claim that predicting images always improves policies. The attention analyses add evidence that retrieved information affects generated actions, but their selected rollouts and broad masks do not identify the whole causal chain. Neither experiment cleanly tests whether an otherwise unchanged policy must generate future images at deployment. e-architecturee-factoriale-backbonee-attentione-causal
5.3 Treat pool growth as a change in accessible task evidence
At inference, the unavailable future query action is removed from retrieval scoring. Algorithm 1 first selects trajectories by the initial scene, then repeatedly matches a subframe, predicts a correction and executes a short prefix. Table 10 gives a 16-frame horizon and one-action stride, distinct from its five-trajectory prefilter. Adding pool demonstrations changes the context available to this frozen computation. In RoboTwin, the reported pool levels increase from 11 to 35 tasks while introducing one evaluation task at each level, and success rises from 9.0% to 31.5%. Reader interpretation: that is evidence for task coverage through retrieval, but pool size and task relevance are confounded. Reproduction must also resolve the final pool row’s seven named additions and the disagreement between live-refresh prose and Algorithm 1’s fixed episode-level candidates. e-retrievale-inferencee-hyperparameterse-pool-growth
5.4 Training and inference
During training
The backbone is Cosmos Predict2.5 (2B). Table 10 gives joint flow matching, action-loss weight 16, AdamW, learning rate 1e-4 with cosine decay and 2,000 warm-up steps, 20,000 iterations, and effective batch 512 from 16/GPU × four accumulation steps × eight GPUs. Component-level freezing during fine-tuning is unspecified. e-hyperparameters
RoboTwin retains 246 paired training episodes across five tasks after four query-episode exclusions for severe retrieval misalignment. Three weighted retrieval pairs per sample use a top-five prefilter. These curated exclusions matter to the reported transfer setting. e-robotwin-setupe-hyperparameters
During inference
Algorithm 1 selects candidates at episode start, then observes, retrieves a subframe, predicts a residual chunk, executes its first actions and repeats. Table 10 specifies horizon H=16 and execution stride K=1; its separate top-K trajectory count is five. Predicted images participate in joint generation, but no explicit imagined-rollout search or image-scoring controller is described. e-inferencee-hyperparameterse-architecture
Pool-extension prose says both retrieval stages rerun against an updated index, while Algorithm 1 fixes Stage 1 for an episode. Mid-episode candidate refresh is unresolved. Action slices also alternate between t:t+H in the main equations and t:t+H−1 in the algorithm. e-inferencee-architecture
5.5 Implementation flow
- Prefilter trajectories
Compare initial scenes using instruction embeddings, SAM3 object information and initial proprioception. Retain the nearest candidate trajectories; RoboTwin uses five. e-retrievale-retrieval-config
- Match a local moment
Score subframes using weighted squared geometric distances and cosine distance between DINO visual features. Training additionally matches the upcoming query action; inference removes that unavailable term. RoboTwin includes history and delta-action terms in its detailed configuration. e-retrievale-retrieval-config
- Condition and denoise
Encode the observed query frame and retrieved chunk as clean latent frames prepended along time. Language enters through cross-attention. The shared Cosmos model denoises query actions and future observations; action latents encode the residual correction. No separate inverse-dynamics action extractor is specified. e-architecture
6. Experiments & results
ReCAP freezes a retrieval-conditioned Cosmos Policy after paired cross-embodiment training. New pool trajectories supply task progression; a learned residual adapts their actions to the target robot, jointly with future-image prediction. Unseen-task execution improves on PushT, RoboTwin and a small real-robot study, but requires compatible action coordinates and curated trajectory data (e-adaptation, e-architecture, e-pusht-results, e-robotwin-results, e-real-results, e-limits).
6.1 Read the original evidence
Table 2. ReCAP improves both trained-angle and held-out-angle success in the controlled pusher transfer. Original paper, p. 12 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the shaded −45° and +45° columns as the training angles. Every other angle belongs to the unseen average at the right. The target is a triangle pusher and the demonstration pool uses a disc pusher; thus the policy must handle different contact geometry even when the goal angle is familiar. Training uses 100 paired demonstrations, while the deployment pool covers the angle range at finer 5° resolution. Compare the four methods within a column before comparing averages: Cosmos has no pool access, Retrieval Only directly executes pool actions, and Co-train (all) learns from the union of target and pool data. e-pusht-setupe-pusht-results
What it supports. ReCAP reaches 34.9% mean unseen success and 50.0% seen success. Co-training is the strongest listed unseen baseline at 19.1%, ahead of direct retrieval replay at 16.9% and target-only Cosmos at 6.0%. This comparison supports learning how to use pool experience beyond merely possessing it.
Where the evidence stops. These are task-angle generalization results in a constrained 2D environment with a dense deployment pool. The source does not state evaluation rollout counts, seed variation or confidence intervals for this table; do not infer them from its percentage increments.
Table 1. Learned correction improves the unseen-task average beyond direct UR5 trajectory replay. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Separate the seen-task block on the left from the held-out-task block on the right, and use the retained abbreviation key to identify each manipulation task. The target robot is Aloha-Agilex; retrieved trajectories come from UR5. Baseline uses target-only Cosmos training, Retrieval Only executes a retrieved action sequence without learning, and Co-training combines both embodiments’ training trajectories. The paired ReCAP training set contains 246 episodes after documented exclusions for retrieval misalignment. Compare averages within each split: the strongest competing method changes from target-only Cosmos on seen tasks to direct retrieval replay on unseen tasks. A single baseline label therefore cannot summarize both comparisons. e-robotwin-resultse-robotwin-setupe-limits
What it supports. ReCAP reaches 31.5% unseen success versus 26.0% for retrieval replay, a reader-calculated gain of 5.5 percentage points; seen success reaches 43.5%. The held-out columns remain uneven: Move Pillbottle Pad is 5.0%, while Click Bell and Hand-over Mic each reach 47.5%.
Where the evidence stops. This table evaluates simulated execution with task-specific pool demonstrations. It does not demonstrate reliable transfer to arbitrary tasks or action spaces. Evaluation rollout counts and uncertainty are not reported; the four removed training episodes also constrain conclusions about robustness to poor retrieval.
Figure 8. Tracked human-hand demonstrations enable two held-out physical robot behaviors with frozen policy weights. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the blue training database as paired robot and human demonstrations for opening the cabinet. The green test database adds human demonstrations of closing the cabinet and placing a bottle in a plastic box; those behaviors were excluded from robot fine-tuning. The right panel’s green hatched bars are the no-retrieval baseline and the solid blue bars are ReCAP. The setup text specifies 25 paired training demonstrations and ten human-hand pool demonstrations per held-out task. The Figure 8 caption separately specifies ten evaluation rollouts per task. Those are different quantities: demonstration count describes adaptation input, whereas rollout count describes evaluation. e-real-setupe-real-resultse-limits
What it supports. Both policies score 90% on opening, the trained behavior. With retrieved human trajectories, bottle placement reaches 80% versus 10% for the baseline, and cabinet closing reaches 30% versus 0%. These are physical execution results; the evidence is broader than generated-video plausibility alone.
Where the evidence stops. The human demonstrations include VR-tracked wrist poses, so this is not learning directly from unprocessed video. Ten rollouts per task give a small evaluation sample, and no confidence intervals are supplied. The paper does not specify the physical robot model or detailed success criteria here.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| PushT: seven unseen goal angles Triangle target, disc pool; 100 paired demonstrations at ±45°. Evaluation spans −60° to +60° in 15° steps; the deployment pool samples at 5° resolution. | 34.9 unseen; 50.0 seen Mean success rate (%) | Unseen: Cosmos Policy 6.0, Retrieval Only 16.9, Co-train (all) 19.1. The strongest listed unseen baseline is co-training, not the 6.0% baseline highlighted in the introduction. This is controlled angle transfer with source-embodiment demonstrations. e-pusht-setupe-pusht-results |
| PushT: residual × next-state objective ablation Retrieval-conditioned Cosmos; average across seven unseen angles. | Absolute: 26.3 without state prediction, 27.4 with; residual: 27.4 without, 34.9 with. Mean success rate (%) | No retrieval: 8.3 without state prediction, 6.0 with. Supports an interaction between retrieval, residual actions and state supervision. Table 3 prints +7.4 percentage points where rounded cells subtract to 7.5; unrounded values are unavailable. e-factorial |
| RoboTwin: five held-out manipulation tasks Aloha-Agilex target, UR5 pool; five paired training tasks and five held-out evaluation tasks. | 31.5 unseen; 43.5 seen Mean success rate (%) | Unseen: Retrieval Only 26.0, co-training 10.0, target-only Cosmos 4.0. Best seen baseline: Cosmos 32.5. Reader-calculated improvement over retrieval replay is 5.5 percentage points. Held-out per-task success ranges from 5.0 to 47.5, so transfer remains uneven. e-robotwin-setupe-robotwin-results |
| RoboTwin: frozen-policy pool progression Nested pools labeled 11, 17, 23, 29 and 35 tasks; one additional held-out task becomes retrievable at each level. | 9.0, 18.5, 19.5, 22.0, 31.5 Unseen-task mean success rate (%) | Same frozen policy across reported pool levels. Pool size and relevant-task inclusion change together. Table 9 lists seven final additions despite describing increments of six; exact membership/count needs clarification. e-pool-growth |
| Real robot: bottle placement and cabinet closing Train on 25 paired open-cabinet demonstrations; add ten tracked human-hand demonstrations per held-out task. Ten evaluation rollouts per task. | Place bottle: 80; close cabinet: 30; seen open cabinet: 90. Task success rate (%) | No-retrieval baseline: 10, 0 and 90 respectively. Physical execution outcomes from a small study without reported confidence intervals. Robot model and detailed success criteria are not specified in the supplied experiment description. e-real-setupe-real-results |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 3. Future-state supervision helps most when actions are residuals over a retrieved trajectory. Original paper, p. 12 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use the cross and checkmark in the first column to toggle the next-state objective; move horizontally to compare absolute action prediction with a residual added to pool actions. The four central cells are mean success percentages on unseen PushT angles. The right column reports residual-minus-absolute differences, and the bottom difference row reports state-on minus state-off. Finally, read the separate no-retrieval line: adding next-state supervision there moves success from 8.3% to 6.0%. Keeping that line visible prevents the mistaken interpretation that a future-image loss improves control independently of how retrieval and action parameterization are organized. e-factoriale-architecture
What it supports. The strongest cell combines residual actions and next-state supervision at 34.9%. Residual actions alone give 27.4%; absolute actions with next-state supervision also give 27.4%. The pattern supports the authors’ proposed interaction, while the negative no-retrieval change limits any claim of a universal benefit from state prediction.
Where the evidence stops. The printed difference is +7.4 percentage points, while subtracting the displayed rounded values 34.9 and 27.4 gives 7.5. Unrounded values are unavailable. Preserve both observations; this training ablation also does not isolate the necessity of future-image generation during deployment.
Table 6. Retrieval attention can sustain successful behavior and also sustain some failures. Original paper, p. 15 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the Baseline column as the original per-angle ReCAP success rates from Table 2, not a new no-retrieval model. The intervention columns use different denominators: 5S means five rollouts selected because they originally succeeded, and 5F means five selected failures at each angle. Their starting rates are therefore 100% and 0% by construction. L10 block and L15 block mask action-to-retrieval attention before softmax at the named decoder layer, covering retrieved frames, proprioception and actions. The arrows compare outcomes before and after each intervention; the parenthetical negative values are percentage-point changes within the selected successful subsets. e-causale-attentione-failurese-pusht-results
What it supports. Blocking L10 reduces selected-success performance by 20–80 percentage points; blocking L15 reduces it by 20–60 points. Yet L10 masking rescues 20–40% of selected failures at six angles. Retrieval therefore has a functional role that can be helpful or harmful, consistent with the authors’ over-anchoring diagnosis.
Where the evidence stops. The prose names 0° among the largest rescues, but its table entry is 20%; ±60° show 40%. Use the table values. Outcome-selected five-rollout subsets and broad attention masks do not establish a general-population effect or uniquely isolate a predicted-state commitment mechanism.
7. Analysis & limitations
7.1 What the evidence leaves open
Shared end-effector coordinates are required; raw video must first become state-action trajectories. Execution-speed or temporal-scale mismatch can undermine residual correction, especially for longer chunks. Scalable embodiment-invariant retrieval remains open. e-limits
Attention probes associate layer 10 with retrieved task regions and layer 15 with predicted end poses. Masking action-to-retrieval links at either layer damages selected successful rollouts; masking layer 10 rescues some failures. These small, outcome-selected interventions establish pathway sensitivity, not a universal causal decomposition. e-attentione-causal
Failure tables show lower mean layer-15 attention on failures at every examined angle and mixed layer-10 under-/over-anchoring. Claimed monotonic dependence on distance from ±45° is not established: ±30° and ±60° have equal nearest-training-angle distance but opposite anchoring signs. e-failures
Primary success tables omit seed variation, uncertainty and explicit rollout counts for PushT/RoboTwin. Future-image quality and inference latency are not quantitatively evaluated. The ablation tests training supervision, not whether generating images at deployment is necessary. e-pusht-resultse-robotwin-resultse-factoriale-hyperparameters
7.2 Questions for discussion
- Does the residual/state interaction persist when training retrieval cannot use future query actions? (e-retrieval, e-factorial)
- How much of pool progression comes from adding the correct task rather than adding more trajectories? (e-pool-growth)
8. Reproducibility audit
8.1 Requirements and known gaps
Reproduction requires paired embodiment data, the task split, trajectory features, clean conditioning layout and Cosmos checkpoint. Table 10 provides optimization and eight-GPU batch configuration but no GPU model, software versions, denoising-step count or measured runtime; PushT feature weights remain qualitative. e-architecturee-robotwin-setupe-retrieval-confige-hyperparameters
Preserve RoboTwin exclusions: Move Can Pot episodes 32/45, Open Microwave 34, Grab Roller 38. Resolve pool count and candidate-refresh discrepancies before reproducing progression. Inference handling of configuration-specific history/delta-action terms is less explicit than Equation (8). e-robotwin-setupe-pool-growthe-retrievale-inferencee-retrieval-config
Proposed checks: repeat the residual/state factorial with matched retrieval and multiple seeds; separately compare equally sized pools with versus without the held-out task. These test the mechanism and distinguish relevant coverage from database size; neither was run here. e-factoriale-pool-growth
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Repeat the residual/state interaction under matched retrieval
Reader-proposed check: on the same triangle/disc PushT split, train all four Table 3 configurations with identical data, sampled retrieval pairs, initialization family, optimization budget and evaluation starts, repeating across independently chosen seeds. Include the no-retrieval pair as a control and report exact success counts with uncertainty. Compare the effect of state supervision under residual versus absolute actions. The mechanism predicts a consistently larger benefit in the residual condition; an effect that vanishes across seeds or matched retrieval would weaken that explanation. Record unrounded averages so the printed +7.4 discrepancy can be resolved. This is a proposed experiment, not a completed reproduction. e-pusht-setupe-factoriale-architecturee-hyperparameters
Check 2: Separate pool relevance from pool size and refresh timing
Reader-proposed check: freeze one RoboTwin checkpoint and compare equal-sized pools that either include the evaluated held-out task or replace those demonstrations with distractor tasks, matching total trajectory counts. Repeat over task-insertion orders and matched starting scenes. A gain concentrated on relevant additions would support task coverage as the mechanism; a similar gain from distractors would challenge that account. In a separate controlled arm, insert the relevant demonstrations after episode initialization and compare fixed candidates with explicit Stage-1 refresh. Improvement only after refresh would resolve the operational limit of Algorithm 1. First reconcile Table 9’s final task count, and log retrieval identities and executed success rather than judging predicted video. e-robotwin-setupe-pool-growthe-inferencee-retrieval
8.3 Reading coverage
Visual audit: The title/author/version page and every non-reference page were rendered and visually inspected. This includes all Figures 1–11, Tables 1–10, Equations (1)–(8), Algorithm 1, experiment setup, limitations, failure analysis, RoboTwin episode exclusions, pool composition, retrieval details and training configuration. All six final original crops were viewed; Table 1 and Table 2 were tightened and viewed again to remove caption fragments while retaining headers and necessary footnotes. Figure 2’s noisy-slot placement was cross-checked against clean/noisy conditioning text; Table 3’s printed delta, Table 6’s rescue prose, Table 9’s final count and Algorithm 1’s refresh schedule remain explicitly qualified. Reference pages 9–11 were read in the text pass and contain no supporting claims used from external papers. No separate supplement or linked code was inspected.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 12, 13, 14, 15, 16, 17, 18, 19, 20. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- 1 Introduction
- 2 Related Work
- 3 Problem Formulation
- 4 Proposed Method
- 4.1 Retrieval-Augmented World Action Model
- 4.2 Retrieval
- 5 Experiments
- 5.1 Experiment Setup
- 5.2 PushT Experiments
- 5.3 RoboTwin Simulation Experiments
- 5.4 Real Robot Experiments
- 6 Discussions
- References
- Appendix
- A Additional PushT Results
- B PushT Mechanism Analysis
- C PushT Failure Case Analysis
- D RoboTwin Setup Details
- E Retrieval Rule Details
- F Hyperparameters
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- All six supplied chunks were read individually, covering the complete 20-page paper, references and Appendices A–F. No supplied text section was omitted. Reference pages 9–11 were read as text; all other pages were also visually inspected.
- Identity note: the PDF identifies arXiv:2606.15631v1 [cs.RO], 14 June 2026. All six catalog authors match the title page. The printed title uses “Don’t” and “Vision-Language-Action”; the supplied observedTitle normalizes the apostrophe and hyphens. No substantive title or author revision difference is established; no other edition was supplied or compared.
- The acquisition warning that text extraction does not reconstruct figure images was addressed by inspecting the retained PDF and final original crops.
- Separate supplemental material availability has not been fully verified. No separate supplement was supplied.
- Linked project pages, repositories, code, datasets and cited papers were not opened or inspected. No experiments were reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e-identityPDF p. 1, title/author block and arXiv version stamp
Printed title: Retrieve, Don’t Retrain: Extending Vision-Language-Action Models to New Tasks at Test Time. Authors: Jeongeun Park, Juhan Park, Taekyung Kim, Sungjoon Choi, Dongyoon Han, Sangdoo Yun. Affiliations: NAVER AI Lab and Korea University. Stamp: arXiv:2606.15631v1, 14 Jun 2026.
Go to primary source ↓e-adaptationPDF pp. 1–3, Abstract; Figure 1; Section 3, Setting and Train and test access
Paired query/pool training precedes freezing. Deployment adds pool demonstrations without new query-task data or parameter updates; shared action coordinates and semantically similar trajectories are assumed. A new embodiment still requires adaptation.
Go to primary source ↓e-architecturePDF p. 4, Figure 2 and Section 4.1, Equations (1)–(3)
Observed and retrieved inputs become clean temporal latent conditioning; language uses cross-attention. A shared Cosmos video sequence jointly denoises future actions/images; actions are residuals over pool actions. Loss is weighted action plus state supervision.
Go to primary source ↓e-retrievalPDF pp. 18–19, Appendix E, Equations (5)–(8)
Initial-scene language/object/proprioceptive retrieval precedes local matching. Squared L2 handles geometric features and cosine distance handles visual features. Future query actions enter training cost but are removed in Equation (8) at inference.
Go to primary source ↓e-inferencePDF p. 19, Algorithm 1, Pool extension at deployment and Inference loop; p. 20, continuation
Algorithm fixes initial-scene candidates outside the loop, then reruns subframe retrieval and executes the first K actions. Pool-extension prose says both stages rerun on the current index. Main equations use a different action-slice endpoint from Algorithm 1.
Go to primary source ↓e-pusht-setupPDF p. 5, Section 5.1, PushT Environment; Figure 3
Triangle target and disc pool train on 100 paired demonstrations at ±45°. Nine evaluation angles use 15° increments; seven are unseen. Deployment pool covers −60° to +60° at 5° resolution.
Go to primary source ↓e-pusht-resultsPDF p. 12, Appendix A, Table 2, Seen/Unseen averages and method definitions
Unseen success: Cosmos 6.0%, Retrieval Only 16.9%, Co-train 19.1%, ReCAP 34.9%. Seen ReCAP is 50.0%. Replay executes retrieved actions directly; co-training uses target and pool trajectories. Seed variability and evaluation rollout counts are not stated here.
Go to primary source ↓e-backbonePDF p. 6, Figure 5(a–c) and Section 5.2
Retrieval raises π0.5 unseen success from 6.6% to 25.1%, and Cosmos from 6.0% to 34.9%. The future-image objective comparison and ROI curves motivate the mechanism analysis.
Go to primary source ↓e-factorialPDF p. 12, Table 3; p. 13, Appendix A continuation
Absolute/residual success without state prediction: 26.3/27.4%; with prediction: 27.4/34.9%. Printed interaction-related differences are +7.4, despite rounded-cell subtraction giving 7.5. No-retrieval state supervision changes 8.3 to 6.0%.
Go to primary source ↓e-robotwin-setupPDF p. 5, Section 5.1, RoboTwin Simulation Environment; p. 17, Appendix D, Train and test tasks and Excluded query episodes
Aloha-Agilex target and UR5 pool use five training tasks with 50, 48, 49, 49 and 50 paired episodes, totaling 246; four specified query episodes are excluded. Five distinct tasks are held out from fine-tuning.
Go to primary source ↓e-robotwin-resultsPDF p. 7, Table 1, seen/unseen blocks and footnotes; Section 5.3
ReCAP averages 43.5% seen and 31.5% unseen. Baseline: 32.5/4.0; Retrieval Only: 25.5/26.0; Co-training: 27.0/10.0. ReCAP held-out tasks MPP/PBS/CB/HM/LP score 5.0/12.5/47.5/47.5/45.0%.
Go to primary source ↓e-pool-growthPDF p. 7, Figure 7; p. 17, Appendix D, Progressive retrieval pool; p. 18, Table 9
Labeled pools 11/17/23/29/35 yield 9.0/18.5/19.5/22.0/31.5% unseen success with frozen weights. Each stage adds a held-out task. The final table row names seven added tasks, conflicting with its six-addition description and size increment.
Go to primary source ↓e-real-setupPDF p. 5, Section 5.1, Real Robot Setup
Pool data are human-hand video with VR-tracked wrist pose. Training uses 25 paired open-cabinet demonstrations; held-out placement and closing receive ten human-hand pool demonstrations each.
Go to primary source ↓e-real-resultsPDF pp. 7–8, Section 5.4 and Figures 8–9
Figure 8 reports ten rollouts per task: baseline/ReCAP open 90/90%, place 10/80%, close 0/30%. Figure 9 illustrates held-out rollouts. No uncertainty or precise robot-model specification accompanies this setup.
Go to primary source ↓e-limitsPDF p. 8, Section 6, Limitations and Future Work
Authors require compatible low-level action spaces and trajectory-valued pool data, identify temporal/dynamical mismatch as a residual-policy weakness, and leave raw-video lifting and scalable retrieval representations open.
Go to primary source ↓e-attentionPDF pp. 13–14, Appendix B, Table 4, Equation (4), Figure 10 and Table 5
ROI attention is normalized against uniform spatial attention. Probes use iteration 7000 and selected rollout subsets. Retrieved task ROIs peak at layer 10; predicted-end ROIs peak at layer 15; no-retrieval attention is flatter.
Go to primary source ↓e-causalPDF p. 15, Table 6 and Appendix B, Both axes are causally necessary
Masking action-to-retrieved-frame/proprio/action attention before softmax at L10 or L15 reduces selected five-success subsets by 20–80 or 20–60 percentage points. L10 failure rescue is 40% at ±60°, 20% at −30°, −15°, 0°, +30°, and 0% at +15°. Prose incorrectly includes 0° among the largest rescues.
Go to primary source ↓e-failuresPDF pp. 15–17, Figure 11; Appendix C; Tables 7–8
Five successes and five failures per unseen angle are compared. Mean L15 attention is lower on failures at every angle; L10 gaps are negative at ±30°/+15° and positive elsewhere. Coverage ≥0.10 labels 25 over-anchored failures; <0.10 labels ten non-engaging failures. Proposed attention remedies are future work.
Go to primary source ↓e-retrieval-configPDF p. 17, Appendix D, Retrieval scoring weights; p. 19, Per-feature weights; p. 20, Table 10 and Appendix F
RoboTwin uses sign-invariant 20-D proprioception, position weight 4.0, top-five trajectories, SAM3 width-height weight 1.0, proprio/history/image/action/delta-action/time weights 1.0/0.05/0.1/0.1/0.1/0.0. PushT geometric dominance and low visual weight are qualitative, not a full numeric configuration.
Go to primary source ↓e-hyperparametersPDF p. 20, Table 10, Backbone and Training rows; Appendix F
Cosmos Predict2.5 2B, residual joint flow matching, three weighted retrieval pairs, horizon 16, stride one, λ=16, AdamW 1e-4 with cosine/2000-step warm-up, effective batch 512 on eight GPUs, and 20,000 iterations are listed. GPU models, software versions, denoising steps and measured latency are absent.
Go to primary source ↓8.5 Primary sources
Retrieve, Don't Retrain: Extending Vision Language Action Models to New Tasks at Test Time ↗
PDF · 9,883 extracted words
Source fingerprint
c690f8b889c8de32eda0d2546a0b27e9dca14ea9005a52d672470a192d73c431