How to Instruct Your Robot: Dense Language Annotations Power Robot Policy Learning
1. Paper overview
In one sentence: Re-annotating fixed demonstrations with task-appropriate language improves simulated policy learning, but deployment depends on how captions are selected, generated and injected. e-datae-aspectse-instructore-resultse-deve-compositee-scaling
| At a glance | What to know |
|---|---|
| Research problem | Source description Short task labels omit motion, spatial relations, embodiment state and subgoal dependencies already visible in demonstrations. The paper asks whether richer language can extract more learning signal from fixed robot and human-video corpora, and whether useful captions can be supplied without access to demonstration futures at deployment. e-datae-instructor |
| Core mechanism | Source description A four-aspect annotation pipeline covers 11K RoboCasa clips, a 1M-scale MolmoBot corpus and 50K EgoVerse clips without collecting new demonstrations. e-data |
| A key reported result | RoboCasa365 atomic manipulation with the learned instructor: 49% Mean success rate. Table 2 VLA test results: 17 target-split tasks, five seeds × 20 episodes per task, maximum 400 steps; simulator task-checker success. Task-only 44%; best fixed aspect, Physical Motion, 46%; reported fixed-condition oracle 52%. A five-percentage-point test gain, three points below the reported oracle. Table 8 is a different development comparison. e-resultse-eval |
| Reading caution | Source description The authors identify frozen-annotator bias, a heuristic four-aspect schema, residual instructor errors and simulation-only evaluation. Physical deployment is untested. e-limits |
Core contributions
- Source description
A four-aspect annotation pipeline covers 11K RoboCasa clips, a 1M-scale MolmoBot corpus and 50K EgoVerse clips without collecting new demonstrations. e-data
- Source description
A reward-guided instructor generates task-appropriate language asynchronously; experiments compare annotation choices, deployment controls, composite tasks and compute scaling across VLA and WAM policies. e-instructore-resultse-compositee-scaling
Figure 3. Dense language conditions both a VLA and a video-based policy, with different action-generation machinery. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the four colored caption boxes, then follow the task-plus-caption inputs into each policy. On the left, visual and language tokens feed a pretrained VLA and its action head. On the right, distinguish the video VAE path from the joint observation/language encoder; both provide information to the WAM, whose DiT connects to predicted video and action branches. The dashed action-head enclosure says 'post-training only': human-video mid-training has no action head. The drawing is an overview, so use Appendix B.1 for the action head's four attention layers and its access to the frozen Reason 1 prefix. e-policye-wame-vlae-scalinge-evale-consistency
What it supports. The contribution changes supervision and prompt conditioning across two existing policy families. The WAM diagram separates predicted video from continuous actions; a visually plausible future alone is not its manipulation evaluation. Robot action supervision enters after the human-video stage, and reported success comes from simulator execution.
Where the evidence stops. Figure 3 omits noisy action inputs, and the main text calls action decoding deterministic. Appendix B.1 specifies noisy-action conditioning and rectified-flow training. The source does not fully explain the inference procedure reconciling these descriptions; the crop is preserved as printed.
2. Motivation
2.1 The problem and the proposed response
Short task labels omit motion, spatial relations, embodiment state and subgoal dependencies already visible in demonstrations. The paper asks whether richer language can extract more learning signal from fixed robot and human-video corpora, and whether useful captions can be supplied without access to demonstration futures at deployment. e-datae-instructor
2.2 What this reading follows
DeMiAn begins with demonstrations that already exist and asks what additional information their language labels could expose. A short goal identifies the task; a denser caption can describe contact changes, spatial relationships, arm state or subgoal purpose. The paper turns these alternatives into training data and learns an instructor to generate useful language from initial observations at deployment. Its experiments are most informative when three comparisons stay separate: training with different annotation aspects, changing instructions for one fixed policy, and executing with generated rather than ground-truth prompts. This reading follows those distinctions through the architecture, controls, composite tasks and compute curves. e-datae-aspectse-instructore-resultse-deve-compositee-scaling
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Datasets |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
3.1 Evidence-based assessment
Supports the recorded classification
The recorded data-centric category fits reannotation of existing robot and egocentric corpora. This is a method paper about improving their supervision, not evidence of a newly collected dataset. Architecture and prediction quadrant are appropriately not applicable to the contribution as a whole: it is tested on two different policy families plus an instructor. The WAM feature-reading action head does not establish inverse dynamics or one universal joint-prediction architecture. e-datae-policye-wam
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 First separate privileged annotation from deployable instruction
The offline annotator can inspect a segment across time and consult contextual labels, including ground-truth primitive sequences for RoboCasa reasoning captions. That makes its output useful supervision, but it also gives it information unavailable before a rollout unfolds. DeMiAn addresses deployment by training a separate instructor from the task and three initial camera views. Its target is a pipeline caption selected using development success for that task and aspect; it can learn an empty response when dense captions consistently hurt. The action policy is trained across four aspects plus task-only conditioning so it can accept either a generated caption or no caption. The central test is therefore whether this restricted-view instructor supplies useful language, not merely whether a caption of an already observed demonstration helps. e-datae-aspectse-instructore-dev
Figure 2. The instructor prepares language while the policy begins the rollout using the task alone. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow the split after the task and scene: the upper branch generates a caption, while the lower branch starts producing actions immediately. The descending arrow joins the caption to a later action-policy call, rather than sending it backward to revise already executed actions. The labels t=0, t=1…20 and t=21 illustrate the sequence, not a fixed injection schedule. Section 4.2 says the policy uses eight-step open-loop chunks and inserts completed language at the next boundary. The instructor reads three RGB camera frames from the initial observation; it is not shown continuously reinterpreting every later frame. e-asynce-instructor
What it supports. Table 3 reports 49.0% success with asynchronous delivery versus 49.5% with blocking delivery. Asynchronous generation averages 1.87 seconds and reaches the policy at mean step 21.9. The practical claim is overlap between caption generation and acting, allowing control to start before language is ready.
Where the evidence stops. The diagram does not prove latency disappears on other hardware or tasks. Table 3 prints zero injected time for synchronous delivery despite defining that column as generation latency; this ambiguity prevents treating the row as a zero-wait measurement.
5.2 Then distinguish caption representation from caption generation
The VLA's auxiliary caption loss looks like next-token prediction, but its hidden states attend bidirectionally across the supplied caption. Appendix B.1 explicitly acknowledges that later tokens can be copied. The stated purpose is to keep object, relation and substep information decodable through the language-model head while the action expert learns control. It does not turn this policy into the deployment instructor. This distinction also matters empirically: the RoboCasa variant uses caption-loss weight 0.1, while the MolmoBot variant uses zero. Reader analysis: the displayed attention shift is compatible with greater use of language, but does not isolate whether the auxiliary loss causes that use or whether the attended words causally determine successful actions. Token interventions and a matched loss ablation would discriminate those possibilities. e-vlae-attentione-motion
5.3 Finally choose the comparison that answers the question
Table 2 asks what happens when training language changes, whereas Table 8 asks what instructions do for one already trained action checkpoint. Their oracle rows and fixed-aspect scores should not be pooled. The shared-checkpoint experiment supports an instructor advantage over random routing on development episodes, but the strongest fixed GT caption is almost tied with it. Composite execution adds another change: the policy receives phase-dependent prompts selected by simulator-assisted machinery. Better early milestones there do not imply better full-task success, and the generated-instructor condition finishes below both dynamic GT conditions. Reader analysis: the defensible conclusion is that richer language is a useful conditional resource whose value depends on training distribution, availability at deployment and evaluation protocol, rather than a universally improving prompt format. e-resultse-deve-oraclee-compositee-instructor
5.4 Training and inference
During training
The instructor-compatible policy samples uniformly among four caption aspects and task-only prompts. RoboCasa VLA fine-tunes pi05_base for 30K steps, batch 256, with caption-loss weight 0.1. The MolmoBot variant uses pi05_droid and weight zero. Main MolmoBot ablations use 223K trajectories (20%); scaling uses the full 1M corpus. e-vlae-instructor
WAM training uses video-only human-video mid-training before robot action supervision. Appendix B.1 gives 80K video-only, 50K end-to-end and 30K alignment steps, batch 256, AdamW learning rate 5×10⁻⁵, weight decay 10⁻⁵ and 3K warmup. With action labels, rectified-flow action loss accompanies video flow matching; Reason 1 remains frozen. e-wame-scaling
Instructor SFT uses 3.2K examples, four GB200 GPUs, effective batch 256 and AdamW with a cosine schedule. Top-1 SFT and two DPO variants are described as unsuccessful alternatives, without quantitative ablation scores. e-instructor
During inference
Three initial RGB camera frames and the task feed the instructor. The policy starts task-only; the generated caption enters at the next action-chunk boundary after completion, maintaining an eight-step open-loop cadence. This is language conditioning of executed actions, not a caption generated by the VLA itself. e-instructore-asynce-vla
Composite-task deployment additionally uses a subgoal state machine with simulator conditions and a done-flag threshold to switch prompts. It therefore relies on externally supplied phase-transition machinery. e-composite
5.5 Implementation flow
- Describe different information in the same demonstration
Sample up to ten uniformly spaced frames per segment. Qwen3-VL-30B-A3B-Instruct receives one call per aspect, with structured output and length constraints. Long RoboCasa demonstrations are split into single primitives; MolmoBot clips are used as-is. EgoVerse prompts describe human hands, with arm pose referring specifically to the first frame. e-datae-aspects
- Condition two policy families
The VLA uses openpi 0.5 with a PaliGemma backbone and flow-matching action expert. The WAM uses Cosmos-Predict 2.5 video features and a separate action head. Appendix B.1 specifies four action-attention layers accessing DiT features, a frozen Cosmos Reason 1 prefix and action queries, with a seven-dimensional action output. e-policye-wam
- Distill task-dependent caption utility
Measure development success w(τ,k) for task τ and aspect k using fixed-caption injection. Sample SFT targets through a temperature-2 softmax with top-three truncation; use an empty target when every aspect loses to task-only conditioning. Qwen3.5-2B learns to generate the selected pipeline caption from initial observations and the task. e-instructore-dev
6. Experiments & results
DeMiAn re-annotates existing demonstrations with four kinds of dense language, then trains an instructor to supply useful captions during control. Its clearest test result is a five-percentage-point RoboCasa VLA gain. Benefits depend on task, annotation and deployment protocol; all execution experiments are simulated.
6.1 Read the original evidence
Table 5. Better early milestones do not guarantee better completion when an instructor replaces ground-truth subgoal prompts. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Compare columns within a prompting regime before comparing across regimes. The '-fix' columns hold one composite instruction throughout the episode; '-dynamic' changes the prompt to match the current atomic phase. GT means ground-truth prompts, while 'instr.' uses the learned instructor. Read downward from phase-one reachability to phase-two reachability and then strict full-task success; these are distinct criteria, not interchangeable measures. The experiment covers eighteen composites built from seen atomic tasks, with twenty episodes per task/configuration and a 1,200-step limit. Prompt switching depends on a subgoal state machine, lenient simulator conditions and a done-flag threshold. e-compositee-limits
What it supports. Fixed-prompt DeMiAn improves full completion from 13% to 15%. With dynamic GT prompts it reaches 22%, versus the baseline's 19%. The instructor reaches 18% despite stronger early milestones than baseline dynamic GT: 61% versus 57% for phase one and 30% versus 26% for phase two.
Where the evidence stops. The source prose calls 18% 'between' the GT configurations, but both GT full-task results are higher, at 19% and 22%. These simulator-assisted phase transitions do not demonstrate autonomous subgoal detection or successful long-horizon physical deployment.
Figure 5. Annotation cost shifts the starting budget, while later training can reach higher success at comparable compute. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the left panel as human-video WAM mid-training followed by RoboCasa post-training, and the two right panels as VLA post-training on the full million-clip MolmoBot corpus. The right panels split Pick/Pick+Place from NextTo/Color and use logarithmic compute axes; the left has its own scale. Higher curves mean better simulated task success, while movement left means less compute. The shaded regions and arrows charge dense-caption generation before training gains are assessed. Appendix A.3 estimates one aspect using active MoE parameters and input/output token counts. Do not equate the one-aspect charge with the cost of generating every aspect for every clip. e-scalinge-coste-vlae-instructor
What it supports. The authors report matching baseline NextTo/Color performance with approximately 62% less compute, saving about 1.3×10²⁰ FLOPs. The plots support a compute-performance tradeoff for the tested training schedules, with annotated policies reaching stronger later performance; they do not show a monotonic benefit at every budget.
Where the evidence stops. Error bars are visible but their statistical definition is not stated. The plotted budgets do not establish full system cost including reward-table evaluation, instructor training and deployment. The one-aspect million-clip charge is approximately 5.0×10¹⁹ FLOPs; four aspects are roughly four times larger.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| RoboCasa365 atomic manipulation with the learned instructor Table 2 VLA test results: 17 target-split tasks, five seeds × 20 episodes per task, maximum 400 steps; simulator task-checker success. | 49% Mean success rate | Task-only 44%; best fixed aspect, Physical Motion, 46%; reported fixed-condition oracle 52%. A five-percentage-point test gain, three points below the reported oracle. Table 8 is a different development comparison. e-resultse-eval |
| SlideDishwasherRack annotation ablation Table 2, RoboCasa365 VLA fixed-compute training comparisons with the same visual data, architecture and action targets. | 75% with Physical Motion Success rate | 38% task-only; 65% Scene Composition; 46% Arm Pose; 50% Reasoning. A 37-point task-specific improvement; no fixed caption aspect wins universally. e-motione-policy |
| Instructor selection with a shared action checkpoint Table 8, RoboCasa development set, averaged over 17 atomic tasks; only inference instructions change. | 50.4% SFT instructor Mean success rate | 46.6% random aspect; 44.3% task-only; 50.1% fixed GT reasoning; 52.4% oracle. The 3.8-point advantage over random routing is a development-set result. Advantage over the strongest fixed injection is only 0.3 points; uncertainty is not reported. e-dev |
| Asynchronous instruction delivery Sections 4.2 and Table 3; blocking versus overlapping instructor delivery. | 49.0% async; 1.87 seconds; step 21.9 Success rate; mean generation latency and injection step | 49.5% sync; async median 1.86 seconds and step 23. Observed success differs by 0.5 points. Table 3 prints zero sync injected time despite describing generation latency; it does not establish zero synchronous waiting cost. e-async |
| RoboCasa365 composite manipulation 18 composites of seen atomic tasks, 20 episodes per task/configuration, maximum 1,200 steps; strict full-task checker. | 15% fixed composite prompt; 22% dynamic GT; 18% dynamic instructor Mean full-task success rate | Baseline fixed 13%; baseline dynamic GT 19%. Instructor deployment is below both dynamic GT configurations. Early milestones improve over baseline dynamic GT, but full completion does not. e-composite |
| MolmoSpaces generalization VLA trained on the 223K subset; Table 7 bench-v2 OOD columns. Test protocol uses 180 episodes per benchmark and a 0.5-second sustained success condition. | Instructor: Pick OOD 28%; Pick+Place OOD 27%; NextTo OOD 13% Success rate | Task-only: 18%, 22%, 7%, respectively. These specific OOD improvements do not imply uniform gains: Pick+Place Std/Hard average is 60% versus baseline 64%. e-oode-evale-vla |
| WAM annotation-conditioned manipulation Table 2 WAM rows; RoboCasa atomic-task and MolmoSpaces family summaries. | Instructor: RoboCasa 19%; MolmoSpaces 20% Mean success rate | Task-only: 18% and 16%; reported oracles: 20% and 19%. The instructor can exceed a fixed-condition oracle because the compared policy conditions differ; the oracle is not a universal ceiling. e-resultse-instructor |
| MolmoBot compute scaling on NextTo and Color Full 1M-clip VLA scaling experiment, including annotation-generation FLOPs in the horizontal axis. | Authors report approximately 62% less compute, saving approximately 1.3×10²⁰ FLOPs. Compute required to match baseline success | Task-only training curve in Figure 5. This is the authors' curve-based estimate, not a newly measured result. One-aspect annotation costs are estimated at 5.0×10¹⁹ FLOPs; all four aspects cost roughly four times more. e-scalinge-cost |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 2. The best annotation depends on the task, and the learned instructor must be compared with the appropriate fixed-condition oracle. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the VLA and WAM blocks separately; all entries are success fractions. Within a block, the first five rows vary training language, followed by instructor deployment and the reported oracle. Underlining marks the best fixed training condition, while bold includes the instructor and excludes the oracle. Use the blue averages to orient yourself, then inspect individual tasks such as SlideDishwasherRack. The MolmoSpaces summary averages Standard/Hard for Pick and Pick+Place, takes only in-distribution NextTo, and leaves Color unchanged before averaging those four summaries. These are not the same family averages as Appendix Table 7, which includes additional OOD columns. e-resultse-motione-policye-instructore-oraclee-ood
What it supports. For RoboCasa VLA, task-only success averages 44%, Physical Motion 46%, the instructor 49% and the reported oracle 52%. The much larger SlideDishwasherRack change, 38% to 75% with Physical Motion, explains why selecting language by task can matter even when average fixed-aspect gains look modest.
Where the evidence stops. The instructor uses a mixed-prompt policy, so the fixed-condition oracle is not its universal performance ceiling. Table 7 also inconsistently includes instructor maxima in its VLA oracle. The tables provide point estimates without confidence intervals.
Table 8. A shared-checkpoint comparison isolates instruction delivery from action-policy retraining. Original paper, p. 19 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. The key control is stated in the source caption: every row uses the same action-policy checkpoint. Start with the no-annotation baseline, then compare random per-episode aspect selection against the learned SFT instructor. The four GT rows inject pipeline-generated captions of a fixed aspect; they also summarize the development reward table used to choose instructor training targets. They are not separately trained policies as in the fixed-aspect rows of Table 2. Finally, compare the instructor against the strongest fixed injection and the per-task oracle. The rightmost column measures percentage-point change from the no-annotation baseline, not relative percentage improvement. e-deve-instructore-resultse-eval
What it supports. The instructor reaches 50.4%, versus 46.6% for random selection and 44.3% without annotations. That is evidence for useful learned instruction generation within this development setup. However, fixed GT reasoning already reaches 50.1%, leaving only a 0.3-point instructor advantage over the strongest fixed injection.
Where the evidence stops. These are development results, and development success also informs instructor targets. They should not be substituted for the 49% test result or treated as an independent held-out selection test. No uncertainty is shown for the small differences.
7. Analysis & limitations
7.1 What the evidence leaves open
The authors identify frozen-annotator bias, a heuristic four-aspect schema, residual instructor errors and simulation-only evaluation. Physical deployment is untested. e-limits
The source has unresolved descriptions: Section 2.2 says captions are required only in training, while the deployed instructor supplies them at test time. Main-text deterministic WAM decoding and Figure 3 omit the noisy-action/rectified-flow details in Appendix B.1; the precise inference solver is not established. e-consistency
Oracle construction is inconsistent across tables: Table 7's VLA oracle includes some instructor maxima, whereas Table 2 excludes the instructor. Table 5's prose places 18% between two GT results of 19% and 22%; the numerical ordering contradicts that sentence. Attention visualizations are illustrative, not causal proof of language grounding. e-oraclee-compositee-attention
7.2 Questions for discussion
- Does reward-guided caption generation outperform the strongest fixed caption on test episodes with uncertainty quantified?
- How much of composite-task success survives replacing simulator phase triggers with visual state estimation?
- Would compute gains persist after charging all four annotations, reward-table rollouts and instructor training?
8. Reproducibility audit
8.1 Requirements and known gaps
Reconstruction needs exact segment boundaries, generated captions, checkpoint revisions and reward tables. Openpi optimizer settings are deferred to defaults; instructor learning rate and training duration, WAM inference details and composite done-flag threshold are not numerically specified. Table 6's development seed allocation is unclear relative to its 340-trial total. e-datae-vlae-wame-instructore-compositee-eval
Proposed check: freeze the mixed-prompt policy and compare instructor, random aspect, fixed reasoning and task-only prompts on untouched test seeds; keep reward-table construction development-only and report paired uncertainty. e-deve-eval
Proposed check: at fixed demonstrations and training budget, vary caption-loss weight and independently mask action-relevant caption words at evaluation. A success drop specific to those words would support semantic use beyond the attention visualization. e-vlae-attentione-motion
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Test learned instruction utility without reusing the reward-table evaluation
Reader-proposed experiment, not performed: freeze the mixed-prompt RoboCasa policy and instructor after constructing targets from development episodes only. On identical untouched test initializations, compare task-only prompts, random aspect, fixed GT reasoning and the instructor. Use the same action horizon and explicitly matched injection timing; report per-task success, paired differences and uncertainty across seeds. Pre-register instructor versus random and instructor versus fixed reasoning as separate comparisons. A stable advantage over random but no advantage over fixed reasoning would support useful captioning while weakening the stronger claim that task-dependent selection is necessary. Repeat instructor abstentions as task-only controls. e-instructore-deve-evale-async
Check 2: Separate semantic caption use from the bidirectional auxiliary loss
Reader-proposed experiment, not performed: train matched RoboCasa VLA runs with caption-loss weights zero and 0.1, holding demonstrations, annotations, action objective and training budget fixed. Evaluate both with the same captions, then remove or replace equal numbers of object, motion and direction tokens, with a matched control perturbation of other tokens. Include SlideDishwasherRack and CloseToasterOvenDoor, and measure executed task success alongside attention. If action-relevant perturbations reduce success more than the controls, that supports semantic dependence; if the effect changes with the loss weight, it implicates the auxiliary regularizer. Attention redistribution without differential success loss would weaken the causal reading of Figure 4. e-vlae-attentione-motione-policy
8.3 Reading coverage
Visual audit: The title/author/version page, all five figures, all eight tables, annotation prompts, equations, training configurations, evaluation protocols and instructor hardware details were visually inspected on the declared PDF pages. All seven text chunks were read individually, including reference pages 10–14. Six final original crops were inspected; Table 2 and Figure 5 crop boundaries were corrected and the corrected images inspected. Architecture arrows and noise inputs were checked against Appendix B.1; the WAM description ambiguity is disclosed. Table 3 timing, Table 5 numerical ordering and Table 7 oracle inconsistencies are retained as cautions. Separate supplemental material remains unverified; references were read as text only. No external implementation or reproduced experiment is represented by these visuals.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 15, 16, 17, 18, 19. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Title, authors, affiliations and version (p. 1)
- Abstract
- 1 Introduction
- 2 Related Work, including 2.1–2.2
- 3 DeMiAn: Dense Multi-Aspect Annotation
- 4 Experiments, including 4.1–4.4
- 5 Conclusion
- References
- Limitations and future work (p. 15)
- A DeMiAn Multi-Aspect Annotation, including A.1–A.3
- B Experiment Details, including B.1–B.2
- C Extended Results and Analysis, including C.1–C.2
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- The extraction limitation was addressed by inspecting original PDF figures, tables and equations, including every final crop. Reference pages 10–14 were read as text but not rendered.
- Identity/version scope: the title and all eight authors match the catalog; the inspected title page identifies arXiv:2605.17077v1 [cs.RO], 16 May 2026, Preprint. No different revision or venue edition was supplied or compared.
- External code, dataset artifacts and linked resources were not inspected; no experiments were reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e-identityPDF p. 1, title/author block and arXiv margin
The exact catalog title appears with Bosung Kim, Ruiyi Wang, David Acuna, Jaehun Jung, Alexander Trevithick, Brandon Cui, Yejin Choi and Prithviraj Ammanabrolu; affiliations are University of California, San Diego and NVIDIA. Kim and Wang contributed equally. The artifact is v1, 16 May 2026, Preprint.
Go to primary source ↓e-dataPDF p. 2, Figure 1 and Introduction; p. 4, Section 3, Annotation pipeline
The study reuses 11K RoboCasa, 1M-scale MolmoBot and 50K EgoVerse clips. The pipeline samples at most ten frames, issues one Qwen3-VL call per aspect, uses contextual metadata and splits longer RoboCasa episodes into primitives.
Go to primary source ↓e-aspectsPDF p. 4, Table 1; pp. 15–16, Appendices A.1–A.2
Four aspects target motion/contact, spatial relations, embodiment state and subgoal purpose. Robot arm-pose prompts cover the trajectory, whereas the human-egocentric pose prompt specifies the first frame. Reasoning annotations can use ground-truth primitive and neighboring labels.
Go to primary source ↓e-policyPDF p. 5, Figure 3 and Policy models
The VLA pairs a pretrained vision-language model with an action expert; the WAM has video encoding, joint observation/language encoding, DiT video prediction and a post-training action branch. Annotation comparisons hold model architecture, visual data and action targets fixed.
Go to primary source ↓e-vlaPDF pp. 16–17, Appendix B.1, DeMiAn VLA and displayed unnumbered objective
The masked caption-token cross-entropy supplements action flow matching without a causal attention mask. The source explicitly acknowledges copying. RoboCasa uses pi05_base with weight 0.1; MolmoBot uses pi05_droid with zero caption loss. Both listed variants train 30K steps at batch 256; main MolmoBot ablations use 223K trajectories.
Go to primary source ↓e-wamPDF p. 17, Appendix B.1, DeMiAn WAM, Architecture and Training
Four action-query attention layers access DiT depths, frozen Reason 1 prefix and action queries; noisy action chunks and flow time condition a seven-dimensional action head. Training combines action/video flow losses when labels exist and specifies 80K/50K/30K stages, optimizer settings and batch size.
Go to primary source ↓e-instructorPDF p. 4, Learned Instructor; p. 7, Section 4.2; p. 19, Appendix C.2, Reward table, Training procedure and Ablations
The action policy samples five prompt conditions uniformly. The Qwen3.5-2B instructor reads three initial RGB camera frames and the task. Targets use per-task development SR, temperature 2 and top-three truncation with empty-target abstention. SFT uses 3.2K examples, four GB200 GPUs and batch 256; unsuccessful Top-1/DPO alternatives lack scores.
Go to primary source ↓e-asyncPDF p. 4, Figure 2; p. 5, deployment paragraph; p. 7, Section 4.2, Async deployment; p. 8, Table 3 and caption
The figure branches observations/task into instructor and policy, merging the annotation into the continuing rollout. The text specifies next-chunk injection at an eight-step cadence. Table 3 gives sync/async SR 49.5%/49.0%, async mean/median step 21.9/23 and time 1.87/1.86 seconds; its sync time entry is zero despite the caption's latency definition.
Go to primary source ↓e-evalPDF p. 5, Evaluation protocol; p. 17, Appendix B.2; p. 18, Table 6
RoboCasa uses a simulator task checker; Table 2 VLA results use five seeds times twenty episodes per task. Test seeds are 42–46 with 1,700 total trials, maximum 400 steps. MolmoSpaces has nine test benchmarks, episodes 20–199, 1,620 trials, maximum 607 steps and sustained success for 0.5 seconds. Both list horizon eight and 256×256 cameras. Development RoboCasa lists seeds 123–127, twenty episodes per unit and 340 total trials without clarifying allocation.
Go to primary source ↓e-resultsPDF p. 6, Table 2, VLA/WAM rows and RoboCasa/MolmoSpaces Avg. columns, caption and Section 4.1
VLA baseline/instructor/oracle averages are .44/.49/.52 on RoboCasa and .44/.49/.51 on MolmoSpaces. WAM averages are .18/.19/.20 and .16/.20/.19. MolmoSpaces summaries use Std/Hard for Pick and P+P, ID for NextTo and Color unchanged; their four-family average differs from Table 7's family summaries.
Go to primary source ↓e-motionPDF p. 6, Table 2, DeMiAn VLA / SlideDishwasherRack column; Section 4.1, The oracle gap
Task-only, Physical Motion, Scene Composition, Arm Pose and Reasoning yield .38, .75, .65, .46 and .50 respectively. Physical Motion is the best fixed RoboCasa average at .46.
Go to primary source ↓e-devPDF p. 19, Table 8 and caption
All development conditions share one action checkpoint. SR is 44.3% task-only, 46.1% motion, 48.4% scene, 47.4% pose, 50.1% reasoning, 46.6% random, 50.4% SFT instructor and 52.4% oracle; no uncertainty is tabulated.
Go to primary source ↓e-oodPDF p. 18, Table 7, VLA baseline/instructor rows and OOD/Avg. columns
VLA baseline/instructor OOD results are .18/.28 for Pick, .22/.27 for Pick+Place and .07/.13 for NextTo. Pick+Place averages only Std/Hard and is .64/.60. Pick averages Std/Hard/OOD; NextTo averages ID/OOD.
Go to primary source ↓e-compositePDF p. 8, Section 4.3 and Tables 4–5; p. 9, continuation of composite-task discussion
Eighteen composites use twenty episodes per task/configuration and 1,200 maximum steps. Dynamic prompts use simulator subgoal conditions and a done-flag threshold. Table 5 full-task SR is 13%/15% fixed baseline/DeMiAn and 19%/22%/18% dynamic baseline-GT/DeMiAn-GT/instructor. Instructor phase milestones are 61%/30% versus baseline-GT 57%/26%. The prose's 'between' characterization conflicts with the full-task numbers.
Go to primary source ↓e-scalingPDF p. 9, Section 4.4, Figure 5 and Scaling under matched compute
WAM mid-training uses 50K EgoVerse clips without action labels/head before RoboCasa post-training. Figure 5 compares baseline and annotation-conditioned compute curves; the VLA panels use the full 1M MolmoBot corpus. The text estimates 62% less compute and 1.3×10²⁰ FLOPs saved on NextTo/Color. Error bars are drawn without an identified statistical definition.
Go to primary source ↓e-costPDF p. 16, Appendix A.3, Annotation Cost Calculation
The estimate uses 8.2K input tokens, 150 output tokens and 3B active parameters: approximately 5.0×10¹³ FLOPs per call and 5.0×10¹⁹ per one-aspect million-clip pass. Representative hosted pricing gives about $1.1K per aspect, with all four aspects roughly four times larger.
Go to primary source ↓e-limitsPDF p. 15, Limitations and future work
The source states single-frozen-VLM bias, imperfect aspect selection, simulation-only experiments, a heuristic fixed schema and future joint instructor-policy optimization.
Go to primary source ↓e-attentionPDF pp. 6–7, Section 4.1 and Figure 4/caption
At step 42 of a CloseToasterOvenDoor rollout, the baseline concentrates attention on the beginning token, whereas the motion-caption policy distributes attention over objects, verbs and directions. The image's generic 'Reasoning:' prompt field coexists with the caption's Physical Motion condition; this is one displayed example.
Go to primary source ↓e-consistencyPDF p. 3, Section 2.2, final paragraph; p. 5, Figure 3 and Policy models; p. 17, Appendix B.1
Section 2.2 describes captions as required only during training, unlike the deployed instructor workflow. Main-text WAM decoding is called deterministic and the diagram shows noise entering DiT but no noisy action input; Appendix B.1 explicitly specifies noisy action conditioning and rectified-flow action training. An inference solver reconciling these descriptions is not given.
Go to primary source ↓e-oraclePDF p. 6, Table 2 and Section 4.1; p. 18, Table 7, VLA oracle and instructor rows
Table 2 defines an oracle over fixed training conditions, excluding the instructor. Table 7 VLA oracle includes values uniquely attained by the instructor, including Pick Std .80, Pick OOD .28, Pick+Place OOD .27 and Color .48; its WAM oracle does not follow the same inclusion pattern.
Go to primary source ↓8.5 Primary sources
How to Instruct Your Robot: Dense Language Annotations Power Robot Policy Learning ↗
PDF · 11,246 extracted words
Source fingerprint
7b4bdc06670ebe03b51b80b16099f4f570b875db899abf094b9644cbc2a64cc8