A Careful Examination of Large Behavior Models for Multitask Dexterous Manipulation
1. Paper overview
In one sentence: Diverse action pretraining produces stronger task-specific manipulation policies, but its measured benefit depends on preprocessing, task-completion metrics and controlled evaluation. e-objectivee-architecturee-protocole-unseene-normalizatione-timeouts
| At a glance | What to know |
|---|---|
| Research problem | Source description Single-task imitation policies can perform dexterous manipulation yet remain brittle outside their demonstrations. The question is how much multitask pretraining contributes to data efficiency, success and robustness when architecture and evaluation conditions are controlled. Success videos alone cannot distinguish these effects from experimental noise. e-objective |
| Core mechanism | Source description A controlled evaluation pipeline combines repeated initial conditions, randomized blind hardware comparisons, milestone rubrics and statistical policy comparisons. The study reports 1,800 hardware and over 47,000 simulation rollouts. e-protocole-limitations |
| A key reported result | SetBreakfastTable with reduced task-specific data: LBM finetuned with 15% of task data significantly exceeds the single-task baseline trained with 100%. Mean rubric-based task completion and statistical comparison. Unseen hardware task, nominal evaluation; full task dataset contains 229 demonstrations; 50 rollouts per policy. The 50% and full-data LBM variants share CLD letter a; 15% is b and single-task is c. A measured demonstration-efficiency result for partial completion. Figure 5 does not print exact mean values, so none are reconstructed. e-efficiencye-real-taskse-protocole-metrics |
| Reading caution | Source description Confidence distributions describe fixed checkpoints, not training-seed variability. Bonferroni correction covers policy pairs within a task, not all tasks in a plot. Task-completion comparisons use an approximate Welch test despite discrete outcomes; aggregation also weakens the i.i.d. assumption. e-statisticse-limitations |
Core contributions
- Source description
A controlled evaluation pipeline combines repeated initial conditions, randomized blind hardware comparisons, milestone rubrics and statistical policy comparisons. The study reports 1,800 hardware and over 47,000 simulation rollouts. e-protocole-limitations
- Source description
The fixed architecture separates pretrained generalists, finetuned specialists and single-task baselines. On seen simulation tasks, finetuning significantly beats the baseline on 3/16 tasks nominally and 10/16 under distribution shift; aggregate gains do not imply improvement on every task. e-architecturee-seen
- Reader analysis
Pretraining-data and task-data sweeps support a tradeoff between broader prior experience and downstream demonstrations. These are empirical trends within the tested mixtures, rather than a fitted universal scaling law. e-scaling
Figure 9. The LBM generates action chunks from observations through iterative denoising. Original paper, p. 12 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow the arrows from the input box into the observation embedding, then from noisy action tokens below the transformer to denoised actions above it. The right inset shows observation-conditioned normalization and scaling around self-attention and an MLP. Section IV-B2 specifies a shared CLIP image encoder and a frozen CLIP text encoder with a learned projection; the schematic’s “ViT (Language)” label should be read alongside that more precise description. The photographs above the output illustrate action execution, not a trained future-image output. Equation (1) describes the repeated noise-prediction update, while the transformer drawing summarizes the resulting action transformation. e-architecturee-observationse-deployment
What it supports. A common action-denoising architecture serves the pretrained, finetuned and single-task policies. It conditions on two observation timesteps and produces 16 steps of 20-dimensional bimanual actions. This makes the key comparison one of training experience within the same architecture; a single transformer does not establish a joint world-and-action predictor.
Where the evidence stops. The caption specifies execution of eight steps before replanning. Section IV-D1 calls the horizon 15, while the drawing indexes through t+15; inclusive indexing is a plausible interpretation, not an explicitly documented resolution. No numerical DDIM step count is supplied.
2. Motivation
2.1 The problem and the proposed response
Single-task imitation policies can perform dexterous manipulation yet remain brittle outside their demonstrations. The question is how much multitask pretraining contributes to data efficiency, success and robustness when architecture and evaluation conditions are controlled. Success videos alone cannot distinguish these effects from experimental noise. e-objective
2.2 What this reading follows
Teaching a robot a new dexterous task can require many demonstrations, and a successful video says little about how reliably the policy works. This paper asks whether a broad base of robot experience makes the next specialist easier to train. It keeps a diffusion-policy architecture fixed and compares training histories through blind hardware trials and repeatable simulation. Read the architecture first, then separate complete success from milestone progress in the results. The strongest lesson is about finetuned specialists, with important qualifications: generalist instruction following is brittle, a normalization bug affects the main models, and seemingly simple evaluation choices such as timeouts change the measured outcome. e-objectivee-architecturee-protocole-unseene-normalizatione-timeouts
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Foundational work |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
This table preserves the labels recorded at reading time. The current major category is VLA. View the current classification.
3.1 Evidence-based assessment
Supports the recorded classification
The recorded foundational/action-policy/evaluation placement is supported. This architecture denoises action chunks conditioned on observations; it does not jointly predict future world states and actions or infer actions from predicted futures. A single DiT and multitask training therefore do not establish a One Model world-action architecture. The recorded world-action architecture, prediction paradigm and quadrant remain Not applicable. e-architecturee-protocol
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 From a demonstration to a fresh action chunk
The learned object is a distribution over actions conditioned on observations. During training, a demonstrated action chunk is corrupted at a randomly selected diffusion step, and the DiT learns to predict the added noise. CLIP supplies visual and language features, but their adaptation differs: the image backbone is finetuned while the text backbone remains frozen behind a learned projection. At deployment, the model starts from Gaussian action noise and repeatedly applies the observation-conditioned denoising update. The resulting chunk describes 16 bimanual action timesteps; the robot executes eight before recomputing. Differential inverse kinematics and impedance control translate those targets into physical movement. This distinction explains why the output photographs in Figure 9 should not be treated as evidence of video prediction: the supervised target and deployed output are actions. e-architecturee-traininge-deployment
5.2 Separate transfer from the ability to finish everything
An unseen task is absent from multitask pretraining, but the specialist still receives demonstrations of that task during finetuning. The comparison therefore tests transfer into a newly trained specialist, not zero-shot execution. For a long task such as setting the breakfast table, success demands an entire sequence while the rubric credits intermediate milestones. This makes the two metrics answer different questions. Figure 4 shows stronger separation in task completion than in binary success, and Figure 5 asks whether this progress can be learned with fewer new demonstrations. Reader interpretation: the 15% result is compelling evidence of useful prior experience, but it neither eliminates the pretraining cost nor establishes reliable completion of every rollout. The nominal hardware setting and the particular milestone definition remain part of the finding. e-unseene-efficiencye-real-taskse-metricse-data
5.3 Treat preprocessing and evaluation as part of the experiment
Ramen mixes embodiments and sources whose action scales need careful alignment. The intended normalizer is feature- and timestep-specific; the discovered implementation error assigned some samples another source’s parameters. Figure S21 tests a corrected pretrained policy and finds the clearest aggregate gain under distribution shift. A separate filtering experiment removes low-motion prefixes: this reduces waiting but can increase wrong-task execution by the generalist. The authors hypothesize that ambiguous starting scenes help teach language conditioning when those prefixes remain. Reader interpretation: seemingly benign cleanup can change the learned behavior, so a reproduction must preserve and separately vary these choices. Measurement needs equal care. Table S6 shows that allowing more recovery time changes success rates even without retraining, and repeated evaluation cannot resolve the paper’s unmeasured training-seed uncertainty. e-preprocessinge-normalizatione-filteringe-timeoutse-limitations
5.4 Training and inference
During training
Ramen totals approximately 1,695 hours: 545 hours of TRI data and 1,150 hours of curated OXE data. TRI data comprise 468 hardware hours, 45 simulation hours and 32 UMI hours, totaling 64,262 demonstrations across 532 tasks. Table S7 supplies unnormalized mixture weights, normalized during training. e-datae-mixture
Pretraining uses 48,000 steps, batch 2,560 and constant learning rate 3×10⁻⁴. Finetuning uses batch 320 and 2×10⁻⁵ for 30,000 hardware or 10,000 simulation steps; single-task baselines use 100,000 or 25,000 steps respectively. Vision features are finetuned, with one-tenth the main pretraining learning rate; the text encoder stays frozen. Training resizes images to 256×342, randomly crops to 224×224 and applies color jitter. e-traininge-architecture
Feature/timestep percentile normalization maps the 2nd–98th percentile interval to −1…1 and clips at ±1.5; rotations bypass it. Main pretraining retains low-motion prefixes; simulation finetuning and baselines remove prefixes before motion exceeds 5 cm or 15 degrees. Real task-specific data remain unfiltered. e-preprocessing
During inference
Images are center-cropped after resizing. The policy produces 16 action timesteps, executes eight before recomputing, and reports a 10 Hz policy loop. Robot execution uses differential inverse kinematics with collision avoidance over a joint-impedance controller; observed feedback conditions subsequent chunks. e-deployment
Figure 9 and Section IV-B specify 16 action steps; Section IV-D1 separately calls the prediction horizon 15. The diagram indexes through t+15, suggesting an inclusive convention, but the source does not explicitly reconcile the wording. e-architecturee-observations
5.5 Implementation flow
- Standardize heterogeneous demonstrations
Observations include six RGB views, poses relative to the station and the other end effector, and gripper widths. Missing cameras or arms are zero-padded; unimanual examples randomly swap arm side. End-effector orientation uses six rotation-matrix components, with relative trajectories for control. e-observations
- Condition the action denoiser
A shared CLIP ViT-B/16 produces image CLS features. A CLIP text encoder supplies pooled end-of-sequence features through a learned projection. Two observation timesteps and encoded diffusion time condition eight DiT blocks of width 768 through adaptive layer normalization. The architecture is identical across policy regimes. e-architecturee-training
- Learn and sample action chunks
Training corrupts demonstrated actions and minimizes noise-prediction error. DDIM iteratively denoises Gaussian action noise conditioned on current observations. This is direct action generation: the specified pipeline contains no learned future-image prediction, inverse-dynamics stage or world-model planning loop. e-architecture
- Compare executed behavior
Hardware trials use randomized policy bundles with matched scene overlays; simulation policies share initial-condition seeds. Success rate measures completed tasks. Task completion credits achieved milestones using human rubrics or simulation-state predicates. e-protocole-metrics
6. Experiments & results
This study tests whether diverse robot-action pretraining improves specialist manipulation policies. A fixed diffusion-transformer architecture is pretrained on Ramen and finetuned on individual tasks, then compared with single-task training through controlled simulation and blind hardware trials. Benefits are clearest after finetuning and in partial task completion; language steering, preprocessing errors and evaluation protocols limit stronger conclusions.
6.1 Read the original evidence
Figure 4. Milestone completion exposes transfer benefits that all-or-nothing success can hide. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read vertically before comparing domains. The upper row reports success rates; the lower task panels show distributions of task completion, so their broad violins describe how differently individual rollouts progress. The rightmost column varies the available downstream demonstrations in simulation, and its task-completion violins describe uncertainty in the mean. Red is finetuned LBM and gray is single-task training. The numbers beside task names count demonstrations, not evaluation trials. CLD letters summarize the paper’s hypothesis tests: policies without a shared letter are separated. The hardware and simulated tasks occupy different columns because they are different task suites, not interchangeable measurements of one benchmark. e-unseene-metricse-statisticse-protocole-real-taskse-taskse-limitations
What it supports. Finetuned LBM has significantly higher mean task completion on four of five unseen hardware tasks and four of five nominal simulated tasks. Only two hardware tasks show significant success-rate separation. The distinction matters: completing more necessary steps can be measurable even when reliable end-to-end execution remains difficult.
Where the evidence stops. These are fixed-checkpoint comparisons. Shared letters do not prove equivalence, and error control is not global across all tasks. Section III-B notes that single-task SetBreakfastTable has zero empirical successes even though the Bayesian posterior has positive support.
Figure 5. A small downstream demonstration subset can produce a better breakfast-table specialist. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use the legend carefully because its colors differ from other figures. Gray is the single-task baseline using the full task dataset; teal uses 15% for LBM finetuning; maroon uses 50%; yellow uses all task data. The ordinate shows completion as a fraction from zero to one despite the percent label. Each violin represents the observed distribution of milestone completion, with a horizontal mean marker. The rubric includes opening the cabinet, placing items on the tray, keeping cups upright and pouring cereal. These milestones make the score interpretable as progress through this particular task, rather than a generic quality score. e-efficiencye-real-taskse-metricse-statisticse-data
What it supports. The 15% LBM variant receives letter b while the full-data baseline receives c, establishing the reported difference in mean task completion. The full task dataset contains 229 demonstrations. The 50% and 100% finetuned variants share letter a; that is a lack of detected separation, not proof that extra demonstrations never help.
Where the evidence stops. The experiment concerns one unseen hardware task under nominal conditions. The 15% refers to task-specific finetuning data and excludes the large pretraining corpus. Exact completion means and the rounded subset demonstration count are not printed in this figure.
Table S6. A stopping rule can change the apparent simulation-to-hardware performance gap. Original paper, p. 30 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Choose one task and policy row, then move from Sim to Real to Real with Sim Timeout. The final two columns reuse hardware behavior but apply different stopping criteria; they do not describe separately trained models. In the mug finetuned row, success is 0.465 in simulation, 0.88 under the original hardware procedure and 0.50 when the simulation timeout is imposed. The single-task mug row changes from 0.84 to 0.46 under the same rescore. The top three rows repeat this comparison for kiwi placement. Read across a row before attributing a domain gap to visual realism or better learned control. e-timeoutse-platform
What it supports. The mug example shows a large protocol effect: stricter timing reduces finetuned hardware success by 38 percentage points, bringing it close to simulation. Kiwi retains a larger difference, with finetuned success 0.09 in simulation and 0.70 on timeout-matched hardware, so stopping rules explain only part of the overall discrepancy.
Where the evidence stops. This table gives point estimates without uncertainty intervals. Matching the timeout does not match all dynamics, cameras or policy-specific finetuning data. Hardware operators originally allowed recovery until success, dangerous behavior or sustained lack of progress; nominal domain comparisons inherit that difference.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| PutKiwiInCenterOfTable Seen task, nominal hardware conditions; 49 task demonstrations and 50 rollouts per policy, with the original operator termination protocol. | Finetuned LBM: 0.82. Empirical success rate | Single-task: 0.44; pretrained LBM: 0.20. A 38-percentage-point gain over single-task training in this setting. Table S6 gives point estimates; its timeout-adjusted column is a different evaluation. e-timeoutse-real-taskse-protocole-seen |
| SetBreakfastTable with reduced task-specific data Unseen hardware task, nominal evaluation; full task dataset contains 229 demonstrations; 50 rollouts per policy. | LBM finetuned with 15% of task data significantly exceeds the single-task baseline trained with 100%. Mean rubric-based task completion and statistical comparison | The 50% and full-data LBM variants share CLD letter a; 15% is b and single-task is c. A measured demonstration-efficiency result for partial completion. Figure 5 does not print exact mean values, so none are reconstructed. e-efficiencye-real-taskse-protocole-metrics |
| Five complex unseen tasks in each domain Full task-specific data, nominal hardware and simulation conditions; simulation also evaluated under distribution shift. | Finetuned LBM wins on 4/5 hardware tasks and 4/5 simulation tasks in each simulation condition. Number of tasks with significantly higher mean task completion | Single-task baselines; nominal hardware success-rate separation appears on only 2/5 tasks. Partial progress reveals gains that binary success can miss; success rates remain low on challenging tasks. e-unseene-statistics |
| Pretraining-data sweep on five unseen Kitchen tasks Nominal simulation; compare full Ramen, TRI-only, 50% or 25% of TRI tasks, and single-task training at fixed downstream data fractions. | At 15% task-specific data, all five policy variants are statistically separated in order of increasing pretraining support. Aggregate mean task completion and CLD separation | Full Ramen performs best; single-task performs worst at that fraction. Task diversity and data volume change together. The experiment does not isolate a pure volume effect or establish model-size scaling. e-scaling |
| TurnMugRightsideUp timeout sensitivity Seen nominal task; original hardware rollouts rescored using the simulation timeout. | Finetuned LBM: 0.88 under original hardware termination, 0.50 with simulation timeout. Empirical success rate | Simulation: 0.465; single-task hardware: 0.84, falling to 0.46 with the same rescore. Stopping rules explain much of this apparent domain gap. This is a rescore, not a new policy or retraining result. e-timeouts |
| Corrected normalization on seen simulation tasks Supplemental pretrained-policy comparison under nominal conditions and distribution shift. | Corrected pretrained LBM improves on 4/16 tasks and in aggregate under distribution shift. Success-rate statistical separation | The original pretrained LBM with incorrect source normalizers; nominal aggregate performance is not significantly separated. This limited correction experiment does not replace the main finetuned or hardware evaluations. e-normalization |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 7. Broader pretraining changes how much specialist data is needed. Original paper, p. 10 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Hold the horizontal position fixed to compare pretraining mixtures. Maroon uses all Ramen data; teal removes OXE; green and yellow retain half and a quarter of TRI tasks; gray has no robot multitask pretraining. Then move along one curve to inspect the effect of downstream task demonstrations. Small violins describe uncertainty in the mean completion across five unseen Kitchen tasks under nominal simulation conditions. The dashed line marks the full-data single-task reference. At 15% downstream data the five different CLD letters show complete statistical separation of these variants. The absent gray point at zero reflects that no single-task policy can be trained without task data. e-scalinge-efficiencye-missinge-statisticse-tasks
What it supports. The largest pretraining mixture leads at the low-data setting, while the gap between full Ramen and TRI-only narrows with more downstream data. This supports the paper’s practical tradeoff: broad prior robot experience is particularly useful when acquiring demonstrations for a new specialist is difficult.
Where the evidence stops. Subsampling tasks changes diversity and data volume together. The curve is not a fitted power law, and crossings between tested fractions are interpolations. The caption’s 200 rollouts per task is a target; Table S4 lists missing fractional-pretraining rollouts.
Figure S21. A preprocessing correction changes robustness estimates without changing the policy family. Original paper, p. 39 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Compare teal LBM with blue LBM Fixed within each task before looking at the maroon finetuned specialist. Blue is the additional pretrained policy with corrected source-specific normalization, not a repaired version of every model in the paper. The upper panel uses nominal conditions; the lower panel introduces distribution shift. At the far right, aggregate labels expose a clearer improvement under shift than under nominal conditions. The gray and maroon reference policies place the correction in context, but the causal contrast being examined is the original versus corrected pretrained LBM. Read the task labels and letters together rather than comparing violin widths alone. e-normalizatione-statisticse-limitationse-preprocessing
What it supports. The supplement reports improved shifted success on four of 16 tasks and in aggregate for corrected pretraining. The nominal aggregate policies are not statistically separated. This diagnostic demonstrates why a data-normalization decision can matter to claims about robustness even when the architecture and overall learning paradigm stay the same.
Where the evidence stops. The main real-world and finetuned evaluations were not repeated after the bug was discovered. These supplemental results cannot be substituted for corrected versions of the headline experiments, and they do not quantify variability across independent training runs.
7. Analysis & limitations
7.1 What the evidence leaves open
Confidence distributions describe fixed checkpoints, not training-seed variability. Bonferroni correction covers policy pairs within a task, not all tasks in a plot. Task-completion comparisons use an approximate Welch test despite discrete outcomes; aggregation also weakens the i.i.d. assumption. e-statisticse-limitations
Some pretraining samples used another data source’s normalizer. Main results were not rerun. Filtering removes idling but can worsen wrong-task execution in pretrained generalists, so preprocessing is part of the scientific mechanism, not an interchangeable implementation detail. e-normalizatione-filtering
Finetuning can regress: TurnCupUpsideDown has nominal simulation success 0.335, with 89/200 rollouts never leaving the initial pose. Generalist language steering is brittle, and extension to larger language encoders remains untested here. e-failuree-limitations
Simulation and hardware shifts differ; human stopping rules permit recoveries that fixed simulation timeouts prevent. Missing simulation logs remove 104 rollouts across 39 combinations, reported as 0.2% of evaluation data. e-taskse-timeoutse-missing
Section VIII-A says five unseen simulation tasks, whereas Section IV-E and Table S2 enumerate eight: three short tasks plus five Kitchen tasks. The report follows that explicit enumeration without silently repairing the inconsistent sentence. e-task-count
7.2 Questions for discussion
- How much of the low-data advantage survives uncertainty across independent training runs?
- Can filtering reduce idling without weakening instruction selection in visually ambiguous scenes?
- How should success be compared when allowable recovery time differs across domains?
8. Reproducibility audit
8.1 Requirements and known gaps
Reproduction requires the Ramen mixture, source-specific normalizers, filtered/unfiltered variants, CLIP initialization, and the same training budgets. Table S7’s weights total 1.045 before normalization. The recipe does not specify an actual DDIM step count, full noise schedule, training accelerator budget or complete checkpoint-selection rule. The precise erroneous normalizer-assignment pattern is also not specified. e-traininge-mixturee-preprocessinge-architecture
Hardware reproduction needs two FR3 arms, compliant parallel grippers, calibrated cameras and the described controllers. Nine hardware stations supplied data, but only seven new-platform stations supplied reported hardware evaluations. The Drake-based evaluator, initial-condition distributions and exact rubrics/predicates are necessary to match measurement. e-platforme-deploymente-taskse-real-tasks
Reader-proposed checks: repeat the low-data specialist comparison across training seeds; independently compare original and corrected normalizers while holding filtering and evaluation seeds fixed. Both should report success, milestone completion and failure modes, as detailed in the illustrated edition. e-efficiencye-normalizatione-filteringe-limitations
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Does the 15% breakfast-table advantage survive training randomness?
Reader-proposed, not performed: train at least three independent seeds of the single-task full-data baseline and the LBM specialist using the same fixed 15% demonstration subset. Preserve the reported architecture, per-regime training budget, unfiltered hardware task data and rubric. Evaluate every checkpoint on the same 50 initial conditions using randomized blind bundles and predeclared termination rules. Report both success and all rubric milestones, with uncertainty across seeds as well as rollouts. A stable positive completion difference across seeds would support the central transfer claim; a reversal or effect smaller than between-seed variation would qualify the fixed-checkpoint result. This first check deliberately holds subset choice fixed rather than confounding subset and training randomness. e-efficiencye-real-taskse-traininge-preprocessinge-protocole-limitations
Check 2: Does source-normalizer correction improve the finetuned specialist too?
Reader-proposed, not performed: pretrain matched policy pairs using the original source-assignment error and the corrected normalization, with identical demonstrations, mixture weights, initialization seeds, training budgets and unfiltered prefixes. Finetune both on the same filtered seen-task simulation datasets, and include their unfinetuned checkpoints as controls. Evaluate nominal and shifted conditions with matched initial-condition seeds and fixed timeouts; record missing rollouts explicitly. Measure success, time to first motion and wrong-task execution, with repeated training seeds. Improvement confined to pretrained policies would limit extrapolation from Figure S21; a persistent advantage after finetuning would support extending the correction to specialists. This isolates normalization before a separate factorial study of filtering. Recover the erroneous assignment from the original implementation; if unavailable, label an injected mismatch as a sensitivity study rather than exact reproduction. e-normalizatione-preprocessinge-mixturee-traininge-taskse-filteringe-missinge-limitations
8.3 Reading coverage
Visual audit: Visually inspected the original title/version page and supplementary author list; all cited method, training, evaluation and reproducibility pages; the main results and data sweeps; hardware and simulation setup tables; task counts, rubrics and missing-rollout table; timeout comparison; filtering and normalization diagnostics. All six final original-PDF crops were separately viewed, including higher-resolution renders for narrow figures and Table S6. Figure 9’s arrow flow was checked against Eqs. (1)–(2) and its caption; the language-encoder shorthand and horizon wording are disclosed. Figures 1, S8–S9 and S14–S16 were not visually inspected; their captions were included in the complete text reading. No external videos or separate supplements were viewed.
PDF pages inspected for this edition: 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 22, 23, 24, 25, 26, 27, 30, 31, 32, 33, 37, 38, 39. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract and I. Introduction
- II. Related Work, A–C
- III. Results, A–C
- IV. Materials and Methods, A–E
- V. Discussion and Conclusion, including limitations
- References
- VI. Authors and contributions
- VII. Platform Details
- VIII. Simulation evaluation details, A–D
- IX. Real-world evaluation details, A–D
- X. Breakfast scenario analysis, A–C
- XI. Bayesian analysis for task completion
- XII. Effects of dataset filtering
- XIII. Additional pretraining dataset details
- XIV. Data normalization experiment
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- Version scope: the title and author identity match arXiv:2507.05331v1, dated 7 July 2025, on PDF p. 1. The catalog lists Science Robotics (2026); that publication was not supplied or compared. No equivalence of editions is asserted.
- Author spelling: Section VI credits Ben Burchfiel; the catalog expands this to Benjamin Burchfiel. Metadata preserves the inspected spelling. The title page credits TRI LBM Team; individual authors and affiliations are verified on PDF p. 22.
- All 13 supplied text chunks, covering all 39 PDF pages, were read. Text extraction does not reconstruct figure images; selected original pages and six final crops were separately inspected. Figures 1, S8–S9 and S14–S16 were read through their captions but not visually inspected.
- Separate supplemental material availability has not been fully verified. The embedded supplementary material, printed S1–S18, was supplied and read.
- The project website, code, datasets, videos and separate artifacts were not accessed. No training, policy execution or reproduction experiments were run.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e-identityPDF p. 1, title/byline/arXiv margin; PDF p. 22 (S1), Section VI and affiliation footnote
The exact title, TRI LBM Team credit and arXiv:2507.05331v1 date are present. Section VI lists individual authors including Ben Burchfiel, with Toyota Research Institute affiliation except Hadas Kress-Gazit at Cornell University.
Go to primary source ↓e-objectivePDF p. 1, Abstract and Section I; PDF p. 3, Section III opening
The study targets pretraining effects on data efficiency, task performance and distribution-shift robustness, using single-task comparisons.
Go to primary source ↓e-architecturePDF p. 12, Figure 9, Eqs. (1)–(2), Sections IV-B1–B2
Observation-conditioned DDIM uses a noise-prediction DiT; text features are frozen, vision is trainable, and eight width-768 blocks predict 16×20 action components.
Go to primary source ↓e-observationsPDF p. 15, Section IV-D1
The source defines six camera views, pose/gripper observations, relative action representation, six-dimensional rotations, zero padding, arm swapping and observation history two; it calls the action horizon 15.
Go to primary source ↓e-trainingPDF pp. 12–13, Section IV-B3; PDF p. 14, Table I; PDF p. 15, Table II
Training budgets, image augmentation, learning rates and CLIP ViT-B/16 are specified. Finetuning and baseline step budgets differ between real and simulated tasks.
Go to primary source ↓e-dataPDF pp. 13–15, Section IV-D
Ramen combines approximately 545 TRI hours and 1,150 OXE hours. TRI includes real, simulated and handheld UMI demonstrations with the stated counts and task exclusions.
Go to primary source ↓e-mixturePDF p. 33 (S12), Table S7; PDF p. 32 (S11), Section XIII
Batch weights are unnormalized: real 0.5, sim 0.25, UMI 0.05, plus listed external datasets; total 1.045 is normalized during training.
Go to primary source ↓e-preprocessingPDF pp. 15–16, Sections IV-D2–D3
Normalization uses feature/timestep percentiles and clipping, excluding rotations. A 5 cm or 15-degree threshold removes starting low-motion data for simulation finetuning and baseline training; real task data remain unfiltered.
Go to primary source ↓e-deploymentPDF p. 12, Figure 9 caption; PDF p. 13, Sections IV-B3 and IV-C1
Deployment center-crops images, executes eight of 16 action timesteps before recomputation, reports 10 Hz, and uses inverse-kinematics/joint-impedance control.
Go to primary source ↓e-platformPDF p. 13, Sections IV-C1–C2; PDF pp. 22–23 (S1–S2), Section VII and Table S1
Two FR3 arms form each station. The old gripper/camera platform is excluded from hardware evaluation; Table S1 lists nine hardware stations, seven used for evaluation, and two simulated stations. Simulation uses Drake.
Go to primary source ↓e-protocolPDF p. 3, Section III; PDF pp. 8–11, Sections IV-A1–A3 and Figure 8
Each policy/task/condition gets 50 hardware trials or a target of 200 simulation trials. Hardware uses randomized blind bundles and overlays; simulation uses matched seeds.
Go to primary source ↓e-metricsPDF pp. 10–11, Sections IV-A2–A4; PDF p. 27 (S6), Section IX-D
Hardware task completion averages binary achieved milestones. SR violins use Beta posteriors; per-task TC panels show outcome distributions, while fractional-data curves show Dirichlet-based uncertainty in the mean. The source distinguishes these displays.
Go to primary source ↓e-statisticsPDF p. 11, Sections IV-A4b–A5
CLD letters summarize pairwise tests, with within-task Bonferroni correction. Binary outcomes use sequential paired tests and task-completion means use Welch tests. The source discloses approximate assumptions and no global correction across tasks.
Go to primary source ↓e-tasksPDF p. 16, Section IV-E; PDF pp. 24–25 (S3–S4), Tables S2–S3 and Section VIII-C
The evaluation separates seen and unseen tasks. Simulation shifts mainly appearance/cameras; hardware tests station and object changes. Table S2 gives each Kitchen task 392 demonstrations; predicates depend on task-specific milestones.
Go to primary source ↓e-task-countPDF p. 16, Section IV-E; PDF p. 23 (S2), Section VIII-A; PDF p. 24 (S3), Table S2
Main text and task table enumerate 16 seen and eight unseen simulation tasks, whereas Section VIII-A says five unseen tasks. Table S2 includes three non-Kitchen unseen tasks and five Kitchen tasks.
Go to primary source ↓e-real-tasksPDF p. 27 (S6), Table S5 and Section IX-D; PDF p. 26 (S5), Figure S6
Table S5 records 49 kiwi and 229 breakfast-table demonstrations, labels the latter SetUpBreakfastTable, and specifies evaluation conditions. The breakfast rubric lists cabinet, tray, food and utensil milestones; Figure S6 shows successful task sequences.
Go to primary source ↓e-seenPDF pp. 4–5, Section III-A and Figure 2
Finetuned LBM improves aggregate seen-task results. It significantly exceeds baseline on 3/16 nominal simulation tasks and 10/16 under shift, while generalist-only results are mixed.
Go to primary source ↓e-failurePDF p. 4, Section III-A failure analysis; PDF p. 33 (S12), Figure S13(e)
TurnCupUpsideDown finetuned nominal success is 0.335 and 89/200 trials do not move before timeout. The histogram corroborates delayed or absent motion.
Go to primary source ↓e-unseenPDF pp. 6–7, Section III-B and Figure 4; PDF p. 9, Figure 6
On complex tasks, TC significantly improves on four of five hardware tasks and four of five simulated tasks in each condition. Nominal hardware SR separates on two tasks. SetBreakfastTable single-task empirical success is zero despite the posterior’s positive support.
Go to primary source ↓e-efficiencyPDF pp. 7–8, Section III-B and Figure 5; PDF p. 27 (S6), Table S5
The breakfast task uses 229 demonstrations in full. A 15% finetuned LBM exceeds the full-data single-task baseline in mean task completion; 50% and 100% LBM variants share letter a. The less-than-30% simulation equivalence is explicitly an interpolation.
Go to primary source ↓e-scalingPDF p. 8, Section III-C; PDF p. 10, Figure 7; PDF p. 25 (S4), Table S4 fractional-pretraining rows
The nominal five-task simulation sweep compares full Ramen, TRI-only and subsets of TRI tasks across finetuning fractions. All five variants separate at 15% downstream data. The target of 200 rollouts per task has listed missing-rollout exceptions.
Go to primary source ↓e-timeoutsPDF p. 26 (S5), Section IX-A ending a rollout; PDF p. 30 (S9), Section X-C and Table S6
Hardware termination is operator judged; simulation uses fixed timeouts. Table S6 lists Sim/Real/Real-with-Sim-timeout rates: kiwi finetuned 0.09/0.82/0.70 and mug finetuned 0.465/0.88/0.50, alongside baseline and pretrained policies.
Go to primary source ↓e-normalizationPDF p. 15, Section IV-D2; PDF pp. 32–33 (S11–S12), Section XIV; PDF p. 39 (S18), Figure S21
Main pretraining incorrectly assigned source normalizers to some samples. The main experiments were not rerun; the corrected pretrained policy improves shifted seen-task simulation on four of 16 tasks and in aggregate.
Go to primary source ↓e-filteringPDF pp. 15–16, Section IV-D3; PDF pp. 30–32 (S9–S11), Section XII; PDF pp. 37–38 (S16–S17), Figures S19–S20
Removing initial low-motion examples accelerates motion initiation but can worsen language steering in generalists. Finetuned aggregate performance is similar, with improvements for particular idling tasks.
Go to primary source ↓e-missingPDF p. 25 (S4), Section VIII-D and Table S4
The paper reports 104 missing simulation rollouts across 39 of 238 task/policy/condition combinations, approximately 0.2%; affected row counts are listed.
Go to primary source ↓e-limitationsPDF p. 17, Section V and Section V-A
Reported scale is 1,800 real and over 47,000 simulated rollouts. The authors acknowledge unmeasured training randomness, residual human/environmental noise, small CLIP language encoders and mixed generalist performance.
Go to primary source ↓8.5 Primary sources
A Careful Examination of Large Behavior Models for Multitask Dexterous Manipulation ↗
PDF · 23,361 extracted words
Source fingerprint
c9ba44579974208aa43d60fce2e0294c0fee1f1b8d604abd64613568f9c5a102