Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots
1. Paper overview
In one sentence: UMI makes portable handheld demonstrations transferable by matching sensing, action coordinates and execution timing, while remaining limited by robot feasibility and the coverage of the collected data. e02-probleme03-hardwaree05-action-recoverye07-policye08-latencye10-cupe12-tossinge13-foldinge14-washinge15-generalization
| At a glance | What to know |
|---|---|
| Research problem | Source description Robot teleoperation restricts where and how demonstrations can be collected, while ordinary human video lacks directly transferable robot actions. UMI asks which visual, motion and timing information a portable gripper must preserve so behavior cloning can transfer dynamic and bimanual skills. e02-problem |
| Core mechanism | Source description A GoPro-equipped handheld gripper combines wide visual context, mirror viewpoints, inertial pose recovery and continuous width tracking. A matching robot attachment reduces observation differences between teaching and deployment. e03-hardwaree04-mirrorse05-action-recovery |
| A key reported result | Dynamic tossing with latency matching: 105/120 (87.5%). Correct-bin success per object. 280 demonstrations; six objects sorted into bins beyond reach; matched collection-environment tests. Same checkpoint with measured latencies set to zero: 69/120 (57.5%). A 30-percentage-point improvement isolates the combined inference timing intervention. It does not separate observation from execution compensation. e12-tossing |
| Reading caution | Source description Robot feasibility still depends on deployment geometry and filtering. Texture-poor environments challenge SLAM, and the gripper's weight, bulk and limited dexterity reduce efficiency relative to human hands. e18-limitations |
Core contributions
- Source description
A GoPro-equipped handheld gripper combines wide visual context, mirror viewpoints, inertial pose recovery and continuous width tracking. A matching robot attachment reduces observation differences between teaching and deployment. e03-hardwaree04-mirrorse05-action-recovery
- Source description
Relative trajectories and inference-time latency matching remove key coordinate and timing mismatches. The paper validates their importance through executed robot tasks and targeted ablations. e07-policye08-latencye09-relativee11-cup-ablationse12-tossing
Figure 2. Matching the physical observation interface makes handheld demonstrations usable for robot learning. Original paper, p. 2 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read from the handheld gripper on the left to the robot attachment on the right. Both place the camera in the same relationship to the fingers, and the two central images show the resulting observation similarity. The numbered labels identify complementary functions: the fisheye lens supplies context, mirrors supply additional viewpoints, the IMU supports pose recovery, and visible finger markers support continuous width tracking. The final label concerns selecting demonstrations compatible with robot kinematics. These labels describe a collection pipeline; they are not separate neural prediction heads. Sections HD1–HD6 explain how these physical and processing choices become policy inputs and action labels. e03-hardwaree05-action-recoverye06-feasibilitye10-cup
What it supports. The contribution is an interface that preserves information needed for transfer. The camera records the scene while motion and width measurements recover actions. The matching deployment attachment reduces the gap between demonstration images and robot images, making portable collection useful for learning actual robot behavior.
Where the evidence stops. Visual similarity does not guarantee robot feasibility. Demonstrations still require filtering for the intended embodiment, and the same cup checkpoint encountered joint limits on Franka. The drawing therefore supports an interface design, not unrestricted hardware compatibility.
2. Motivation
2.1 The problem and the proposed response
Robot teleoperation restricts where and how demonstrations can be collected, while ordinary human video lacks directly transferable robot actions. UMI asks which visual, motion and timing information a portable gripper must preserve so behavior cloning can transfer dynamic and bimanual skills. e02-problem
2.2 What this reading follows
A useful demonstration must tell a robot what it saw, how the gripper moved and when that motion should occur. UMI organizes those requirements around a handheld gripper whose camera view resembles the deployed robot's view. Visual-inertial tracking and finger markers recover action labels; a diffusion policy learns desired motion sequences; deployment compensates for sensing and actuator delays. The experiments test physical cup arrangement, tossing, folding and washing, then examine cup transfer to two unseen locations. Read the evidence as a study of demonstration interfaces and policy transfer: it does not establish a learned model that predicts future world observations. e02-probleme03-hardwaree05-action-recoverye07-policye08-latencye10-cupe12-tossinge13-foldinge14-washinge15-generalization
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Datasets |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
3.1 Evidence-based assessment
Supports the recorded classification
The recorded data-collection categories fit UMI as infrastructure for acquiring transferable manipulation demonstrations, although this publication is a method paper rather than a standalone dataset specification. Its learned policy predicts actions; SLAM recovers demonstration poses. No joint future-observation/action predictor or learned inverse-dynamics controller is established, so the world-action architecture and quadrant fields appropriately remain not applicable. e02-probleme05-action-recoverye07-policy
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 First make human motion into measurable robot supervision
Begin with a person squeezing a trigger and moving the handheld gripper. The useful training record is richer than an ordinary video: the camera keeps the same relationship to the fingers that it will have on the robot, while its inertial measurements help recover metric motion. The fisheye lens preserves surrounding context, and side mirrors expose additional viewpoints. Marker-based width tracking records continuous opening rather than a binary grasp state. The best mirror treatment also digitally reflects and swaps the mirror regions, avoiding inconsistent object orientation across views. For bimanual collection, both devices relocalize into one scene map so their relative pose can be computed. Finally, feasibility filtering connects portable human demonstrations to the robot that will execute the learned behavior. e03-hardwaree04-mirrorse05-action-recoverye06-feasibility
Figure 5. Align what the policy sees and when the robot reaches each predicted target. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start in panel (a): images, gripper measurements and arm measurements arrive at different rates and with different delays. Green alignment paths and interpolation place them at common observation times before panel (b) passes their history to Diffusion Policy. The model outputs desired end-effector poses and gripper widths. Panel (c) separates desired poses, action commands and actual motion: commands must precede their desired execution times. Read the 100 ms arm and 120 ms gripper delays as values illustrated in this schematic; the method requires measuring each deployment. Section PD1.2 further explains that predictions already overtaken by accumulated delay are discarded. e07-policye08-latencye12-tossinge19-scope
What it supports. Timing compensation is part of deployment, not an extra learned world model. With the same trained tossing checkpoint, enabling the full matching procedure changes correct-bin performance from 69/120 to 105/120 objects. That is direct evidence that interface timing affects physical execution even when policy weights stay fixed.
Where the evidence stops. The intervention zeros all measured observation and action delays together. Its 30-percentage-point effect cannot be assigned solely to camera synchronization or gripper release timing. The referenced calibration appendix is absent from the supplied PDF.
5.2 Then preserve the meaning of an action across space and time
The policy sees synchronized image and proprioception histories and returns a sequence of desired end-effector poses and gripper widths. Two interface choices keep that sequence meaningful after transfer. Spatially, all poses in one action chunk share the current gripper pose as a reference, so the collection scene does not need the deployment robot's global coordinates. Temporally, each target belongs to a desired execution time. Delayed images must be paired with proprioception from the same physical moment, and commands must be sent early enough for each actuator to reach its target. Predictions that are already outdated are discarded. This is a feedback control interface around an action-generating policy; the learned output is not a simulated future image or a world-state rollout. e07-policye08-latencye09-relative
Figure 6. A relative action chunk shares one current-pose reference before the next inference resets it. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use the legend to separate three representations of the same trajectory. Red points require a global reference for absolute poses. Yellow arrows connect successive poses, corresponding to delta actions. Blue dashed spokes share an anchor at the current inference start, first at time zero and then at time four. The intended definition is given in PD2.1: each desired pose at time t is an SE(3) transform relative to the initial pose at t0. This common reference is the central distinction from repeatedly adding incremental actions. Historical proprioception uses relative poses as well; bimanual coordination additionally needs the pose of one gripper relative to the other. e09-relativee10-cupe11-cup-ablations
What it supports. The representation targets a practical collection problem: different scenes need not share a precisely calibrated robot-base frame. In the cup comparison, relative actions succeed 20/20, delta actions 16/20 and absolute actions 5/20. The authors associate the absolute baseline's weakness with calibration error, so the result is specific to that implementation.
Where the evidence stops. The short blue arrow between times four and five appears to point back toward the anchor. PD2.1 defines the desired pose relative to the starting pose; we follow that text and do not infer transform inversion from the schematic arrowhead.
5.3 Read transfer and mechanism evidence at different levels
The clearest intervention keeps the tossing checkpoint fixed and disables latency matching at inference, reducing correct-bin success from 105/120 to 69/120. My reading is that this directly identifies a deployment-interface failure, although it does not identify which delay source dominates. The folding ablation instead removes an observation feature during both training and evaluation, while the washing comparison changes encoder architecture and pretraining together. Generalization asks a broader question: training with 1,400 demonstrations from varied locations and cups transfers to two new environments, whereas the narrow-domain comparison fails there. Because demonstration quantity and variety change together, that result supports the collection regime without isolating its causal ingredients. The two proposed checks below separate timing components and collection diversity more carefully. e12-tossinge13-foldinge14-washinge15-generalization
5.4 Training and inference
During training
Narrow-domain training uses 305 cup, 280 tossing, 250 folding and 258 washing demonstrations. Washing fine-tunes a CLIP-pretrained ViT-B/16; broader cup training uses ViT-L/14 with 1,400 demonstrations from 30 locations and 15 cups. Encoder freezing rules for all tasks and full optimization/compute details are unavailable here. e10-cupe12-tossinge13-foldinge14-washinge15-generalizatione19-scope
During inference
Measure observation-stream delays and interpolate proprioception at image capture timestamps. Feed synchronized histories to the policy, compensate arm/gripper execution delays separately, discard already outdated predictions and send still-timely commands. Subsequent observations provide feedback; the model predicts desired actions, while robot controllers execute them. e07-policye08-latency
5.5 Implementation flow
- Collect observations that transfer
Mount the camera rigidly relative to matching soft fingers. Use raw 155-degree fisheye images; peripheral mirrors add viewpoints. Digital reflection plus left/right mirror swapping gives the best cup result, rather than simply passing mirror pixels unchanged. e03-hardwaree04-mirrors
- Recover usable action labels
Visual-inertial ORB-SLAM3 tracking recovers metric camera motion from video and IMU. Fiducials recover continuous finger width. Each scene is mapped before demonstrations are relocalized; the shared coordinates make independently tracked grippers comparable. Filter demonstrations for the intended robot's feasibility. e05-action-recoverye06-feasibility
- Represent motion around the current gripper
Each desired SE(3) pose is relative to the sequence's initial end-effector pose, not the previous predicted step. Historical poses use the same local principle; two observations convey velocity. Bimanual policies also receive the relative pose between grippers. e09-relative
- Learn action sequences
Diffusion Policy maps synchronized observation histories to pose/width sequences and accommodates multimodal demonstrations. The supplied paper names this policy family but does not specify a complete diffusion loss, denoising schedule or optimizer configuration. e02-probleme07-policye19-scope
6. Experiments & results
UMI turns handheld gripper demonstrations into deployable visuomotor policies by matching camera geometry, recovering metric actions and aligning robot timing. Its contribution is a collection-and-control interface with real robot evaluations, rather than a learned predictor of future world observations.
6.1 Read the original evidence
Figure 9(c). Diverse demonstrations transfer to two new locations, with separate counts for familiar and unseen cups. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. The location columns report successes over trials; the final column pools locations within each row. Training Cup means the cup appeared during training, even though the environment did not. Unseen Cup means both object identity and environment are held out. Add 16/25 and 12/15 to obtain 28/40 for familiar cups; add 6/10 and 9/10 to obtain 15/20 for unseen cups. The overall counts are 22/35 at the cafe and 21/25 at the fountain. The top row is a separate model using narrow-domain demonstrations with the same pretrained backbone; it has fewer evaluation trials than the diverse-data model. e15-generalization
What it supports. The pooled diverse-data result is 43/60, reported as 71.7% in Section VI and rounded to 0.72 in the table. The narrow-domain comparison obtains zero successes in ten trials at each location. This establishes successful transfer beyond the collection locations for this cup task and these two test environments.
Where the evidence stops. The training comparison changes demonstration count, environment variety and cup variety together. It supports the value of the broader collection regime, but does not isolate diversity at fixed dataset size or establish generalization to an arbitrary unseen environment.
Figure 12, benchmark setup and error table. Action-label accuracy is measured against motion capture independently of policy success. Original paper, p. 11 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. The photograph identifies external motion-capture equipment used for the benchmark, not a required observation source during normal UMI collection. Read the two columns as distinct units: translation in millimeters and rotation in degrees. The first row is mean absolute trajectory error for one gripper. The second measures relative pose error between grippers, which matters when a centralized bimanual policy needs their spatial relationship. Section VII reports seven single-gripper and seven bimanual benchmark tasks. These statistics assess recovered motion labels; they are not measured robot endpoint errors, maximum tracking deviations or success probabilities for downstream manipulation. e05-action-recoverye17-accuracye18-limitations
What it supports. Metric visual-inertial tracking provides mean per-gripper errors of 6.1 mm and 3.5 degrees, while inter-gripper estimates have mean errors of 10.1 mm and 0.8 degrees. This supplies quantitative support for using tracked demonstrations and shared-map gripper relationships as policy training information.
Where the evidence stops. The caption summarizes tracking as below one centimeter, but the inter-gripper entry is 10.1 mm. Retain that exact value. Means also leave worst-case errors and failure rates unresolved, particularly in texture-poor environments identified by the authors.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Cup arrangement in the collection environment 305 training episodes; 20 manually matched initial states per comparison; upright placement with handle within ±15 degrees of robot-left. | UR5: 20/20 (100%); same checkpoint on Franka FR2: 18/20 (90%). Episode success | Narrow view 11/20; absolute actions 5/20; delta actions 16/20; direct mirror images 17/20; no mirrors 18/20. Supports interface choices within this task. Absolute-action calibration is a confound; Franka failures hit joint limits. Figure 7 says reorient right, while the task prose specifies robot-left. e10-cupe11-cup-ablations |
| Dynamic tossing with latency matching 280 demonstrations; six objects sorted into bins beyond reach; matched collection-environment tests. | 105/120 (87.5%). Correct-bin success per object | Same checkpoint with measured latencies set to zero: 69/120 (57.5%). A 30-percentage-point improvement isolates the combined inference timing intervention. It does not separate observation from execution compensation. e12-tossing |
| Bimanual cloth folding 250 demonstrations from two people; 20 randomized sweater configurations in the collection environment. | 14/20 (70%). Complete-task success | Without inter-gripper pose during training and testing: 6/20 (30%). Supports explicit relative pose for coordination; the feature is removed in both training and deployment. e13-folding |
| Dish washing 258 demonstrations including recovery from added ketchup; seven sequential operations. | CLIP-pretrained ViT-B/16 policy: 14/20 (70%). Complete-task success | ResNet-34 trained from scratch: 0/10 (0%), using the task text's denominator. Backbone architecture and pretraining both change. Figure 8's generic 20-episode caption does not describe every baseline; Figure 10 robustness examples remain qualitative. e14-washinge20-robustness |
| Cup arrangement in unseen environments 1,400 demonstrations; unseen cafe table and water fountain; 40 training-cup and 20 unseen-cup trials. | 43/60 (71.7%); training cups 28/40 (70%); unseen cups 15/20 (75%). Episode success | Same pretrained backbone with narrow-domain data: 0/10 at each location. Demonstrates transfer to two held-out locations. Dataset size and diversity change together; evaluation denominators differ. The table rounds overall success to 0.72. e15-generalization |
| Demonstration throughput Same operator, 15-minute sessions including resetting, randomization and robot faults. | Authors report over 3× teleoperation throughput for cups. Collection throughput comparison | Space-mouse teleoperation produced no successful tossing demonstration during its session. A comparison to one teleoperation interface, not all teleoperation methods. Inconsistent Figure 11 labels prevent a reliable absolute hourly-rate reading. e16-throughput |
| SLAM action-label accuracy Motion-capture benchmark with seven single-gripper and seven bimanual tasks. | ATE: 6.1 mm, 3.5 degrees; RPE: 10.1 mm, 0.8 degrees. Mean absolute trajectory error (ATE) and inter-gripper relative pose error (RPE) | Motion-capture ground truth; no competing tracker reported. These are mean tracking errors, not policy precision guarantees. The caption's less-than-1-cm summary excludes the 10.1 mm RPE value. e17-accuracy |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 8. Different tasks expose different failure points in the observation-to-action interface. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each row horizontally: initial-state overlays establish the test variation, middle images illustrate failures, and charts or tables quantify outcomes. In the cup row, reduced field of view, alternate action coordinates and mirror processing are separate comparisons. The tossing row pairs a velocity trace with per-object success; its green trace denotes latency matching and red denotes its removal. Folding tests the inter-gripper pose input, while washing compares visual encoders. Use the overall column for complete-task outcomes. Substep rates are separate reported diagnostics: their non-monotonic values should not be treated as cumulative survival probabilities or multiplied into an overall success rate. e10-cupe11-cup-ablationse12-tossinge13-foldinge14-washing
What it supports. Tossing improves from 0.575 to 0.875 with matching; folding improves from 0.30 to 0.70 with inter-gripper pose; washing reaches 0.70 with the pretrained ViT configuration versus 0.00 for the scratch ResNet. These comparisons support distinct interface and representation choices, rather than one universal explanation for all task failures.
Where the evidence stops. The caption broadly says 20 episodes, but tossing is 120 object outcomes and the washing ResNet baseline is 0/10 in the text. Washing changes architecture and pretraining together. No uncertainty intervals or training-seed variation are reported.
7. Analysis & limitations
7.1 What the evidence leaves open
Robot feasibility still depends on deployment geometry and filtering. Texture-poor environments challenge SLAM, and the gripper's weight, bulk and limited dexterity reduce efficiency relative to human hands. e18-limitations
Reported success fractions have no confidence intervals or training-seed variation. Small task-specific evaluations and two unseen cup environments do not establish arbitrary-environment or arbitrary-robot transfer. e10-cupe12-tossinge13-foldinge14-washinge15-generalization
7.2 Questions for discussion
- How much of the tossing improvement comes from observation synchronization versus actuator-delay compensation?
- Does collection diversity still help when demonstration count and encoder initialization are held constant?
8. Reproducibility audit
8.1 Requirements and known gaps
Reproduction requires matching camera/finger geometry, fisheye and mirror preprocessing, visual-inertial mapping, width tracking, robot feasibility filtering and measured stream/actuator delays. The stated hardware is a 780 g gripper with 80 mm stroke; costs are $73 plus $298 for camera/accessories, excluding robots. e03-hardwaree04-mirrorse05-action-recoverye06-feasibilitye08-latency
A faithful rerun needs the missing appendix calibration and evaluation protocols, filtering thresholds, training horizons, diffusion settings and compute details. The paper states hardware/software are open-sourced, but that statement alone does not establish an inspected or reproducible implementation. e01-identitye02-probleme19-scope
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Separate observation matching from execution matching
Reader-proposed check, not performed: use one frozen tossing checkpoint and the same six objects and initial-state sets in four conditions: both compensations enabled, observation matching only, execution matching only, and neither. Measure each stream and actuator delay first, retaining those measurements while selectively disabling each correction. Record per-object correct-bin success, command-versus-motion timestamps, release timing and the elbow velocity trace. Repeat matched sets and report paired uncertainty. If one component alone recovers nearly all of the full condition's advantage, it is the dominant mechanism; if both are required, the interaction supports the combined interface. The missing calibration appendix must be resolved before calling this a faithful reproduction. e08-latencye12-tossinge19-scope
Check 2: Test location diversity at equal demonstration count
Reader-proposed check, not performed: collect two 305-demonstration cup datasets with matched cups, demonstrators and action/pose distributions, but use one location for one dataset and multiple locations for the other. Train the same ViT-L/14 policy configuration from identical pretrained initialization with matched optimization budgets and several seeds. Evaluate both on exactly the same held-out locations, cups and initial states, reporting familiar-cup and unseen-cup outcomes separately. A repeatable advantage for the varied-location dataset would support a diversity effect at fixed quantity. If the advantage appears only after increasing the count, the original broad-data comparison cannot be explained by location diversity alone. e10-cupe15-generalizatione19-scope
8.3 Reading coverage
Visual audit: All six supplied text chunks were read individually. All 14 original PDF pages were rendered and visually inspected, including the title/author block, Figures 1–12, method and training text, all result tables and the ending references. Six final original crops were separately viewed. Figure 6's short blue arrow was checked against PD2.1; Figure 8 denominators against task text; Figure 9 rounding against Section VI; and Figure 12's caption against its table. Figure 7's right/left wording and Figure 11's bar-label/axis inconsistency are disclosed in the base report. Appendices A–D, referenced in the paper, are absent from the supplied artifact. No separate videos, code or supplements were inspected.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14. Appendix coverage: not present.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Title, authors and Abstract (p. 1)
- I. Introduction (pp. 1–2)
- II. Related Works, A–C (pp. 2–3)
- III. Method, A. Demonstration Interface Design, HD1–HD6; B. Policy Interface Design, PD1–PD2.3 (pp. 3–6)
- IV. Evaluations (p. 6)
- V. Capability Experiments, A–D (pp. 6–10)
- VI. In-the-Wild Generalization Experiments (p. 10)
- VII. Data Collection Throughput and Accuracy (pp. 10–11)
- VIII. Limitations and Future Works; IX. Conclusion (p. 11)
- Acknowledgments and References (pp. 12–14)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Identity/version: the inspected title and all eight authors match the catalog. This is the supplied RSS 2024 proceedings artifact; no separate revision identifier or revision chain is supplied, so equivalence to other editions is not asserted.
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout. This acquisition limitation was addressed by visually inspecting all 14 PDF pages and six final original crops.
- Separate supplemental material availability has not been fully verified.
- Referenced appendices A–D are absent from this supplied PDF. Calibration procedures, detailed collection/evaluation protocols and mapping details delegated to them remain unreviewed.
- Code, project videos and separately linked resources were not inspected; no experiments were reproduced.
- Figure 11 has inconsistent printed bar values and CPH-axis heights; an absolute hourly throughput is not inferred. Source discrepancies in Figures 6, 7, 8 and 12 are disclosed alongside the affected claims.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01-identityPDF p. 1, title, author block, equal-contribution note and proceedings header
The exact title and eight catalog authors match the RSS 2024 title page. Cheng Chi and Zhenjia Xu have equal-contribution marks. The header identifies Delft, Netherlands, July 15–19, 2024; no separate revision number is printed.
Go to primary source ↓e02-problemPDF pp. 1–3, Abstract, Sections I–II and beginning of Section III
UMI combines handheld data collection and policy learning to reduce the observation/action embodiment gap while capturing dynamic, bimanual and long-horizon demonstrations without robots at collection time.
Go to primary source ↓e03-hardwarePDF p. 2, Figure 2; pp. 3–4, Figure 3 and Section III-A, HD1–HD2
A wrist-mounted GoPro and matching fingers align human and robot views. Raw 155-degree fisheye images retain central resolution and peripheral context; no external observation camera is required.
Go to primary source ↓e04-mirrorsPDF p. 4, Figure 4 and HD3; p. 8, Section V-A, Effect of side mirrors
Mirrors provide virtual viewpoints in the image. The best cup result uses digital reflection plus left/right mirror-image swapping: 20/20 versus 18/20 without mirrors and 17/20 when mirror images are provided directly.
Go to primary source ↓e05-action-recoveryPDF pp. 4–5, HD4–HD5; p. 6, PD2.3
ORB-SLAM3-based visual-inertial tracking uses synchronized GoPro video and IMU to recover metric poses; fiducials track continuous gripper width. Mapping each scene and relocalizing both grippers into the shared map enables relative inter-gripper pose.
Go to primary source ↓e06-feasibilityPDF p. 5, HD5–HD6 and Putting everything together
Soft fingers and continuous width control support compliant grasping. Demonstrations can be filtered using robot base location, kinematics and recovered poses. The gripper weighs 780 g, has 80 mm stroke, and costs $73 for the printed gripper plus $298 for camera/accessories.
Go to primary source ↓e07-policyPDF p. 5, Figure 5 and Section III-B
Diffusion Policy consumes synchronized RGB, end-effector pose and gripper-width histories and predicts desired pose/width sequences. All reported experiments use this policy family; ACT is only suggested as a possible alternative.
Go to primary source ↓e08-latencyPDF pp. 5–6, Figure 5, PD1.1–PD1.2
Observation streams are aligned to image capture times using measured delays and interpolated proprioception. Commands are advanced for actuator delay; actions already outdated by observation, inference and execution latencies are discarded. Figure 5 illustrates 100 ms arm and 120 ms gripper execution delays.
Go to primary source ↓e09-relativePDF p. 6, PD2.1–PD2.3 and Figure 6, diagram and caption
Each action sequence represents desired SE(3) poses relative to its initial pose, unlike stepwise delta actions or globally referenced absolute actions. Historical poses are relative too; a two-observation history provides velocity information. The short blue arrow between t=4 and t=5 appears directed toward the anchor, so its arrowhead alone should not determine transform direction.
Go to primary source ↓e10-cupPDF pp. 6–7, Section V evaluation protocol and Section V-A; p. 8, Figure 8 cup row
Capability tests reuse the collection environment with randomized, manually matched initial states. Cup training uses 305 episodes from two demonstrators; upright placement with handle within ±15 degrees of robot-left succeeds 20/20 on UR5 and 18/20 on Franka FR2 using the same checkpoint. Franka failures violate joint limits. Figure 7 labels reorientation to the right, whereas the task prose defines robot-left.
Go to primary source ↓e11-cup-ablationsPDF pp. 7–8, Section V-A, No Fisheye lens, Alternative action spaces, Effect of side mirrors; Figure 8 cup row
The rectified/cropped 69-degree-view baseline succeeds 11/20, delta actions 16/20, absolute actions 5/20, direct mirror images 17/20, and no mirrors 18/20. Authors attribute the absolute-action weakness to likely calibration error; the comparison does not establish intrinsic inferiority under accurate calibration.
Go to primary source ↓e12-tossingPDF p. 8, Figure 8 tossing row and Section V-B; p. 9, No Latency Matching
Training uses 280 demonstrations. Six objects are sorted into shape-appropriate bins beyond reach. UMI succeeds on 105/120 object tosses; setting all measured observation/action latencies to zero with the same checkpoint gives 69/120. The caption's generic 20-episode label must not replace the object-level denominator.
Go to primary source ↓e13-foldingPDF p. 7, Figure 7 folding row; p. 8, Figure 8 folding row; p. 9, Section V-C
A centralized bimanual policy trained on 250 demonstrations from two people folds sleeves, bottom hem, rotates the sweater and folds again. Overall success is 14/20; removing inter-gripper pose during training and testing gives 6/20, with asynchronous grasping described as a failure mode.
Go to primary source ↓e14-washingPDF p. 7, Figure 7 washing row; p. 8, Figure 8 washing row; pp. 9–10, Section V-D
The seven-step task uses 258 demonstrations from one person, including added-ketchup recovery. Fine-tuning a CLIP-pretrained ViT-B/16 with Diffusion Policy gives 14/20 overall success. The ResNet-34-from-scratch baseline gives 0/10 in the text, despite Figure 8's generic 20-episode caption.
Go to primary source ↓e15-generalizationPDF p. 9, Figure 9(c); p. 10, Section VI
Three demonstrators collect 1,400 demonstrations across 30 locations and 15 cups in 12 person-hours. The CLIP-pretrained ViT-L/14 policy reaches 28/40 on training cups and 15/20 on unseen cups in two unseen environments, totaling 43/60 (71.7%; table rounds to 0.72). The narrow-domain-data comparison uses the same pretrained backbone and scores 0/10 at each location.
Go to primary source ↓e16-throughputPDF pp. 10–11, Section VII, Throughput and Figure 11(d)
The same operator is assessed over 15 minutes, including resets, randomization and faults. Text reports UMI over three times faster than space-mouse teleoperation on cups; teleoperation produces no successful tossing demonstration. Figure 11's printed bar values and CPH-axis heights disagree, preventing an unambiguous absolute hourly-rate extraction.
Go to primary source ↓e17-accuracyPDF p. 11, Figure 12 table/caption and Section VII, Accuracy
On seven single-gripper and seven bimanual tasks with motion-capture ground truth, mean per-gripper ATE is 6.1 mm and 3.5 degrees, and inter-gripper RPE is 10.1 mm and 0.8 degrees. The caption's less-than-1-cm wording does not cover the 10.1 mm relative error.
Go to primary source ↓e18-limitationsPDF p. 11, Section VIII, Limitations and Future Works
Authors identify unknown deployment kinematic limits at collection time, texture-dependent SLAM, and the gripper's weight, bulk and reduced dexterity relative to human hands as limitations.
Go to primary source ↓e19-scopePDF pp. 5–8, appendix cross-references; pp. 11–14, conclusion, acknowledgments and references
Calibration details are referred to A1–A4, data collection to B, evaluation to C, and mapping details to D. The supplied 14-page artifact ends with references and contains none of those appendices; Section III-B names Diffusion Policy without a complete optimization or compute specification.
Go to primary source ↓e20-robustnessPDF p. 10, Figure 10 and caption; p. 9, Section V-D
Selected images illustrate base movement, novel objects, altered lighting and added-condiment recovery. They do not supply a numerical robustness benchmark or trial denominator.
Go to primary source ↓8.5 Primary sources
Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots ↗
PDF · 10,128 extracted words
Source fingerprint
7b964be9c247bea4e7afc2bd2fe4c60658fa0f2a1f048e453e5fcef73e6cdef0