DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset
1. Paper overview
In one sentence: DROID makes varied real-world demonstration collection practical with a shared portable robot platform, improving average co-trained policy success while leaving zero-shot deployment and the isolated effect of scene diversity unresolved. e-probleme-resourcee-platforme-protocole-policye-maine-ablatione-discussion
| At a glance | What to know |
|---|---|
| Research problem | Source description Robot policies must tolerate changes in lighting, objects and workspace layout, but observation–action demonstrations are costly to collect outside controlled laboratories. DROID addresses this collection bottleneck through a shared portable platform and distributed human teleoperation. The research question is whether broader experience improves downstream policy reliability. e-probleme-platform |
| Core mechanism | Source description The dataset contains 76k successful demonstrations, approximately 350 interaction hours, 564 scenes, 52 buildings and 86 verb-defined tasks. Collection involved 50 people and 18 robots across 13 institutions over 12 months. Roughly 16k unsuccessful trajectories are additionally released but excluded from the headline count and policy co-training. e-resourcee-protocole-policy |
| A key reported result | Robustness to out-of-distribution changes: DROID 0.53 Reported average executed-task success rate. Seven plotted settings over six tasks: distractors, novel chips/objects, and a camera shift for lentil cooking; 10 rollouts per setting and method. OXE 0.36; no co-training 0.17. The reported 17-percentage-point gain is over OXE. The displayed average is consistent with averaging seven settings; chips contributes two variants. It is not a zero-shot evaluation without local demonstrations. e-taskse-evaluatione-main |
| Reading caution | Reader analysis All evaluations use the shared DROID embodiment and local demonstrations. Generalization across robot morphologies and deployment in entirely new scenes without in-domain data are not established; the latter is explicitly left open in the discussion. e-taskse-discussion |
Core contributions
- Source description
The dataset contains 76k successful demonstrations, approximately 350 interaction hours, 564 scenes, 52 buildings and 86 verb-defined tasks. Collection involved 50 people and 18 robots across 13 institutions over 12 months. Roughly 16k unsuccessful trajectories are additionally released but excluded from the headline count and policy co-training. e-resourcee-protocole-policy
- Author claim
Episodes provide three synchronized RGB views, depth, calibration, robot state/actions and language. The authors announce a CC-BY 4.0 dataset release, training code, pretrained checkpoints and hardware/control documentation. These are source release claims, not verification of current artifact availability. e-resourcee-releasee-protocol
- Source description
The paper combines dataset diagnostics with physical policy comparisons and a size-matched scene-diversity ablation. Its contribution is a data resource and collection interface, using an existing diffusion-policy learning pipeline. e-diversitye-policye-ablation
Figure 2. The portable collection platform standardizes the robot while allowing its surroundings and camera views to change. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the Franka Panda and portable standing desk, then follow the annotation arrows to the two adjustable external cameras, wrist camera, gripper, laptop and Quest headset. These arrows identify physical components; they are not a neural-network information-flow diagram. Section III-A confirms the hardware labels and explains that the Quest controllers command arm pose and gripper motion. Section III-B supplies the operational link: move the platform to a new workspace, choose camera views, calibrate them, and collect prompted tasks. The shared hardware helps make distributed demonstrations compatible while the surrounding scenes can vary substantially. e-platforme-protocole-policye-tasks
What it supports. The platform separates a stable embodiment from variable observation conditions. Two external cameras and a wrist camera support richer recorded observations, while the movable desk enables collection in different workspaces. This is a concrete infrastructure contribution to dataset diversity; the photograph itself does not measure policy generalization.
Where the evidence stops. The robot is moved between collection scenes; the figure does not demonstrate autonomous navigation. All reported policy evaluations use this hardware stack. The evaluated policy consumes the two external RGB streams and proprioception, so recording a wrist view and depth does not prove those modalities caused the reported gains.
2. Motivation
2.1 The problem and the proposed response
Robot policies must tolerate changes in lighting, objects and workspace layout, but observation–action demonstrations are costly to collect outside controlled laboratories. DROID addresses this collection bottleneck through a shared portable platform and distributed human teleoperation. The research question is whether broader experience improves downstream policy reliability. e-probleme-platform
2.2 What this reading follows
A robot can see many demonstrations yet encounter very few genuinely different workspaces. DROID changes the collection process: participating groups move a common robot setup between scenes, vary camera views and objects, and record language-labeled demonstrations. Read the hardware figure as the mechanism enabling that diversity, the dataset comparison as a coverage inventory, and the policy plots as tests of its practical value. The reported policies still receive local task demonstrations. Their average improvement is substantial, but the novel-chip result and the scene-sampling ablation reveal why a dataset-level success story needs careful qualification. e-probleme-resourcee-platforme-protocole-policye-maine-ablatione-discussion
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Datasets |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
3.1 Evidence-based assessment
Supports the recorded classification
The dataset, demonstration and collection-interface classification is supported. Architecture, prediction paradigm and quadrant are appropriately not applicable to this resource: its evaluation policy predicts actions without a world-prediction component. The combined “Cross-robot & multitask data” tag is supported in its multitask aspect; DROID itself standardizes one robot platform and does not demonstrate cross-embodiment transfer. e-resourcee-platforme-policye-tasks
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
5. Method in detail
5.1 Turn a collection rule into a learning opportunity
DROID’s mechanism begins before any policy is trained. A stationary collection setup can produce many trajectories while repeatedly exposing a model to the same workspace. The DROID protocol instead asks collectors to relocate the platform, define feasible tasks, calibrate external cameras and follow randomly sampled task prompts. Periodic scene augmentations introduce additional variation. The resulting data retains synchronized observations and actions, with success labels and independently written language commands. Reader interpretation: this process can make accidental correlations with a particular background or camera pose less reliable, encouraging policies to use task-relevant information. The paper measures several coverage proxies to support that interpretation, but the collection rules do not by themselves prove which variation changes learned representations or downstream behavior. e-probleme-platforme-protocole-definitionse-diversity
5.2 Trace the evaluated policy from images to executed actions
The evaluation uses a language-conditioned diffusion policy implemented through Robomimic. Two external RGB images are resized and encoded by an ImageNet-pretrained ResNet-50. Their features are concatenated with a frozen DistilBERT instruction embedding and robot proprioception; an MLP passes the conditioning information to a U-Net diffusion head. Training mixes successful DROID demonstrations equally with the local task data. At deployment, the policy generates a 16-step action sequence, executes eight steps without another policy inference, then conditions a new prediction on updated observations. This is an action-generation and execution loop. The supplied paper does not introduce a future-image predictor or a planning search over imagined states. Dataset depth, calibration and wrist images provide additional research possibilities, but they are not specified as inputs to this evaluated policy. e-resourcee-policye-evaluation
5.3 Separate an average benefit from a causal explanation
Figure 8 answers whether adding DROID helps these task policies under the stated protocol. Its average improvements are meaningful, but the chips conditions show why aggregate success cannot be read as a guarantee for every object change. Figure 10 asks a narrower question: does broader scene sampling help when the auxiliary demonstration count is fixed? It compares 7,362 trajectories from the most populated 20 scenes with an equally sized uniform successful-trajectory sample, keeping 50/50 co-training. The diverse sample improves all three displayed OOD tasks. Reader interpretation: this strengthens the argument for broad data collection, while leaving correlated task, object and collector variation unresolved. Neither experiment removes local demonstrations, so the discussion’s goal of useful behavior in new scenes without in-domain data remains open. e-maine-evaluatione-ablatione-discussion
5.4 Training and inference
During training
A separate policy is trained per downstream task, mixing in-domain demonstrations and successful DROID demonstrations 50/50. Task demonstration counts are 70 for waffle closing, 50 for chips, 60 for apple-in-pot, 150 for toasting, 50 for desk cleanup and 50 for lentil cooking. e-taskse-policye-evaluation
The comparisons use in-domain data alone or 50/50 co-training with curated Open X-Embodiment (OXE) data. The OXE mix follows Octo curation but removes Language Table, reported as 5% of that mix. Thus the baseline is neither all OXE nor a controlled single-embodiment match to DROID. e-evaluatione-oxe
During inference
Generate 16 action steps, execute eight open loop, then run policy inference again with updated observations. These are robot actions evaluated through physical rollouts. The supplied formulation does not generate future images, run a world-model planner or derive actions through inverse dynamics. e-policy
5.5 Implementation flow
- Standardize hardware while moving between workspaces
A Franka Panda 7-DoF arm and Robotiq 2F-85 gripper sit on a wheeled height-adjustable desk. Two adjustable Zed 2 stereo cameras and a wrist Zed Mini record observations. Quest 2 controllers provide teleoperation; Polymetis supports control, with actions recorded in joint and end-effector spaces at 15 Hz. e-platform
- Make diversity part of collection
At each new scene, collectors position and calibrate external cameras using a checkerboard and OpenCV, then list feasible tasks. The GUI randomly prompts tasks and periodic changes to lighting, objects, cameras or base position. Collectors mark success; crowdsourcing supplies up to three independent language instructions per episode. e-protocol
- Measure several kinds of coverage
Scenes require substantial workspace changes; rearranging objects alone does not qualify. Tasks are deduplicated instruction verbs. Semantic parsing extracts verbs/objects, GPT-4 deduplicates verbs, and GPT-4V assigns scene types. Camera poses and first gripper-closing positions diagnose viewpoint and interaction coverage. The latter is an interaction proxy, not exhaustive contact measurement. e-definitionse-diversity
- Encode observations for action diffusion
The Robomimic-based policy resizes external RGB images to 128 × 128, encodes them with an ImageNet-pretrained ResNet-50, and concatenates visual features, frozen DistilBERT language embeddings and proprioception. An MLP conditions a U-Net diffusion head that generates action trajectories. The wrist view and depth are available in the dataset but are not listed among these policy inputs. e-policye-resource
6. Experiments & results
DROID expands robot demonstration coverage by moving a standardized manipulation platform through many real workspaces. Its 76k successful episodes support diffusion-policy co-training that improves average executed-task success, while retaining task-specific failures. The central evidence concerns data diversity and supervised robot control, rather than learned world prediction (e-resource, e-policy, e-main).
This is a dataset paper using an existing policy algorithm. Its supplied method visual is the collection platform, not a neural architecture diagram; Table I quantifies dataset properties, while policy outcomes and the ablation are reported as bar charts in Figures 8 and 10 rather than numerical result tables. The referenced appendices and Figures 11–15 are not in this proceedings PDF, limiting access to detailed configurations and expanded diagnostics. e-platforme-definitionse-policye-maine-ablatione-scope
6.1 Read the original evidence
Table I. DROID’s distinguishing scale is its scene coverage, alongside language and camera calibration. Original paper, p. 3 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the trajectory, verb and scene columns together. DROID lists 76k trajectories, 86 verbs and 564 scenes; BridgeData V2 lists 60.1k, 82 and 24. The OXE aggregation is larger in trajectories and verbs but lists 311 scenes. The retained table notes distinguish DobbE’s tool-based collection from robot data and mark OXE as an aggregation, so its row is not independent of all earlier rows. Footnote 2 elsewhere on this page explains that RH20T’s 13k counts unique multi-view trajectories rather than counting each camera stream separately. The scene definition also excludes mere object rearrangements. e-definitionse-resource
What it supports. DROID’s scene count substantially exceeds that of BridgeData V2 despite comparable verb counts, making scene coverage a plausible contributor to different transfer behavior. The language and calibration columns describe useful dataset features. These are descriptive resource comparisons, not evidence that every DROID-trained policy must outperform policies trained on other data.
Where the evidence stops. Counts use the paper’s definitions and its contemporary dataset snapshot. OXE is an aggregation, and DobbE uses human-held tools. The table does not control robot morphology, data quality or training compute, and its scene count alone cannot establish a causal performance benefit.
Figure 4. Scene coverage expands beyond a few repeated laboratory workspaces, while remaining uneven. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use the vertical axis as a count of distinct scenes, not demonstrations or success rates. The horizontal categories divide workspaces into named scene types and an Unknown bin. Compare the DROID bars with the smaller Bridge V2, RT-1 and RH20T bars within each category. Section IV explains that GPT-4V assigns DROID scene types, whereas the small prior datasets are categorized manually; the caption says prior scene counts/types were verified manually or with their authors. This chart complements Table I by showing where the additional scenes occur, rather than treating every increment in the total count as interchangeable. e-definitionse-diversitye-scopee-ablation
What it supports. DROID covers many more scene categories than the plotted comparison datasets, including several household room types. The tall industrial-office and home-kitchen bars also show that broader coverage is not uniform coverage. That distribution helps motivate testing scene diversity directly instead of inferring robustness from the total number of demonstrations.
Where the evidence stops. Scene-type labels are descriptive annotations, not held-out evaluation splits. GPT-4V’s referenced prompt is in an appendix absent from this PDF. The figure gives no evidence that every category has enough demonstrations for reliable transfer, or that scene variety is independent of task and object variety.
Figure 8. Average success favors DROID, with an OOD exception on novel chip bags. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the shared legend first: light green is local demonstrations alone, teal is OXE co-training, and orange is DROID co-training. The top panel reports in-distribution success across six tasks; the bottom has seven OOD settings because chips has separate distractor and novel-chip tests. Each setting uses 10 rollouts per method. The shaded averages show 0.75 versus 0.53 and 0.43 in distribution, and 0.53 versus 0.36 and 0.17 out of distribution. The caption identifies the average whiskers as standard errors. Compare individual bars before accepting its broad wording: novel chips favors OXE, and chip distractors tie. e-taskse-policye-evaluatione-main
What it supports. The caption’s 22-point in-distribution improvement is over no co-training, the strongest average baseline in that panel. Its 17-point OOD improvement is over OXE. These gains concern real robot task execution after training with local demonstrations. They support a useful average co-training benefit, not uniform superiority across every tested variation.
Where the evidence stops. The caption says DROID improves across all tasks, but novel-chip success is 0.30 versus OXE’s 0.60, while chip distractors tie at 0.40. Preserve these exceptions. Ten trials per setting and unreported training-seed variability limit the precision of comparisons; the chart is not a zero-shot benchmark.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| In-distribution manipulation across six tasks DROID hardware at four locations; separate task policies with 50–150 local demonstrations; 10 rollouts per task setting and method, with initial robot/object position noise. | DROID 0.75 Mean success rate, as labeled in Figure 8 | No co-training 0.53; OXE 0.43. The caption reports a 22-percentage-point gain over the strongest average baseline, which here is no co-training. Average bars have standard-error whiskers; numerical errors are not tabulated. e-taskse-evaluatione-main |
| Robustness to out-of-distribution changes Seven plotted settings over six tasks: distractors, novel chips/objects, and a camera shift for lentil cooking; 10 rollouts per setting and method. | DROID 0.53 Reported average executed-task success rate | OXE 0.36; no co-training 0.17. The reported 17-percentage-point gain is over OXE. The displayed average is consistent with averaging seven settings; chips contributes two variants. It is not a zero-shot evaluation without local demonstrations. e-taskse-evaluatione-main |
| Place Chips on Plate with novel chips OOD chip-bag instance change; the same co-training comparison and 10-rollout protocol. | DROID 0.30 Success rate | OXE 0.60; no co-training 0.20. The separate distractor condition ties DROID and OXE at 0.40. These bars qualify the caption’s broad improvement claim: DROID does not beat OXE on every OOD condition. e-taskse-evaluatione-main |
| Scene diversity at fixed demonstration count Two 7,362-trajectory DROID subsets: the 20 most demonstration-rich scenes versus a uniform successful-trajectory sample; 50/50 co-training on three OOD tasks. | Diverse scenes 0.60 Success rate averaged over the three displayed tasks | 20 scenes 0.40. Diverse/20-scene rates: toasting 0.60/0.30, desk cleanup 0.80/0.70, lentil cooking 0.40/0.20. The 20-point average advantage supports broad sampling at fixed dataset size. Object, task and collector composition are not separately matched, so this does not isolate scene count alone. e-ablation |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 10. At equal trajectory count, broader scene sampling improves the three-task OOD average. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read this figure together with Section V-C on the next page. Teal is the subset formed from the 20 scenes containing the most demonstrations; orange is a uniform sample of successful demonstrations from across DROID. Both contain 7,362 trajectories and use the same 50/50 mixture with task data. The plotted gains are 0.30 to 0.60 for novel-object toasting, 0.70 to 0.80 for desk distractors, and 0.20 to 0.40 for lentils with distractors and a camera shift. Unlike Figure 8, this average covers only those three OOD tasks, so the two averages have different denominators. e-ablation
What it supports. The diverse subset reaches 0.60 average success versus 0.40 for the 20-scene subset. Holding trajectory count fixed makes this more informative than comparing a large diverse dataset with a small narrow one. The supported conclusion is that this broad-sampling strategy helps under the tested co-training protocol.
Where the evidence stops. The subsets are not reported as matched for verbs, objects, collectors or camera distributions. Thus the study controls data quantity but does not isolate scene count from correlated diversity. Error bars appear on the average bars, but their numerical values and a separate uncertainty definition are not supplied here.
7. Analysis & limitations
7.1 What the evidence leaves open
All evaluations use the shared DROID embodiment and local demonstrations. Generalization across robot morphologies and deployment in entirely new scenes without in-domain data are not established; the latter is explicitly left open in the discussion. e-taskse-discussion
Ten rollouts per setting give coarse success estimates. Figure 8 supplies average standard-error bars but no numerical uncertainty table or training-seed analysis. Scene counts, deduplicated verbs and first gripper-closing locations quantify selected aspects of diversity rather than guaranteeing behavioral coverage. e-evaluatione-maine-definitionse-diversity
Collector-selected workspaces/tasks and a long-tailed verb distribution can leave uneven coverage. The ablation changes the sampled scene population together with associated objects and behaviors; its size control narrows, but does not eliminate, alternative explanations. e-protocole-diversitye-ablation
7.2 Questions for discussion
- Would diverse scenes still help after matching verb, object and collector distributions? (e-ablation, e-diversity)
- What causes DROID to trail OXE on novel chip bags while improving average robustness? (e-main)
- How much local demonstration data can be removed before the reported benefit disappears? (e-policy, e-discussion)
8. Reproducibility audit
8.1 Requirements and known gaps
A reproduction needs the calibrated DROID platform, successful-episode filter, task demonstrations, matching language/vision preprocessing, action representation and 50/50 sampling. The OXE comparison additionally needs the stated curated mixture and Language Table exclusion. e-platforme-protocole-policye-oxe
The supplied body gives no formal diffusion loss, optimizer schedule, batch size, training duration, compute configuration or seed count. It refers training details to absent Appendix F and evaluation details to absent Appendix E. Exact sample manifests, overlap checks and success-scoring details would need resolution before claiming a faithful reproduction. e-policye-taskse-scopee-ablation
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Match behavior composition while varying scene coverage
Reader-proposed experiment, not performed: reproduce the two 7,362-trajectory sampling strategies, then add a comparison in which low- and high-scene subsets are matched as closely as feasible for verb, object and collector distributions. Report unmatched strata rather than claiming perfect control. Hold the local demonstrations, 50/50 batch mixture, policy architecture and training budget fixed; first resolve the absent training settings. Use at least three training seeds and 30 paired initializations per method on each of the three Figure 10 OOD tasks, recording binary success and seed variability. A persistent advantage for broader scenes after matching would strengthen the scene-diversity interpretation. An advantage that disappears would implicate correlated behavior composition in the original gain. e-ablatione-diversitye-policye-taskse-scope
Check 2: Re-test the novel-chip exception under paired conditions
Reader-proposed experiment, not performed: train the three Figure 8 chip policies using the same 50 local demonstrations, with the stated DROID and curated OXE mixtures. Fix preprocessing, architecture and training budget after resolving missing settings. Evaluate 30 paired trials per seed for both the novel-chip and added-distractor conditions, using identical initial poses and object arrangements across methods. Prespecify held-out chip instances and record grasp, transport and placement failures separately. Repeat with at least three training seeds. If OXE’s novel-chip advantage persists while the distractor condition remains tied, it supports a condition-specific transfer difference. If the ranking reverses across seeds or paired trials, the original 10-rollout result is too unstable to support that explanation. e-taskse-policye-evaluatione-oxee-maine-scope
8.3 Reading coverage
Visual audit: All 13 pages of the supplied proceedings PDF were rendered and visually inspected, including title/credits (p. 1), affiliations and definitions (p. 2), Table I and its separate RH20T footnote (p. 3), platform/protocol (p. 4), diversity analysis (pp. 5–6), task and policy details (pp. 7–8), OXE curation and ablation (pp. 9–10), discussion (p. 10), and the references-only ending (pp. 11–13). Figures 1–10 and Table I were inspected. Each of the five final crops was separately viewed with readable labels and legends. The table crop retains its caption because it contains the scene definition and the DobbE/OXE qualifications; the distant RH20T footnote is explained in its reading guide. Figure 2 arrows were checked against its caption and Section III-A as component labels. Figure 8’s per-condition bars qualify its caption’s broad improvement claim; Figure 10 was checked against the subset construction on p. 10. Referenced appendices and Figures 11–15 are absent from this PDF; no supplemental videos or external artifacts were inspected.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13. Appendix coverage: not present.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Title, author block and abstract (PDF pp. 1–2)
- I. Introduction (p. 2)
- II. Related Work, including dataset definitions and Table I (pp. 2–3)
- III. DROID Data Collection Setup: platform and protocol (pp. 3–4)
- IV. DROID Dataset Analysis (pp. 4–7)
- V. Experiments: setup, training, robustness and scene diversity (pp. 7–10)
- VI. Discussion and acknowledgment (p. 10)
- References [1]–[53] (pp. 10–13)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- The complete supplied 13-page RSS 2024 proceedings PDF was read. Its title matches the catalog; the title page identifies Robotics: Science and Systems 2024, Delft, July 15–19, 2024. No revision number or revision history is supplied; no other edition was inspected.
- The observed author list substantially agrees with the catalog but includes Kevin Lin and omits catalog-listed Jean Mercat, Emi Tran and Michael C. Yip. The PDF uses Jason Ma, Kyle Hatch, Homer Walke, Jonathan Yang, Abby O’Neill and Andrew Wang where the catalog gives longer or different name forms; Arhan Jain appears after Youngwoon Lee. Metadata follows the PDF spelling and order, including Pannag R Sanketi and Roberto Martín-Martín. These differences do not establish a revision chronology.
- The PDF references Appendix/Section B (recorded features), Appendix C (scene-labeling prompt), Appendix E (task details), Appendix F (training details), and Figures 11–15; none is included in this supplied artifact. Appendix status means absent from this PDF, not absent from every edition.
- Text extraction does not reconstruct figure images; all 13 retained PDF pages and Figures 1–10 plus Table I were therefore visually inspected separately. Five final original crops were inspected.
- Separate supplemental material availability has not been fully verified. Supplementary videos, dataset files, project website, code and checkpoints were not inspected, and experiments were not reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e-titlePDF p. 1, proceedings header, title and full author block
The observed title is DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset. The header identifies RSS 2024 in Delft, July 15–19, 2024. Credits begin Alexander Khazatsky, Karl Pertsch and Suraj Nair and end Thomas Kollar, Sergey Levine and Chelsea Finn; Kevin Lin is included.
Go to primary source ↓e-affiliationsPDF p. 2, Affiliations footnote, entries 1–16
The numbered author affiliations run from Stanford University to Yonsei University; these affiliations should not be equated with the count of data-collecting institutions.
Go to primary source ↓e-problemPDF p. 2, Section I, paragraphs 1–3
The introduction links generalizable manipulation to diverse observation–action data and describes the logistical, safety, hardware and labor barriers to collecting it outside laboratories.
Go to primary source ↓e-resourcePDF p. 1, Figure 1 and caption; p. 3, Section III opening
The overview reports 76k trajectories, 350 hours, 564 scenes, 52 buildings and 86 tasks/verbs; Section III reports 18 robots, 50 collectors, 13 institutions and 12 months. Episodes include RGB views, depth, calibration and language.
Go to primary source ↓e-releasePDF p. 2, Section I final paragraph; p. 3, Section III release paragraph
The authors announce the full dataset under CC-BY 4.0, a visualizer, policy-training code, pretrained policy checkpoints and hardware/control documentation.
Go to primary source ↓e-definitionsPDF p. 2, footnote 1; p. 3, Table I, caption and footnote 2
Tasks are counted using unique instruction verbs; scenes require substantial workspace changes. Table I gives DROID 76k/86/564, BridgeData V2 60.1k/82/24 and OXE 1.4M/217/311 for trajectories/verbs/scenes. DobbE is tool-based, OXE aggregates datasets, and RH20T counts are normalized to unique multi-view trajectories.
Go to primary source ↓e-platformPDF p. 4, Figure 2, caption and Section III-A
The portable setup has a Franka Panda 7-DoF arm, Robotiq 2F-85, two Zed 2 cameras, a wrist Zed Mini and Quest 2 teleoperation. The text specifies Polymetis, 15-Hz action recording, a control box, NUC and laptop.
Go to primary source ↓e-protocolPDF p. 4, Section III-B
Collectors calibrate cameras with a checkerboard and OpenCV, enter a task list, follow random task/augmentation prompts and label success. About 16k unsuccessful trajectories are released outside the 76k successful count. Crowdsourcing provides up to three independently written instructions per episode.
Go to primary source ↓e-diversityPDF p. 5, Figure 3 and Section IV; p. 6, Figures 4–6 and Section IV(a–e)
The analysis uses sampled third-person frames and language. Semantic parsing and GPT-4 deduplication produce verb/object diagnostics; GPT-4V labels scene types. Figure 4 compares scene-category counts, Figure 5 reports 1,417 viewpoints, and Figure 6 visualizes first gripper-closing positions as an interaction proxy.
Go to primary source ↓e-tasksPDF p. 7, Figure 7 and Section V-A(a)
Six tasks span four locations on DROID hardware. Demonstration counts are waffle 70, chips 50, apple 60, toasting 150, desk 50 and lentils 50. OOD variants include distractors, novel objects/chips and a camera shift for lentils; additional task details are referred to Appendix E.
Go to primary source ↓e-policyPDF pp. 7–8, Section V-A(b), Policy training
The Robomimic diffusion policy takes language, two external RGB streams and proprioception, and outputs absolute end-effector/gripper actions. Images are 128 × 128; ResNet-50 features join frozen DistilBERT and state features, followed by an MLP and U-Net diffusion head. The head generates 16 actions, eight execute before replanning, and co-training mixes local/successful DROID data 50/50. Further details are referred to Appendix F.
Go to primary source ↓e-evaluationPDF p. 8, Section V-B, opening and three comparison bullets
Separate task policies undergo A/B evaluations with 10 rollouts for every task setting and method. Baselines use local data alone or 50/50 mixtures with DROID or OXE; in-distribution testing adds initial-position noise, while OOD testing changes the scene or object.
Go to primary source ↓e-oxePDF pp. 8–9, Section V-B, OXE bullet continuation; p. 9, footnote 3
OXE spans 22 embodiments and approximately 300 scenes in the discussion. The evaluated mix follows Octo curation and removes Language Table, stated as 5% of the Octo mix, because of repetitiveness and infrastructure burden.
Go to primary source ↓e-mainPDF p. 8, Figure 8, both panels, Average bars, chip conditions and caption
In-distribution average labels are no co-training 0.53, OXE 0.43, DROID 0.75. OOD labels are 0.17, 0.36 and 0.53 over seven displayed conditions. Novel chips gives 0.20/0.60/0.30, and chip distractors gives 0.10/0.40/0.40 in the same method order. The caption describes average standard errors and gains of 22 and 17 absolute percentage points, despite its overbroad per-task improvement wording.
Go to primary source ↓e-ablationPDF p. 9, Figure 10 and caption; p. 10, Section V-C
The 20 most demonstration-rich scenes supply 7,362 trajectories; a uniform sample of 7,362 successful episodes supplies the diverse comparison. Both use 50/50 co-training. Figure 10 gives 20-scene/diverse success of 0.30/0.60 for toasting, 0.70/0.80 for desk cleanup and 0.20/0.40 for lentils, with averages 0.40/0.60.
Go to primary source ↓e-discussionPDF p. 10, Section VI
The discussion identifies combining datasets, learning visual representations, selecting data subsets and operating in new scenes without in-domain data as open questions.
Go to primary source ↓e-scopePDF p. 4, Section III-B; p. 5, Figure 3 caption; p. 6, Section IV; pp. 7–8, Section V-A; pp. 10–13, ending
The body points to Section/Appendix B, Appendices C, E and F and Figures 11–15. This supplied proceedings PDF ends with references [1]–[53] and contains none of those appendix sections or figures.
Go to primary source ↓8.5 Primary sources
DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset ↗
PDF · 9,340 extracted words
Source fingerprint
f37321af021aba1d19351a166241db48230b82e85c3fccf2054261fd46ca339a