RoboNet: Large-Scale Multi-Robot Learning
1. Paper overview
In one sentence: Shared robot experience can reduce target-robot data needs, but the tested models sometimes benefit more from a relevant subset than from the full diversity of RoboNet. datasetforwardinverse-methodfrankabaxtergripper
| At a glance | What to know |
|---|---|
| Research problem | Source description Robot-specific data collection repeatedly relearns similar interactions. RoboNet asks whether shared image/action experience can support object relocation across viewpoints, backgrounds, grippers and robots without supplying the learned predictor with object geometry or robot kinematics. datasetforward |
| Core mechanism | Source description The initial release reports roughly 162,000 trajectories and nearly 15 million frames across seven platforms and four institutions. Table 1 documents 113 camera configurations; RGB exposure, objects, tables and grippers vary. datasetcollection |
| A key reported result | Franka few-shot object relocation: 40%. Human-judged success rate.. Exclude Franka/R3/Fetch during pretraining; fine-tune on 400 Franka trajectories. Scratch: 20% with 400 trajectories; 35% with 8,000. The 20-fold reduction concerns target-robot data, not total training data. Table 4 gives no uncertainty or evaluation denominator. frankaevaluation |
| Reading caution | Author claim Tasks are simple, relatively low-fidelity pushing and pick-and-place. Random policies restrict nuanced interactions; imperfect predictions limit control. Larger models improving physical performance remains a hypothesis. limitationscapacity |
Core contributions
- Source description
The initial release reports roughly 162,000 trajectories and nearly 15 million frames across seven platforms and four institutions. Table 1 documents 113 camera configurations; RGB exposure, objects, tables and grippers vary. datasetcollection
- Source description
The paper evaluates forward video prediction with planning and a separate supervised inverse model. These demonstrate dataset utility; they do not form a unified future/action generator. forwardinverse-method
- Source description
Attribute-filterable trajectories, HDF5 storage, MP4 videos and manual upload quality checks support sharing. The paper advertises project code; a dataset license is not stated in the supplied PDF. database
Figure 1. Shared experience spans robot hardware, camera placement and scenes. Original paper, p. 2 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each short image sequence from left to right along its time arrow. The named groups identify seven robot platforms; multiple rows within a group show different scenes or views rather than stages in one universal task. Look at the differences in gripper appearance, table texture and camera placement. Section 4 explains why the action interface matters alongside this visible diversity: root-frame end-effector deltas can be shared even though an identical command need not produce identical image motion. The figure introduces the learning problem; the quantitative composition is recorded separately in Table 1. datasetcollectiondata-counts
What it supports. The visual supports RoboNet's premise that robot experience can vary along several dimensions at once. The paper reports roughly 162,000 trajectories and nearly 15 million frames across seven platforms. A learner must use visual context to interpret actions across these differing appearances and viewpoints.
Where the evidence stops. This montage is illustrative, not a balanced sampling chart. It does not establish equal coverage of robots or interactions. Table 1's six-arm caption conflicts with its seven robot entries; the report preserves that inconsistency.
2. Motivation
2.1 The problem and the proposed response
Robot-specific data collection repeatedly relearns similar interactions. RoboNet asks whether shared image/action experience can support object relocation across viewpoints, backgrounds, grippers and robots without supplying the learned predictor with object geometry or robot kinematics. datasetforward
2.2 What this reading follows
RoboNet asks what a robot can inherit from other robots' experience. Its database combines videos and actions across different arms, grippers, viewpoints and scenes. The paper then tests two ways to use that experience: predicting future images to plan object relocation, and predicting actions directly for visual reaching. Read the results as a study of transfer under specific physical tasks. Pretraining helps new-robot adaptation, yet Sawyer-only data beats broader pretraining on Baxter and on a changed Sawyer gripper. That tension makes the paper useful for thinking about dataset composition, model capacity and the gap between predicting an image and controlling a robot. datasetforwardinverse-methodfrankabaxtergripper
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Not assigned |
| Architecture | Not assigned |
| Prediction paradigm | Not assigned |
| Quadrant | Not assigned |
This table preserves the labels recorded at reading time. The current major category is Datasets. View the current classification.
3.1 Evidence-based assessment
Classification assessment not applicable
The catalog is unassigned, so there is no substantive quadrant claim to confirm. Reader analysis: this dataset supports action-conditioned forward modeling with separate planning and a separate inverse-dynamics baseline. Figure 6 does not jointly generate futures and actions; sharing one predictor across robots does not establish a One Model architecture. datasetarchitectureplanninginverse-method
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Start with the action interface, then ask what context must recover
RoboNet makes cross-robot learning possible by sharing a high-level action description: changes in end-effector position and yaw plus a gripper command. This reduces dependence on a particular robot's joint configuration, but does not remove embodiment. A root-frame motion appears different when the camera moves, and a different gripper changes contact with the object. The predictor therefore receives visual context as well as candidate actions. Appendix A specifies two context frames; the separate inverse model uses two frame/action pairs. Reader interpretation: context serves as evidence about the current embodiment and viewing geometry, rather than an explicit calibrated camera model. This explanation has a boundary: Google R3 and Fetch use different exploration policies, and the common blocking interface imposes stops between actions. Shared data still carries collection-specific constraints. collectionarchitectureinverse-method
Figure 6. Visual foresight learns an action-conditioned image transformation model. Original paper, p. 13 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow the upper path from the 48×64 RGB input through downsampling, Conv-LSTM blocks and upsampling. The yellow action branch is tiled before entering the recurrent network; the top skip paths preserve earlier features. On the right, distinguish the flow field from the compositing masks. Flow transforms the input-image branch, while the lower branch supplies the initial image I_0 for compositing. The output is a future image, not an action command. Appendix A's equations describe recurrent state and image warping; MPPI, described below this diagram, separately searches for actions using the model. architectureplanningtraining
What it supports. The architecture makes actions conditions on predicted consequences. Combined with pixel propagation, those predictions support goal-directed planning at inference time. Figure 6 also reveals image-compositing detail that the compact warping equation omits. Its caption credits the diagram to prior visual-foresight work, so it should not be read as a wholly new RoboNet architecture.
Where the evidence stops. Figure 5 depicts one context frame, but Appendix A.1 says the experiments use two. Figure 6 labels the output I_{t+1}, whereas Equation (2) marks it as predicted with a hat. Neither drawing specifies every training-loss or implementation detail.
5.2 Trace the complete route from a selected pixel to an executed action
Visual foresight begins with a user-selected object pixel and a desired image location. The model predicts how candidate actions transform images; the same flow propagates a distribution over that pixel's future location. Expected distance to the goal turns a video rollout into a planning score. MPPI samples and refines action sequences, using 600 samples, a 15-step horizon and three iterations in this paper. Only the first action is executed before observing and replanning. Crucially, replanning still needs the object's current pixel position. RoboNet uses predictor propagation, which the authors acknowledge is noisy. Thus prediction error can affect both anticipated consequences and subsequent state estimation. The one-step inverse baseline takes a different route: it regresses an action directly from visual context and a goal image, without this forward search. forwardarchitectureplanninginverse-method
5.3 Separate transfer, data selection and capacity evidence
The strongest data-efficiency example is Franka: pretraining followed by 400 target trajectories yields 40% success, compared with 35% for scratch training on 8,000. This establishes a target-data advantage in that task set, not a reduction in all data used. Baxter and the new-gripper experiment ask a different question and show that relevant Sawyer-only pretraining can outperform broader RoboNet pretraining. The authors connect this to underfitting. Their capacity diagnostic finds lower held-out Sawyer image error for a 500M-parameter model than for a 200M model, while the control experiments use 5M–75M models. Reader analysis: the pieces motivate the hypothesis, but do not complete its causal test. A larger predictor has not here been shown to remove the subset advantage or improve executed control under the same evaluation protocol. frankabaxtergrippercapacityevaluation
5.4 Training and inference
During training
Future images supervise video prediction; the loss formula is not fully specified. Appendix A gives 160k pretraining iterations, batch size 16, Adam and 150k additional fine-tuning steps. Learning rate starts at 1e-3, but the 200k-step decay wording needs clarification. architecturetraining
Adaptation withholds the target robot during pretraining, then uses 300–400 target trajectories. R3 and Fetch are also excluded for compute reasons. The inverse baseline learns one-step action regression on Sawyer and Franka data. kukafrankabaxterinverse-method
During inference
MPPI samples 600 sequences over 15 steps, uses three iterations with exponential cost weighting and low-pass smoothing, then executes the first action. A new image closes the loop; predictor propagation estimates the designated pixel and can be noisy. planningforward
Inverse control directly predicts an action from context, current image and goal image, repeating each step. This greedy controller needs no forward rollout and is evaluated on reaching. inverse-methodinverse-results
5.5 Implementation flow
- Make actions shareable
Most robots use end-effector XYZ/yaw deltas and binary gripper commands rather than joint-space actions. Models infer camera-to-robot relationships from context. Per-robot safety bounds remain necessary; blocking execution stops the robot at each step. collection
- Collect interaction without task labels
Gaussian exploration uses robot-specific variances and a height-triggered grasping primitive. Google R3 instead samples pushes; Fetch explores x and y only. These exceptions constrain observed interactions. collection
- Predict consequences
A deterministic SAVP variant combines convolutional LSTMs, convolutions and skip connections. Actions enter a tiled branch; flow warps images. Figure 6 also shows masks compositing a transformed image with an initial-image branch. architecture
- Turn predictions into control
Propagate selected pixels through predicted flow, score candidate sequences by expected goal distance, then execute and replan. Prediction supplies the planner with a model of consequences. forwardplanning
6. Experiments & results
RoboNet pools robot experience to make visual control transferable. Pretraining improves adaptation with a few hundred target-robot trajectories, but relevant subsets can outperform the broader pool. Its central contribution is a shared dataset evaluated through two distinct control algorithms.
6.1 Read the original evidence
Table 2. Training across viewpoints reduces measured relocation error in both tested views. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read down a column to compare training conditions while holding the evaluation view fixed. The left column is the seen view; the right is a held-out view. Lower distance is better. Section 5.2 specifies one training viewpoint versus 90 and five relocation tasks from two views. The original caption identifies the ± values as standard errors. Section 5.1 and Appendix D explain that these are distances after executing robot actions, with fixed start and goal positions and tape measurement, not image-prediction reconstruction errors. Read across a row only to inspect the change in viewpoint. viewpointevaluation
What it supports. The multi-view model reports 9 ± 2.2 cm in the seen view and 16.2 ± 2.9 cm in the held-out view. The single-view model reports 14.8 ± 3.8 and 23.2 ± 2.6 cm, respectively. Both comparisons favor diverse-view training within this benchmark.
Where the evidence stops. This tests a new view with a previously seen robot, not a new robot. The small task set and standard errors do not establish statistical significance. The paper explicitly warns against comparing distances across separate experiments with different task difficulty.
Tables 3–5. Few-shot adaptation helps, while Baxter exposes the value of selecting pretraining data. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Treat the three tables as separate experiments. Within Kuka or Franka, compare scratch training with 400 target trajectories against pretraining followed by the same 400-trajectory fine-tuning set, then inspect the larger scratch-data baseline. The exclusions under the pretraining rows prevent target-robot leakage; Section 5.3 also excludes R3 and Fetch for computational reasons. In Baxter, all rows use 300 target trajectories, and the key contrast is Sawyer-only versus broader pretraining. Success is judged by an operator, so these percentages measure executed task outcomes. They are not interchangeable with the distance metric in Table 2. kukafrankabaxterevaluation
What it supports. Kuka pretraining plus 400 trajectories reaches 40%, versus 30% from scratch with 1,800. Franka reaches 40%, versus 35% from scratch with 8,000. Baxter reverses the simple diversity story: Sawyer-only pretraining reaches 83%, compared with 58% for broader RoboNet pretraining and 33% from scratch.
Where the evidence stops. The tables provide neither evaluation denominators nor uncertainty for these success rates. Compare methods within each robot's task set. The target-data advantage excludes pretraining data, and the subset comparison changes data composition as well as quantity.
Figure 4. A physical Franka rollout connects image-based goals to executed cup relocation. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the legend: red marks the starting position, green the goal and blue the final position. The arrow in the goal image points from the selected cup toward its intended location. Then compare the predicted and actual rows at the same labeled times rather than treating every image as a robot observation. The prediction becomes visibly blurred while the actual sequence shows the gripper interacting with the cup. The caption identifies this as a Franka example after RoboNet pretraining without Franka and fine-tuning on 400 Franka trajectories; Table 4 provides the aggregate result. frankaforwardplanning
What it supports. The authors present this as a successful grasp-and-move task involving a thin plastic cup. Its role is to show what object relocation and a visual goal look like in physical execution. The side-by-side rows also help distinguish the model's forecast from the robot's observed behavior.
Where the evidence stops. This is a selected successful sequence, not an estimate of reliability or continuous video evidence. The stills cannot establish every contact event or the replanning history. Aggregate success remains the separate Table 4 measurement.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Zero-shot viewpoint generalization Five relocation tasks from seen/held-out views; one-view versus 90-view training. | Multi-view: 9 ± 2.2 seen; 16.2 ± 2.9 held-out. Final distance, cm; mean ± standard error; lower is better. | Single-view: 14.8 ± 3.8 seen; 23.2 ± 2.6 held-out. Diverse views improve both tested views; this is viewpoint transfer, not unseen-robot adaptation. viewpointevaluation |
| Zero-shot background generalization Same multi-view model; similar tasks in a substantially different environment. | 14.4 ± 2 cm. Final distance, cm; mean ± standard error. | No matched single-view baseline for this background setting. Figure 3 illustrates one successful push; the mean is a separate task-set measurement. background |
| Kuka few-shot object relocation Exclude Kuka/R3/Fetch during pretraining; fine-tune on 400 Kuka trajectories. | 40%. Human-judged success rate. | Scratch: 10% with 400 trajectories; 30% with 1,800. Target data needs decrease; Table 3 gives no uncertainty or evaluation denominator. kukaevaluation |
| Franka few-shot object relocation Exclude Franka/R3/Fetch during pretraining; fine-tune on 400 Franka trajectories. | 40%. Human-judged success rate. | Scratch: 20% with 400 trajectories; 35% with 8,000. The 20-fold reduction concerns target-robot data, not total training data. Table 4 gives no uncertainty or evaluation denominator. frankaevaluation |
| Baxter pretraining-subset comparison Same 300 Baxter adaptation trajectories; scratch, Sawyer-only and broader pretraining. | Sawyer-only pretraining: 83%. Human-judged success rate. | Broader RoboNet pretraining without Baxter: 58%; scratch: 33%. Relevant pretraining wins here; dataset size and composition are not isolated. baxter |
| Sawyer predictor capacity diagnostic Held-out Sawyer sequences; simplified deterministic large predictors. | 500M parameters: 0.0847 ± 0.045. Average per-pixel l1 error; lower is better. | 200M parameters: 0.104 ± 0.057. The ± statistic is unspecified. This prediction test does not measure larger-model control gains. capacity |
| Inverse-model visual reaching One Sawyer/Franka-trained model; five reaching tasks per setting. | Sawyer front: 4/5; Sawyer unseen view: 5/5; Franka front: 4/5. Successful tasks, human judged. | No alternative-model baseline in Table 6. Small reaching tests support transfer; authors report poor complex object manipulation. inverse-results |
| Unseen-gripper adaptation Sawyer with new Robotiq gripper; 300 adaptation trajectories; ten evaluation trajectories per model. | Sawyer-only pretraining: 9.8 ± 2.1. Average final distance, cm; lower is better. | All-RoboNet: 14.7 ± 2.1; zero-shot: 15.5 ± 2.6; scratch: 17 ± 1.8. The ± statistic is undefined. Broader pretraining underperforms a relevant subset. gripper |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 7. A relevant pretraining subset outperforms broader data after a gripper change. Original paper, p. 15 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the rows as four distinct ways to handle a Sawyer fitted with an unseen Robotiq two-finger gripper. The zero-shot model receives no target adaptation; the scratch and fine-tuned conditions use the 300 newly collected trajectories described in Appendix E. Lower average distance is better, and each condition is evaluated over ten trajectories. Compare Sawyer-only pretraining directly with all-RoboNet pretraining to see the data-composition effect. Figure 8 on the same page shows a qualitative pushing example, but it is the table that supports the numerical comparison between these four conditions. gripperbaxtercapacity
What it supports. Sawyer-only pretraining produces the lowest reported average distance, 9.8 ± 2.1 cm. Broader pretraining gives 14.7 ± 2.1 cm; zero-shot control gives 15.5 ± 2.6 cm and scratch training 17 ± 1.8 cm. Together with Baxter, this result challenges a monotonic benefit from adding diverse pretraining data.
Where the evidence stops. Unlike Table 2, Table 7 does not define the ± statistic. Do not relabel it as a standard error. The authors attribute the subset advantage to limited capacity and underfitting, but this comparison does not isolate that causal mechanism.
7. Analysis & limitations
7.1 What the evidence leaves open
Tasks are simple, relatively low-fidelity pushing and pick-and-place. Random policies restrict nuanced interactions; imperfect predictions limit control. Larger models improving physical performance remains a hypothesis. limitationscapacity
Success depends on human judgment; different robot experiments are not directly comparable. Five-task viewpoint/reaching studies and ten-trajectory gripper tests provide limited coverage, not arbitrary manipulation competence. evaluationviewpointinverse-resultsgripper
Counts are not fully reconciled: Table 1 has seven robot entries but its caption says six; trajectory counts differ from Section 4.2's subtotal. Franka is 7.9k in Table 1 and 8,000 in Table 4; both labels are retained. data-countsfranka
7.2 Questions for discussion
- Would larger predictors remove Sawyer-only pretraining's advantage under matched data and compute budgets?
- Does lower video error predict better executed relocation after controlling for tracking error?
8. Reproducibility audit
8.1 Requirements and known gaps
Recover attribute-based robot/view holdouts, matching action conventions, safety bounds and exploration parameters stored under policy-description. Immutable split files and dataset licensing are not supplied in the PDF. collectiondatabase
Clarify the learning-rate schedule, loss, per-experiment architecture sizes, planner weighting/filter settings and compute budget. Preserve fixed start/goal placements; report trial counts and measured distance alongside judged success. trainingarchitectureplanningcapacityevaluation
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Does capacity remove the pretraining-subset advantage?
Proposed check, not performed: use Baxter as the target, exclude Baxter/R3/Fetch from pretraining, and compare equal-sized Sawyer-only and mixed-robot pretraining subsets. Fine-tune every model on the same 300 Baxter trajectories. Repeat this comparison with a small and a larger predictor, keeping optimization steps and the evaluation objects, start/goal positions and planner settings fixed; report training compute and results across seeds. Measure prediction error, final physical distance and success with trial counts. The capacity explanation predicts that the mixed-data disadvantage should shrink as capacity grows. Persistence despite better training fit would argue for distribution relevance or another mechanism. baxterdata-countscapacitytrainingplanningevaluation
Check 2: Does better video prediction improve control or only its images?
Proposed check, not performed: reproduce the 200M-versus-500M Sawyer prediction comparison on the same held-out sequences, then evaluate both models on identical physical relocation tasks. Keep the source planner's 600 samples, 15-step horizon and three iterations fixed. Measure per-pixel error, final goal distance and success, with repeated trials and uncertainty. Add a diagnostic condition supplying manually annotated current pixel positions at each replanning step in place of predictor propagation. If prediction error decreases without control improvement, the assumed link is weakened; if accurate pixel positions specifically rescue control, tracking error is a separable bottleneck. This extends the paper's prediction-only capacity test. capacityplanningevaluation
8.3 Reading coverage
Visual audit: Read all five primary-text chunks, including references and Appendices A–E. Rendered and visually inspected the listed PDF pages at 200 DPI: title/authors/version; dataset composition; both methods; training and planning equations; every original Figure 1–8 and Table 1–7; evaluation, capacity and limitations. These pages cover all scientific evidence used in the report, visual explanations and proposed checks. Inspected all six final crops: Figure 1 at 200 DPI, Figure 6 at 300, Table 2 at 400, adjacent Tables 3–5 at 250, Figure 4 at 400 and Table 7 at 400. Crops retain original labels, legends and table headers; captions are interpreted using the inspected full pages. Bibliography-only pages 9–11 were read as text, not rendered. Supplemental videos and external code/data remain uninspected. Figure 5's illustrative one-frame context versus the stated two-frame experiments, Figure 6's unhatted output label, Equation (4)'s sum/average wording and unspecified Table 7 uncertainty are explicitly preserved.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 12, 13, 14, 15. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- 1 Introduction
- 2 Related Work
- 3 Data-Driven Robotic Manipulation
- 4 The RoboNet Dataset
- 4.1 Data Collection Process
- 4.2 The Diverse Composition of RoboNet
- 4.3 Using and Contributing to RoboNet
- 5 Robot-Agnostic Visual Control: Model Training and Experiments
- 5.1 Visual Foresight: Experimental Methodology
- 5.2 Visual Foresight: Zero-Shot Generalization to New Viewpoints and Backgrounds
- 5.3 Visual Foresight: Few-Shot Adaptation to New Robots
- 5.4 Visual Foresight: Model Capacity Experiments
- 5.5 Inverse Model: Multi-Robot and Multi-Viewpoint Reaching
- 6 Discussion
- Acknowledgments
- References
- A Visual Foresight Preliminaries
- A.1 Action conditioned video-prediction model
- A.2 Sampling-based Planning
- B Data Collection Details
- B.1 State and Action Space
- B.2 Exploration Policy
- C Database Implementation Details
- D Description of Benchmarking Tasks
- E Experimental evaluation of adaptation to unseen gripper
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- All five supplied text chunks were read, including references and Appendices A–E. Original PDF Figures 1–8 and Tables 1–7 were visually inspected; bibliography-only pages 9–11 were read as text.
- Identity matches the catalog title and all nine authors. The inspected edition is arXiv:1910.11215v2 dated 2 January 2020, with a CoRL 2019 footer. The catalog's 24 October 2019 submission date is not this revision date; v1 was not supplied or compared.
- No external project pages, code, dataset files or supplemental videos were inspected, and no experiments were reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
identityPDF p. 1, title, author/affiliation block, arXiv margin and conference footer
The title and nine authors match the catalog. This PDF identifies arXiv:1910.11215v2, 2 January 2020, and CoRL 2019. Affiliations are UC Berkeley, Stanford University, University of Pennsylvania, and CMU.
Go to primary source ↓datasetPDF pp. 1–2, Introduction and Figure 1; pp. 4–5, Sections 4.2–4.3, Figure 2 and Table 1
RoboNet shares image/action trajectories across seven robot platforms and four institutions. The introduction reports roughly 162,000 trajectories and nearly 15 million frames. Table 1 lists 113 camera configurations, seven arena types, ten inserts, ten gripper configurations and four lab environments.
Go to primary source ↓data-countsPDF p. 1, Introduction; p. 4, Section 4.2; p. 5, Table 1 including caption
Table 1 lists Sawyer 68k, Baxter 18k, WidowX 5k, Franka 7.9k, Kuka 1.8k, Fetch 5k and GoogleRobot 56k trajectories. Its caption says six arms despite seven entries. Section 4.2 reports 104.4k collected trajectories plus 5k Fetch and 56k Google; these rounded counts do not fully reconcile with Table 1.
Go to primary source ↓collectionPDF p. 4, Sections 4.1–4.2; pp. 13–14, Appendix B.1–B.2
The common interface uses root-frame end-effector deltas. Most robots have XYZ, yaw and gripper state; gripper actions are binary and actions block until completion or timeout. Random Gaussian policies use per-robot variances; grasping is biased by height thresholds. Google R3 uses pushing motions; Fetch explores only x and y. RGB cameras vary in exposure and appearance.
Go to primary source ↓databasePDF p. 5, Section 4.3; p. 14, Appendix C
The paper describes contribution uploads and manual quality checks, filtering trajectories by attributes, a Pandas interface, HDF5 storage and MP4 videos. It points to project-hosted code; the supplied PDF does not state a dataset license or provide an immutable split manifest.
Go to primary source ↓forwardPDF p. 3, Section 3, problem statement and prediction preliminaries
Image/action trajectories train an action-conditioned video model; designated object pixels and goal pixels define relocation tasks. A planner selects a sequence, executes its first action and replans from a new image.
Go to primary source ↓inverse-methodPDF p. 4, Section 3, inverse-model preliminaries; p. 7, Section 5.5
The separate inverse model is trained by supervised regression with horizon T=1. Inference uses two context frame/action pairs, the current image and a goal image to predict an action, then repeats closed-loop. The evaluated training subset contains Sawyer and Franka.
Go to primary source ↓architecturePDF p. 12, Appendix A.1, Equations (1)–(2) and Figure 5; p. 13, Figure 6
The deterministic recurrent video model uses convolutional LSTMs, convolution layers and skip connections to predict flow transformations. Figure 6 shows action tiling, compositing masks and an initial-image branch. Figure 5 depicts one context frame; the text explicitly says all experiments use two.
Go to primary source ↓trainingPDF p. 12, Appendix A.1, Training details and Figure 5
The recipe states 160k pretraining iterations, batch size 16, Adam, and another 150k fine-tuning steps. Learning rate starts at 1e-3; the wording about linear annealing to zero after 200k steps is ambiguous. Figure 5 shows prediction/true-image losses without a fully specified loss formula.
Go to primary source ↓planningPDF p. 12, Appendix A.2, Equations (3)–(4); p. 13, Appendix A.2, Equation (5) and following paragraphs
Flow propagates an initially one-hot designated-pixel distribution. The cost uses expected distance to a goal; Equation (4) sums over time although adjacent prose says averaged. MPPI uses 600 samples, a 15-step horizon, three iterations, exponential cost weighting and temporal low-pass filtering. Predictor propagation estimates the designated pixel after execution.
Go to primary source ↓evaluationPDF p. 5, Section 5.1; p. 14, Appendix D and Figure 7
Relocation uses held-out objects, fixed initial/goal positions and distance measured with a tape. An operator judges success by whether the object mostly covers the goal. Models within an experiment share tasks; the authors warn against comparison across experiments because robots and operators change task difficulty.
Go to primary source ↓viewpointPDF pp. 5–6, Section 5.2; p. 6, Table 2
The study compares one-view and 90-view training on five relocation tasks from two views. Seen-view distance is 14.8 ± 3.8 versus 9 ± 2.2 cm; held-out-view distance is 23.2 ± 2.6 versus 16.2 ± 2.9 cm. Table 2 identifies standard errors.
Go to primary source ↓backgroundPDF p. 6, Section 5.2 and Figure 3
The same multi-view model achieves 14.4 ± 2 cm, standard error, on a similar task set with a substantially different background. Figure 3 presents a successful pushing example with separate predicted and actual trajectories.
Go to primary source ↓kukaPDF p. 6, Section 5.3 and Table 3
Kuka success rates are 10% with 400 scratch trajectories, 30% with 1,800 scratch trajectories, and 40% after RoboNet pretraining excluding Kuka, R3 and Fetch followed by 400-trajectory fine-tuning.
Go to primary source ↓frankaPDF p. 6, Section 5.3 and Table 4; p. 7, Figure 4 and Section 5.3
Franka success is 20% with 400 scratch trajectories, 35% with 8,000 scratch trajectories and 40% with RoboNet pretraining excluding Franka, R3 and Fetch plus 400-trajectory fine-tuning. Figure 4 shows a successful plastic-cup relocation with predicted and actual sequences.
Go to primary source ↓baxterPDF p. 6, Section 5.3 and Table 5; p. 7, Section 5.3
With 300 Baxter adaptation trajectories, success is 33% from scratch, 83% after Sawyer-only pretraining and 58% after RoboNet pretraining without Baxter. The surrounding text also states R3 and Fetch were excluded from pretraining for computational reasons.
Go to primary source ↓capacityPDF p. 7, Section 5.4; p. 8, Discussion
On held-out Sawyer sequences, a 200M-parameter predictor has average per-pixel l1 error 0.104 ± 0.057; a 500M predictor has 0.0847 ± 0.045. The control models have 5M–75M parameters. Similar plateaued training/validation error motivates the authors' underfitting hypothesis; the larger-model comparison is a prediction evaluation.
Go to primary source ↓inverse-resultsPDF p. 7, Section 5.5 and Table 6
The same inverse model succeeds in 4/5 Sawyer front-view, 5/5 Sawyer unseen-view and 4/5 Franka front-view reaching tasks. Goals are images of reachable gripper states; a human judges success. The authors report poor behavior on more complex object manipulation.
Go to primary source ↓gripperPDF p. 14, Appendix E; p. 15, Table 7 and Figure 8
For a Sawyer with an unseen Robotiq two-finger gripper, 300 new trajectories support adaptation; evaluation averages ten trajectories per model. Distances are zero-shot 15.5 ± 2.6, scratch 17 ± 1.8, Sawyer-only pretraining 9.8 ± 2.1 and all-RoboNet pretraining 14.7 ± 2.1 cm. The table does not identify what its ± values represent.
Go to primary source ↓limitationsPDF p. 8, Section 6, Limitations and Future Directions
The authors identify low-fidelity pushing/pick-and-place tasks, imperfect video predictions and limited interactions from predetermined random policies. Improved prediction translating into improved control and richer collection policies are future directions.
Go to primary source ↓8.5 Primary sources
RoboNet: Large-Scale Multi-Robot Learning ↗
PDF · 8,811 extracted words
Source fingerprint
01a03705dfb76192445b8c09fa3e85d67a1d9165b2c00df3672cfd10cc66052a