RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robots
1. Paper overview
In one sentence: RoboCasa turns diverse simulated kitchens and human demonstrations into larger imitation-learning datasets, improving average atomic-task success while leaving composite behavior and real-world reliability unresolved. e02e03e04e05e06e08e09e10e11e07
| At a glance | What to know |
|---|---|
| Research problem | Author claim Real robot demonstrations are expensive and cover limited situations. The authors argue that useful simulation must combine physical and visual realism, diverse scenes and tasks, and large demonstration datasets. RoboCasa tests whether these ingredients can improve generalization and support real-robot learning. e02 |
| Core mechanism | Source description A MuJoCo/RoboSuite kitchen platform combines 10 floor plans and 12 styles into 120 scenes, with articulated appliances and 2,509 objects across 153 categories. AI tools supply textures and many assets; embodiment support exceeds the single Panda platform evaluated in the main experiments. e02e03e06 |
| A key reported result | Multitask atomic-task generalization: Generated-3000: 47.6; Generated-300: 35.0; Generated-100: 26.3. Reported average task success (%). 24 evaluated tasks; 50 trials per task across five fixed scenes, unseen object instances and two unseen styles; AI textures in training, human-curated textures at evaluation. Human-50: 28.8. The largest generated setting improves the reported average by 18.8 percentage points. Generated-data scaling helps on average, but Human-50 beats Generated-100. Human training includes navigation while generated training excludes it; this is not a matched-size human-versus-synthetic comparison. No uncertainty is reported. e06e08 |
| Reading caution | Source description Successful generated trajectories can still contain collisions and jerky motion. Tasks require human implementation; the release concentrates on kitchens and coarse manipulation, without highly dexterous, deformable or bimanual skills. e12 |
Core contributions
- Source description
A MuJoCo/RoboSuite kitchen platform combines 10 floor plans and 12 styles into 120 scenes, with articulated appliances and 2,509 objects across 153 categories. AI tools supply textures and many assets; embodiment support exceeds the single Panda platform evaluated in the main experiments. e02e03e06
Figure 3. Layout variation supplies a structured source of environmental diversity. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the two rows as ten spatial templates, rather than ten trained policies or ten evaluation trials. The labels distinguish one-wall, L-shaped, galley, U-shaped, G-shaped and wraparound arrangements, including island variants. Section III-B adds a separate style dimension: every plan can take any of twelve styles, producing 120 scenes. Textures can then vary again across walls, floors, counters and cabinet panels. This distinction matters when interpreting generalization: a change in appearance is different from a change in spatial organization. The main evaluation uses five fixed scenes and explicitly holds out two styles; the figure itself does not identify those evaluation scenes. e03e06e16
What it supports. The framework separates spatial layout from architectural appearance, allowing systematic environmental variation. These ten plans combine with twelve styles to form the reported 120 scenes. The visual supports the diversity of room organization; it does not measure how many of those configurations a policy can solve.
Where the evidence stops. These are designed kitchen templates, not a measured sample of household diversity. The learning study uses native MuJoCo images, despite the platform's separate Omniverse rendering support; the pictured environments alone do not establish transfer quality.
2. Motivation
2.1 The problem and the proposed response
Real robot demonstrations are expensive and cover limited situations. The authors argue that useful simulation must combine physical and visual realism, diverse scenes and tasks, and large demonstration datasets. RoboCasa tests whether these ingredients can improve generalization and support real-robot learning. e02
2.2 What this reading follows
A generalist robot needs varied experience, but constructing a simulator does not automatically provide useful training behavior. RoboCasa connects scene design, task authoring and demonstration generation. Its kitchens vary in layout, style, objects and textures; language models suggest activities that humans turn into tasks; MimicGen adapts human trajectories to new configurations. The experiments then ask whether this experience improves action policies. Read the visuals as separate tests of environment coverage, dataset scaling, composite transfer and real-robot co-training. The results support a practical data-generation strategy, with substantial remaining gaps in reliability and no demonstration of a unified learned world-action model. e02e03e04e05e06e08e09e10e11e07
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Benchmarks & simulators |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
3.1 Evidence-based assessment
Supports the recorded classification
The recorded benchmark/simulator category is supported by the environment, task suite and demonstration-generation contributions. World-action architecture, prediction paradigm and quadrant are not applicable: the evaluated Transformer predicts actions, while MuJoCo provides physical dynamics. AI-generated assets, LLM-authored tasks and action diffusion do not establish a unified learned world-action model. e02e04e05e07e14
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
5. Method in detail
5.1 Separate the simulator's variation from the policy's generalization
RoboCasa makes several design choices independently adjustable. A floor plan determines spatial organization; a style changes design elements such as appliances and cabinet appearance; additional textures change surfaces; object instances introduce new shapes and affordances. These dimensions create many possible observations, but their presence in a simulator is only the starting point. Training data must actually visit useful configurations, and evaluation must say which dimensions were held out. Here the reported policy test uses unseen object instances and two unseen styles among five fixed evaluation scenes. It does not claim that every floor plan is unseen. The main data also use the lightweight MuJoCo renderer. Thus the platform's support for photorealistic rendering and multiple robot embodiments should be read separately from the particular images and Panda embodiment used to obtain the learning results. e02e03e06e16
Figure 6. Language models propose tasks; human curation and implementation make them simulatable. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow the light-blue activity prompt into GPT-4 and then the green activity list. The darker-blue task prompt supplies an activity, available objects and skills, and examples; its output is a concrete blueprint containing a goal, objects, fixtures and a skill list. The right-hand snapshots illustrate the microwave example. The caption permits GPT-4 or Gemini 1.5 for the second stage. Section IV-B and the appendix supply an essential step absent from the arrows: humans filter or modify flawed proposals and implement the tasks. The arrows therefore describe task authoring, not a learned policy receiving language-model commands while it controls the robot. e04e17
What it supports. The release combines twenty selected activity categories with 75 implemented composite tasks. LLM prompting helps expand the candidate task space, while simulator constraints and human review decide which suggestions become usable benchmarks. This is a method for defining experience, separate from the MimicGen method for generating demonstrations.
Where the evidence stops. Skill 3 in the numbered list is pick(bowl), but the third snapshot in the left column repeats place(bowl). The faithful diagram is schematic; this label conflict prevents treating it as an unambiguous executable sequence.
5.2 Trace one demonstration from privileged generation to feedback control
MimicGen starts from an executed human demonstration with a known sequence of object-centric subtasks. In a new scene it transforms each segment using the relevant object's current pose, connects the segments and executes the resulting trajectory. Simulation success checks reject failed attempts. This procedure can turn a small human seed set into a much larger action dataset, but it relies on subtask definitions and state information available during generation. The learned BC policy has a different information flow: language, camera images and proprioception are encoded and passed through a Transformer to predict actions. It predicts ten actions but executes only the first before replanning. The robot therefore receives fresh observation feedback instead of blindly replaying a complete generated demonstration. The reported setup does not learn a future world-state predictor or use the task-authoring LLM as an online controller. e05e06e07e04
5.3 Treat scaling, composition and real transfer as different tests
The atomic study asks whether additional generated experience improves a fixed policy family on held-out situations. Its average improves, although individual tasks do not follow a uniform trend. The composite study asks a different question: can atomic pretraining help a separately fine-tuned policy complete a longer sequence from fifty target demonstrations? Its low success suggests that reusable manipulation behavior alone is insufficient under the tested setup. That last sentence is reader analysis, not an isolated causal finding: the paper does not independently vary sequence planning and low-level dexterity. Finally, the real study uses co-training with target-task real demonstrations, rather than the composite fine-tuning protocol or zero-shot deployment. Together these experiments motivate simulation as a useful data source, while leaving the mechanisms and conditions for reliable general-purpose behavior unresolved. e06e08e09e10e11e12
5.4 Training and inference
During training
RoboMimic behavioral cloning trains the approximately 20M-parameter Transformer for 500k gradient steps at learning rate 1e-4 with warmup. The paper does not give a formal loss equation or freezing schedule. Dataset images use native MuJoCo rendering; photorealistic Omniverse support is not the rendering protocol used for these experiments. e07e16
Atomic policies are trained separately on Human-50 or each generated dataset size. Composite experiments compare training from scratch with fine-tuning an atomic pretrained policy, using 50 human demonstrations per target. Real experiments instead co-train real target-task data with simulation data; these are distinct transfer protocols. e06e09e10
During inference
The BC policy processes a ten-observation history, predicts ten actions, executes the first, then replans from feedback. Privileged object poses support dataset generation; the described deployed policy consumes images, proprioception and language. No inference-time MimicGen trajectory transformation or LLM task generation is described. e05e07
Simulation uses operational-space control. The real setup uses different DROID-based control, cameras, lighting and base placement. Section V-C states 20 Hz simulation versus 15 Hz real control; Section VII separately states a 0.04-second simulator timestep. These settings should remain distinct until their relationship is clarified. e10e13
5.5 Implementation flow
- Construct varied but structured scenes
Layouts and architectural styles are composable. Random textures alter walls, floors, counters and cabinets; articulated microwave doors and stove knobs allow physical interaction and appliance state changes. Asset diversity is curated, with defective candidates filtered before inclusion. e03
- Turn activity suggestions into tasks
GPT-4 supplies candidate activities; humans select twenty. GPT-4 or Gemini 1.5 proposes task blueprints using available skills, objects, fixtures and examples. Humans repair logical errors and write implementations. This is offline benchmark authoring, not an LLM controlling the robot during evaluation. e04e07
- Expand demonstrations through execution
MimicGen decomposes demonstrations into known object-centric subtasks, transforms segments to new reference-object poses, stitches them and executes them in simulation. Subtask boundaries require annotations or reusable detection functions. Rejection sampling retains successful rollouts. Navigation is excluded from generated data. e05e06
- Encode observations and predict actions
CLIP encodes the language goal; dedicated ResNet-18 stacks process the eye-in-hand and two workspace views, with FiLM conditioning. Encoded observations and proprioception feed a six-layer Transformer. Its prediction target is robot actions, with no learned future-image or world-state prediction specified. e07
6. Experiments & results
RoboCasa packages diverse kitchen simulation, human-curated tasks and scalable demonstration generation for imitation learning. Its strongest controlled trend is higher average atomic-task success with larger generated datasets; composite-task and real-robot results remain modest. The contribution is infrastructure for learning and evaluation, with BC policies as baselines.
6.1 Read the original evidence
Figure 7. Larger generated datasets improve overall performance, with uneven gains across skills. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the legend: the orange bars use fifty human demonstrations per task; the three blue-green bars use 100, 300 or 3,000 generated demonstrations per task. The horizontal groups aggregate related tasks into skill families, while the last group reports overall success. Section V-A defines the common evaluation: fifty trials per task across five fixed scenes, unseen object instances and two unseen styles. Use Figure 13 for exact numbers instead of estimating bar heights. The generated training sets exclude navigation, leaving 24 evaluated manipulation tasks; the larger Human-50 training set includes that additional task. e06e08
What it supports. The exact overall values are 28.8% for Human-50 and 26.3%, 35.0% and 47.6% for the three generated settings. Increasing generated-data volume improves the reported aggregate, and the largest setting exceeds Human-50 by 18.8 percentage points. The smallest generated set does not beat the human baseline.
Where the evidence stops. Human and generated datasets differ in size and task coverage, so this is not a controlled estimate of synthetic versus human demonstration quality. No error bars or seed uncertainty are supplied, and individual tasks can regress as data increases.
Figure 8. Atomic pretraining helps several longer tasks, but none becomes reliable. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each row as a separate single-task policy experiment, rather than one multitask composite policy. Both columns use fifty human demonstrations for the target task. Scratch begins without the atomic pretrained policy; Fine-tuning adapts the policy trained on the full generated atomic dataset. The five examples require sequences such as moving vegetables onto a cutting board, stocking cans beside existing cans, or placing a pan and sponge into a sink before turning on water. Follow both the relative improvement and the absolute rate: a nonzero entry demonstrates occasional completion, but does not imply dependable execution of the whole sequence. e09e04
What it supports. ArrangeVegetables rises from 2% to 12%; MicrowaveThawing, RestockPantry and PreSoakPan rise from zero to 2%, 6% and 4%. PrepareCoffee remains at zero. Pretraining supplies some reusable behavior, but the source reports continuing difficulty with fine manipulation and transitions between task stages.
Where the evidence stops. Only five of the 75 composite tasks are tested here. The table supplies no uncertainty, and Section V-B does not state its trial or seed counts. Atomic-data improvements therefore cannot be extrapolated to robust performance across the complete composite suite.
Figure 10. Simulation co-training improves a small real-kitchen study while success remains low. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. First separate the upper Seen Obj block from the lower Unseen Obj block. The distinction refers to categories present in the real demonstrations: each task has fifty demonstrations from five categories, and evaluation adds three unseen categories. Read horizontally to compare Real only with Real + Sim for the same task. The latter uses the same target-task real data plus single-stage simulation demonstrations. Section V-C identifies the ± values as standard deviations across three seeds, not confidence intervals. Each Task average row summarizes three tasks without an accompanying uncertainty estimate. The three rows concern counter-to-sink, sink-to-counter and counter-to-cabinet execution by a real Panda. e10e11
What it supports. Co-training raises the reported task average from 13.6% to 24.4% on seen categories and from 2.6% to 9.3% on unseen categories. For seen counter-to-cabinet transfer, the row changes from 8.0 ± 1.6% to 22.0 ± 5.8%. These are real execution results under supervised co-training, with limited absolute reliability.
Where the evidence stops. Real demonstrations are required, so this is not zero-shot sim-to-real transfer. The unseen counter-to-cabinet estimate is 11.1 ± 11.0%, illustrating substantial seed variability. The study does not isolate which simulator property causes the gain.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Multitask atomic-task generalization 24 evaluated tasks; 50 trials per task across five fixed scenes, unseen object instances and two unseen styles; AI textures in training, human-curated textures at evaluation. | Generated-3000: 47.6; Generated-300: 35.0; Generated-100: 26.3. Reported average task success (%) | Human-50: 28.8. The largest generated setting improves the reported average by 18.8 percentage points. Generated-data scaling helps on average, but Human-50 beats Generated-100. Human training includes navigation while generated training excludes it; this is not a matched-size human-versus-synthetic comparison. No uncertainty is reported. e06e08 |
| Composite-task transfer from atomic pretraining Five independently trained single-task policies; 50 human demonstrations per task; scratch versus atomic-pretraining fine-tuning. | Fine-tuning: ArrangeVegetables 12, MicrowaveThawing 2, RestockPantry 6, PreSoakPan 4, PrepareCoffee 0. Task success (%) | Scratch: 2, 0, 0, 0, 0, respectively. Pretraining improves four examples but leaves all success rates low. These five tasks do not establish performance across all 75 composite tasks. Trial counts and uncertainty are unspecified for this comparison. e09 |
| Real-kitchen pick-and-place transfer Three tasks, each with 50 real demonstrations; separate task policies, three seeds; five seen and three unseen object categories relative to real demonstrations. | Real + Sim: 24.4 on seen categories; 9.3 on unseen categories. Task-averaged success (%) | Real only: 13.6 seen; 2.6 unseen. Improvements are 10.8 and 6.7 percentage points. Co-training helps this real deployment but is not zero-shot transfer. Per-task rows report seed standard deviations; task averages do not. Unseen-category success remains below ten percent on average. e10e11 |
| PickPlaceCounterToSink policy comparison Separate single-stage comparison in Section IX; BC history 10 versus diffusion history 2; data and evaluation trial counts unspecified. | BC-Transformer: 56. Success rate (%) | Diffusion Policy: 12; diffusion prediction/action horizons are 16/8, with DDIM 100 training and 10 inference timesteps. Architecture, history and execution horizon differ. This is not an isolated history-length ablation and must not be merged with the 44% BC multitask result in Figure 13. e07e08e14 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 13. Task-level results qualify the aggregate scaling trend. Original paper, p. 16 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read across each row to track one task as the training dataset changes. Unlike the percentage axis in Figure 7, these cells are proportions: 0.94 means 94% success. The last row supplies the reported averages, while the named rows reveal differences hidden by grouping skills together. Compare CloseSingleDoor and CoffeeSetupMug in the final column to see the large spread in difficulty. Then inspect CloseDoubleDoor across the last two columns and TurnOffMicrowave across all four. These counterexamples are why a larger average cannot be read as a guarantee that every behavior improves. No navigation row appears in this evaluated set. e06e08
What it supports. Generated-3000 reaches 94% on CloseSingleDoor but only 12% on CoffeeSetupMug. CloseDoubleDoor falls from 62% with Generated-300 to 46% with Generated-3000; TurnOffMicrowave declines from 70% with Human-50 to 60% with Generated-3000. The table supports an average scaling benefit with important task-specific exceptions.
Where the evidence stops. This is a data-size diagnostic, not an isolated ablation of physical realism, AI textures or policy architecture. With no per-task uncertainty supplied, the observed regressions identify questions for repeated evaluation rather than established causal failure mechanisms.
7. Analysis & limitations
7.1 What the evidence leaves open
Successful generated trajectories can still contain collisions and jerky motion. Tasks require human implementation; the release concentrates on kitchens and coarse manipulation, without highly dexterous, deformable or bimanual skills. e12
Average scaling hides regressions: CloseDoubleDoor drops from 62% to 46% between Generated-300 and Generated-3000. Neither physical realism nor AI asset generation is independently ablated, so the real-robot gains cannot be uniquely attributed to either factor. e08e10e11
Source inconsistencies remain visible: Figure 6 lists pick(bowl) as skill three but labels the corresponding snapshot place(bowl); Section VIII-C says 25 tasks where the main generated-data protocol specifies 24. The explicit main protocol governs the reported experiment scope. e17e16
7.2 Questions for discussion
- Does removing undesirable successful trajectories improve transfer at fixed dataset size (e12)?
- How much of the BC–diffusion gap survives matched history and feedback schedules (e07, e14)?
8. Reproducibility audit
8.1 Requirements and known gaps
Reproduction requires scene/style/object splits, demonstration segmentation, success checks, native rendering, language conditioning and the stated policy configuration. Exact optimizer/batch settings, warmup duration, action normalization, code versions and real/simulation sampling ratio are not fully specified in the supplied implementation and experiment descriptions. e05e06e07e10e16
The simulator benchmark uses an RTX A5000 and AMD EPYC 7543 32-Core CPU: ten episodes yield 25.2 fps with rendering versus 31.9 without, with resets of 9.50 versus 9.46 seconds. These measurements do not specify training hardware or total training cost. e13
Proposed checks: match history and execution horizons when comparing BC with diffusion; separately compare collision-filtered generated data against equal-sized randomly selected successful trajectories. Both would test mechanisms left unresolved by the reported results. e12e14
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Disentangle policy family, observation history and feedback cadence
Reader-proposed experiment, not performed here: repeat the Section IX PickPlaceCounterToSink comparison with the same training trajectories, visual encoder design, scene/object splits and evaluation trials. Cross BC-Transformer versus Diffusion Policy with observation histories of two versus ten frames. In a separate controlled comparison, match the number of executed actions before replanning and document prediction horizon and compute. Use repeated training seeds and report task success with uncertainty. If extending diffusion history closes the gap under matched cadence, that supports the authors' history explanation; if the gap persists, history alone is insufficient. Keep this experiment separate from Figure 13's multitask result. e07e14e06e08
Check 2: Test whether successful but rough trajectories limit learning
Reader-proposed experiment, not performed here: begin with the same pool of successful MimicGen trajectories and define collision and motion-jerk filters before evaluation. Compare training on filtered data with an equally sized random subset of successful trajectories, matched by task, scene and object distribution; retain an unfiltered full-data baseline. Keep the BC configuration and held-out scenes fixed. Record task completion and undesirable contacts during policy execution, plus generator rejection rate. Higher success or fewer contacts at matched size would support trajectory quality as a useful improvement beyond volume. No benefit would weaken that explanation; any gain must also be checked for changes in task difficulty caused by filtering. e05e06e07e08e12
8.3 Reading coverage
Visual audit: Visually inspected the title and full author block, platform comparison, scene/assets figures, task-generation diagram and caption, aggregate and task-level scaling results, composite and real-robot tables, real setup, and appendix pages covering simulator hardware/timing, dataset rendering, policy configuration and task definitions. All six final original crops were viewed; narrow result tables were rendered at 400 DPI. Figure 6's pick(bowl)/place(bowl) mismatch is disclosed. Section V-C's 20 Hz simulation control and Section VII's 0.04-second timestep are preserved separately in the base report. Pages 9–11 contain acknowledgments/references and were read as text, not visually inspected. No external code, videos, supplements or later editions were inspected.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 12, 13, 14, 15, 16. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- PDF pp. 1–2: Abstract and I. Introduction
- PDF pp. 2–3: II. Related Work
- PDF pp. 3–4: III. RoboCasa Simulation, A–C
- PDF pp. 4–6: IV. RoboCasa Activity Dataset, A–C
- PDF pp. 6–8: V. Experiments, A–C
- PDF p. 8: VI. Conclusion and limitations
- PDF pp. 9–11: Acknowledgments and References
- PDF p. 12: VII. Simulator and VIII. Tasks and Datasets, A–C
- PDF pp. 12–13: IX. Policy Learning Implementation
- PDF pp. 14–16: Figures 11–13, task catalogs and full atomic results
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- All six supplied text chunks were read individually. The extraction's image omission was addressed by visually inspecting PDF pages 1–8 and 12–16 and all six final crops; reference-only pages 9–11 were read as text.
- This is the supplied RSS 2024 proceedings edition. Its title and complete author list match the catalog. No explicit revision identifier or revision chain is supplied; equivalence to any other edition was not assessed.
- No external code, project website, datasets, videos or separate supplements were inspected, and no experiments were reproduced.
- The PDF describes the 75 composite tasks but provides only representative examples in its appendix; full task implementations and their exact success predicates are outside the supplied material.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title, author block and proceedings header
The title and all eight authors match the catalog. Credits are Soroush Nasiriany, Abhiram Maddukuri, Lance Zhang, Adeet Parikh, Aaron Lo, Abhishek Joshi, Ajay Mandlekar and Yuke Zhu. Affiliations are The University of Texas at Austin and NVIDIA Research; Maddukuri and Zhang share equal contribution. The header identifies Robotics: Science and Systems 2024, Delft, July 15–19, 2024. No explicit revision number is printed.
Go to primary source ↓e02PDF pp. 1–3, Abstract, Section I and Section III-A; Figure 1; Table I, RoboCasa column
RoboCasa addresses scarce robot data with kitchen simulation, task definitions and demonstrations. It extends RoboSuite/MuJoCo for room-scale mobile manipulation, inherits controllers and model formats, supports multiple embodiments and offers Omniverse rendering. These are simulator capabilities, not evidence of a learned world model.
Go to primary source ↓e03PDF p. 4, Sections III-B–C and Figures 3–4; p. 5, Figure 5
Ten floor plans combine with twelve styles for 120 scenes. Four surface categories each have 100 AI-generated textures. The asset collection has 2,509 objects across 153 categories, including 1,592 from Luma.ai, plus articulated appliances with state changes. Objaverse supplies the other object source.
Go to primary source ↓e04PDF p. 5, Sections IV-A–B; p. 6, Figure 6 and caption; p. 12, Section VIII-B
Twenty-five atomic tasks cover eight skill families. GPT-4 activity suggestions are manually reduced to twenty activities; GPT-4 or Gemini 1.5 then proposes tasks conditioned on skills, objects, fixtures, constraints and examples. Humans filter or modify logical flaws and implement 75 composite tasks. Appendix examples include an unavailable blender and misuse of a coffee button.
Go to primary source ↓e05PDF pp. 5–6, Section IV-C, including rejection-sampling paragraph
Four operators collect 50 demonstrations per atomic task, totaling 1,250, in randomized scenes using a SpaceMouse. MimicGen uses known object-centric subtask sequences and annotated demonstrations, transforms segments to new object poses, stitches them and executes the trajectory. Only successful generation attempts are retained; simulation processes can run in parallel.
Go to primary source ↓e06PDF p. 6, Section V-A, dataset list, footnotes 1–2 and evaluation paragraphs; p. 7, Figure 7 caption
Human-50 contains all 25 atomic tasks. Generated-3000 uses 72,000 Objaverse trajectories on 24 tasks; navigation is excluded because the generator cannot produce mobile manipulation trajectories. Another 28,000 AI-object trajectories belong to the release. Smaller generated subsets contain 2,400 and 7,200 trajectories. Evaluation reports 50 trials per task over five fixed scenes, unseen object instances, two unseen styles and human-curated textures, using a Panda on an Omron base.
Go to primary source ↓e07PDF p. 12, Section IX; p. 13, first paragraph
BC-Transformer consumes ten observations and a language goal encoded by CLIP. Observations contain end-effector and mobile-base poses and three camera images, each with a dedicated ResNet-18 encoder; FiLM is used in the visual representation. A six-layer Transformer has about 20M trainable parameters. It predicts ten actions, executes the first and replans. Training uses 500k gradient steps, learning rate 1e-4 and warmup. A formal loss equation, batch size, full optimizer/warmup specification and freezing schedule are not given here.
Go to primary source ↓e08PDF p. 7, Figure 7; p. 16, Figure 13, Average and named task rows
Reported average success proportions are 0.288, 0.263, 0.350 and 0.476 for Human-50, Generated-100, Generated-300 and Generated-3000. CloseDoubleDoor falls from 0.62 to 0.46 between the last two columns; TurnOffMicrowave is 0.70, 0.70, 0.62 and 0.60. Generated-3000 reaches 0.94 for CloseSingleDoor and 0.12 for CoffeeSetupMug. PickPlaceCounterToSink is 0.44 in this multitask table. No uncertainty is tabulated.
Go to primary source ↓e09PDF p. 7, Section V-B and Figure 8, all rows
Separate policies use 50 human demonstrations per composite task, either from scratch or by fine-tuning atomic-data pretraining. Scratch/fine-tuning success percentages are ArrangeVegetables 2/12, MicrowaveThawing 0/2, RestockPantry 0/6, PreSoakPan 0/4 and PrepareCoffee 0/0. Reported failures include fine manipulation and transitions between stages. Seed counts, evaluation trial counts and uncertainty for this table are not stated in Section V-B.
Go to primary source ↓e10PDF p. 8, Section V-C and Figure 9
Real experiments use a Panda with DROID infrastructure on three pick-and-place tasks. Each task has 50 real demonstrations from five object categories. Real + Sim co-trains target-task real data with all single-stage MimicGen simulation data. Results average three seeds and evaluate five seen and three unseen categories, defined relative to real demonstrations. Simulation uses operational-space control at a stated 20 Hz; the real controller differs and runs at 15 Hz. Cameras, lighting and base placement also differ.
Go to primary source ↓e11PDF p. 8, Figure 10, Seen Obj/Unseen Obj rows and task averages; Section V-C
Real-only versus Real + Sim success percentages are: seen counter-to-sink 12.7 ± 2.5 versus 22.0 ± 2.8, sink-to-counter 20.0 ± 5.9 versus 29.3 ± 4.1, counter-to-cabinet 8.0 ± 1.6 versus 22.0 ± 5.8; unseen counterparts 3.3 ± 4.7 versus 8.9 ± 7.9, 1.1 ± 1.6 versus 7.8 ± 4.2, and 3.3 ± 4.7 versus 11.1 ± 11.0. Errors are standard deviations across three seeds. Task averages are 13.6 versus 24.4 for seen objects and 2.6 versus 9.3 for unseen objects; no uncertainty is supplied for these averages.
Go to primary source ↓e12PDF p. 8, Section VI, limitations paragraph
Authors report jerky motions and collisions in trajectories that satisfy task success, ongoing human implementation of LLM-proposed tasks, weak composite-task learning, kitchen-only scope and missing highly dexterous, deformable and bimanual manipulation. Filtering undesirable successful trajectories is proposed as future improvement.
Go to primary source ↓e13PDF p. 12, Section VII
The simulator speed benchmark uses ten PickPlaceCounterToCab episodes, NVIDIA RTX A5000 rendering and an AMD EPYC 7543 32-Core CPU for physics. Reset/step performance is 9.50 seconds/25.2 fps with rendering and 9.46 seconds/31.9 fps without it. The section states 0.04 seconds per world timestep, equivalent to 25 fps. This is simulator benchmarking hardware, not a policy-training compute specification.
Go to primary source ↓e14PDF p. 13, Section IX, diffusion-policy comparison
Diffusion Policy shares the ResNet/FiLM observation encoder design, uses observation history 2, prediction horizon 16, action horizon 8, and DDIM with 100 training and 10 inference timesteps. The single-stage PickPlaceCounterToSink comparison reports 56% BC-Transformer versus 12% Diffusion Policy. The authors hypothesize that BC's longer history of 10 matters. Training data, seed counts and evaluation trial counts for this particular comparison are not specified.
Go to primary source ↓e15PDF p. 12, Sections VIII-A–B; pp. 14–15, Figures 11–12
Language variants distinguish object categories, stove burners and navigation destinations. Figure 11 lists all atomic tasks; Figure 12 gives representative composite tasks across twenty activities, not a complete enumeration of all 75. Some task prose contains editorial inconsistencies, including TurnOffSinkFaucet described as beginning water flow on p. 14.
Go to primary source ↓e16PDF p. 12, Section VIII-C; p. 6, Section V-A and footnotes
Experimental dataset images use the lightweight native MuJoCo renderer with randomized AI textures; Omniverse rerendering is described as a public-release option. Section VIII-C loosely calls the datasets 25-task datasets, whereas the main protocol explicitly excludes navigation from the 24-task generated dataset.
Go to primary source ↓e17PDF p. 6, Figure 6, numbered Skills list, screenshot labels and caption; p. 5, Section IV-B
The figure's third numbered skill is pick(bowl), but the third screenshot in its left column repeats place(bowl). The verified high-level pipeline is activity prompting followed by task prompting and human implementation. The depicted labels should not be treated as an unambiguous executable program.
Go to primary source ↓8.5 Primary sources
RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robots ↗
PDF · 9,893 extracted words
Source fingerprint
ffad4c8fb66bbeb11ffb89f1a9e999ae73f83e84c126373aebbfe7f5f95630a5