CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Tasks
1. Paper overview
In one sentence: CALVIN turns language-driven robot manipulation into a test of skill chaining and scene transfer, revealing that competent isolated actions can coexist with almost no five-instruction completion. e02e03e07e08e10e14
| At a glance | What to know |
|---|---|
| Research problem | Source description A policy can succeed from familiar demonstration states yet fail when asked to connect skills or recognize the same instruction in a changed scene. CALVIN makes those distinctions measurable through continuous simulated robot control, held-out language formulations and separate seen-scene and unseen-scene evaluations. e02e07 |
| Core mechanism | |
| A key reported result | Five-instruction completion in environment D: 1: 48.9%; 2: 12.9%; 3: 2.6%; 4: 0.5%; 5: 0.08%. Reported success percentage for completing successive instructions in a row.. MCIL with static RGB; train D, test D; LH-MTLC's 1,000 five-task chains, held-out instruction wording and neutral robot initialization. The same sensor/split row reports 53.9% MTLC, whose task distribution and starting-state protocol differ. The baseline seldom sustains a full chain. These are successive-prefix completion rates, not conditional per-step success probabilities; the paper supplies no confidence intervals or seed aggregation details. e07e10e16 |
| Reading caution |
Core contributions
- Source description
Approximately 24 hours of unstructured teleoperation cover roughly six hours per environment and 2.4 million interaction steps. Three untrained collectors explored without downstream task instructions. Only 1% of recorded interaction data receives procedural language annotations. e04e05
Figure 1. Shared interaction elements make held-out-scene transfer measurable. Original paper, p. 1 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the large arrow as a training-to-test split. The three panels inside the orange region are environments A, B and C; the blue region contains D. Compare the desk textures and the locations of the drawer, sliding cabinet, button and switch. Section III-A explains that the robot, desk and static-camera positions remain fixed while interaction-element layouts and textures change. The language boxes illustrate tasks available in these scenes; they are not a temporal action sequence. This image depicts the hardest split, A,B,C to D. The paper also evaluates D to D and training on all four environments before testing on D. e02e03e07
What it supports. The figure establishes a controlled form of scene transfer: common interaction elements are encountered in a different layout and appearance. It makes zero-shot environment evaluation meaningful without requiring an entirely different robot or manipulation vocabulary. Test instructions also use held-out formulations, as specified in the evaluation protocol.
Where the evidence stops. The unseen scene shares its elements with training scenes. This supports a test of transfer across related workspaces; it does not establish open-world object generalization or deployment on a physical robot.
2. Motivation
2.1 The problem and the proposed response
A policy can succeed from familiar demonstration states yet fail when asked to connect skills or recognize the same instruction in a changed scene. CALVIN makes those distinctions measurable through continuous simulated robot control, held-out language formulations and separate seen-scene and unseen-scene evaluations. e02e07
2.2 What this reading follows
Imagine a robot that opens a drawer when placed near its handle, yet struggles when it must first find a block, carry it across the desk and then follow another instruction. CALVIN separates these demands using related simulated workspaces, unstructured play demonstrations and explicit success tests. The central reading distinction is between the benchmark and its MCIL baseline: CALVIN defines the data and challenge, while MCIL learns a latent-plan action policy. Follow the environment split, sensor interface and task definitions before reading the performance table. The resulting scores measure executed simulated manipulation under particular start states and task distributions, not visual prediction quality or physical-robot transfer. e02e03e07e08e10e14
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Benchmarks & simulators |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
3.1 Evidence-based assessment
Supports the recorded classification
The recorded benchmark/simulator classification is supported by the environment, dataset and evaluation contribution. Architecture, prediction paradigm and quadrant remain not applicable to CALVIN as a resource. Its shared MCIL baseline policy and latent plans do not establish a joint future/action world model or an inverse-dynamics prediction paradigm. e02e03e08
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Turn exploratory play into two kinds of supervision
The robot demonstrations were collected as free-form play rather than as expert trials for each named benchmark task. MCIL can exploit this because a visited state supplies its own image goal: the preceding state–action sequence is treated as a demonstration of reaching it. Language needs a different route, since a future state does not automatically specify a sentence. CALVIN therefore detects a completed task in a sampled 64-frame window and attaches a synonymous instruction. Requiring no task completion in the first half retains navigation toward the relevant object before manipulation. Only 1% of interactions are language-labeled, so image-goal training provides much of the control supervision. Keep the data counts separate: the main text says over 400 instructions, while Appendix B reports 389 unique instructions; the supplied paper does not reconcile them. e04e05e06e08e17
Figure 2. The available interface is broader than the baseline configuration. Original paper, p. 3 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with Observation Space: the static RGB camera provides 200 by 200 color images, while the gripper RGB camera provides 84 by 84 images. The proprioceptive block lists robot information available alongside visual sensing. Then read Action Space as a choice of control representation: absolute end-effector pose in the world frame, relative displacement in the gripper frame, or joint positions, each paired with gripper control. These are alternative supported interfaces. Section IV-B fixes the reported MCIL baseline to absolute Cartesian actions and concatenated modality features. Consequently, a result for an added gripper camera evaluates that camera within a particular controller and fusion design. e03e09e13
What it supports. The interface lets researchers vary both perception and action coordinates, but the reported baseline explores only part of that space. This distinction motivates a controlled follow-up: local camera observations could interact with gripper-relative actions, an interaction the authors suggest but do not test in the baseline table.
Where the evidence stops. This is a simulator-interface table, not an MCIL network diagram. Its available sensors and action spaces should not all be attributed to every evaluated row. The tactile-image dimensions are preserved exactly as printed.
5.2 Trace a latent plan into an executed action
The MCIL baseline addresses multiple possible demonstrations through a conditional sequence-to-sequence variational autoencoder. During training, a demonstration is encoded through a latent plan and a policy decoder reconstructs its actions using the state and goal. Image goals and language goals provide different contexts for the shared policy. At test time, the model receives a goal, infers a plan and follows it while observing the environment in closed loop. The selected sensor encoders contribute concatenated features, and the reported controller emits absolute Cartesian actions. The KL term regularizes training; it is not an inference-time task evaluator. Likewise, the word plan does not imply a predicted future video or a simulator rollout search: this paper describes a latent representation used by an imitation policy. CALVIN supplies the environment in which the resulting actions execute. e08e09e03
Figure 9. The evaluator checks state changes with task-specific preconditions. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each condition against both endpoints of a sequence. A successful lift from the table requires the block to rise at least 5 cm and forbids initial gripper contact; placement in the drawer requires an object already lifted by the gripper at the first frame. Thus, the same final image cannot by itself establish every task. For doors, the table gives at least 12 cm of sliding motion and at least 10 cm of drawer travel. Appendix A explains that the detector compares restored first and final simulator states. During chain evaluation, the detector decides when the controller may present the next language instruction. e14e15e07e06
What it supports. CALVIN's success measure includes operational preconditions, not just resemblance to a desired final scene. These conditions support automatic play labeling and success-gated evaluation. They also explain why chain construction must respect feasibility: placing an object starts with a different required state from lifting it from a surface.
Where the evidence stops. Figure 6 describes light-bulb on/off as switch up/down; this appendix instead specifies down for on and up for off. Its grouped drawer row also lists push-in/pull-out for open/close, whereas the appendix gives explicit task-specific directions. Preserve the graphic and verify detector behavior before reproduction.
5.3 Interpret failure across successive instructions
A chain trial begins with a feasible sequence and a neutral robot pose. The controller provides the current instruction, then advances only when the environment's state detector declares success. This tests whether skills remain useful as earlier actions change the scene and robot configuration. It does not require the agent to invent its own high-level decomposition. Reader analysis: the decline from 48.9% first-prefix success to 0.08% five-prefix success establishes a severe composition problem for the baseline, but does not isolate a single cause. The authors report pose sensitivity and color confusion; the task-distribution plot also shows unequal skill frequencies. A useful interpretation therefore keeps language grounding, state coverage and subgoal transitions as competing explanations. The 53.9% isolated-task score uses a different initialization protocol and cannot be substituted as a universal per-step success probability. e07e10e12e13e14
5.4 Training and inference
During training
The baseline uses Adam at learning rate 0.0001, KL weight 0.001 and randomly sampled 16–32-step windows padded to 32. It applies no image augmentation and uses the same training hyperparameters across splits. These training windows are distinct from the 64-frame language-labeling windows. e09e06
Modality features are concatenated; depth is fused with RGB by channel concatenation. The gripper encoder has convolution widths 32/64/64 and a 128-unit ReLU MLP; tactile encoding uses a pretrained ResNet-18. Supplied MiniLM sentence embeddings have 384 dimensions. The paper does not specify every pretrained module's freezing policy. e09e05e16
During inference
At test time the goal conditions an inferred latent plan, which the policy follows with fresh state observations in closed loop. Low-level actions execute in the simulator. The evaluation controller supplies subgoals sequentially using the state detector; successful chains therefore do not establish autonomous language-task decomposition. e08e03e07
5.5 Implementation flow
- Construct a controlled scene shift
Environments A–D change textures and the positions of static interaction elements while retaining the desk, robot and static-camera positions. The layouts share scene elements; zero-shot testing concerns a held-out arrangement and appearance, not wholly new object categories. e03e07
- Recover supervision from play
For goal-image imitation, a visited state becomes the reached goal and preceding actions become its demonstration. Language annotation instead samples 64-frame windows, requires a detected task over the full window and excludes any task completion in its first half, retaining approach motion before manipulation. e08e06
- Condition a shared action policy
MCIL uses a sequence-to-sequence conditional variational autoencoder to encode demonstrations into latent plans and reconstruct their actions, conditioned on state and goal. Image-goal and language-goal contexts train a shared policy with context encoders. This is action reconstruction and control, with no explicit future-state rollout model described. e08
- Evaluate isolated tasks and chains separately
MTLC uses ten rollouts per task from valid unseen demonstration starting states. LH-MTLC uses 1,000 feasible five-task chains, filtered for cycles, redundancy and similarity, with neutral robot initialization between sequences. Both use unseen instruction phrasings. The next instruction is released only after current-task success. e07
6. Experiments & results
CALVIN tests whether a language-conditioned robot policy can compose manipulation skills across five successive instructions and transfer across related simulated scenes. It supplies play data, sparse language labels, multimodal interfaces and state-based success checks. Its MCIL baseline learns actions from demonstrations through latent plans; it does not demonstrate learned future-world prediction. Static-camera MCIL reaches 53.9% short-horizon success in environment D but only 0.08% five-instruction completion under a different, neutral-start evaluation protocol (e02, e07, e08, e10).
The supplied paper has no MCIL network architecture drawing and no controlled ablation of the latent-plan mechanism. The edition therefore uses the original environment split and interface table to explain the benchmark, the original sensor-comparison results, and the original evaluation-distribution chart as a diagnostic. Figure 7 is explicitly a task-mixture diagnostic rather than an experimentally ablated model component. e02e03e08e09e10e11e12
6.1 Read the original evidence
Figure 8. Short-horizon competence leaves a severe gap in skill chaining. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. First select a sensor group using the green checks and red crosses. Within each group, the three rows change the train-to-test environment split. Next separate MTLC, evaluated over 34 tasks from demonstration starting states, from LH-MTLC, evaluated on 1,000 chains with neutral initialization. The columns labeled 1 through 5 are successive-prefix completion rates. In the first row, static-RGB MCIL trained and tested in D reaches 53.9% MTLC, then 48.9%, 12.9%, 2.6%, 0.5% and 0.08% across chain prefixes. Compare sensors only within a fixed split, and remember that adding a modality also adds its encoder and the baseline's fusion pathway. e07e09e10e11e12e16
What it supports. The strongest lesson is the collapse in sustained completion, not a uniformly superior sensor suite. Static RGB plus tactile reaches 54.2% MTLC in D, yet only 28.5% on the first chain instruction and 0% on all five. Its isolated-task advantage over static RGB therefore does not carry over to the chain metric.
Where the evidence stops. MTLC and LH-MTLC use different starts and task distributions. No confidence intervals or seed aggregation are provided. Keep 0.08% as a reported value; the stated 1,000 chains do not justify converting it into an integer success count.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Five-instruction completion in environment D MCIL with static RGB; train D, test D; LH-MTLC's 1,000 five-task chains, held-out instruction wording and neutral robot initialization. | 1: 48.9%; 2: 12.9%; 3: 2.6%; 4: 0.5%; 5: 0.08%. Reported success percentage for completing successive instructions in a row. | The same sensor/split row reports 53.9% MTLC, whose task distribution and starting-state protocol differ. The baseline seldom sustains a full chain. These are successive-prefix completion rates, not conditional per-step success probabilities; the paper supplies no confidence intervals or seed aggregation details. e07e10e16 |
| Zero-shot language control in unseen environment D Static-RGB MCIL trained on A,B,C and tested on D; compare with A,B,C,D training and D-only training. | A,B,C→D: MTLC 38.6%; one instruction 20.2%; two 0.2%; five 0%. MTLC success and LH-MTLC prefix completion percentages. | A,B,C,D→D: MTLC 35.6%, one instruction 28.2%, five 0%; D→D: MTLC 53.9%. Seen-environment training does not uniformly outrank held-out-scene training on every metric. Changing the training environments also changes available data; this does not isolate appearance shift alone. e04e07e10 |
| Sensor-combination comparison in environment D MCIL D→D, retaining absolute Cartesian actions and the reported baseline fusion strategy. | Static RGB plus tactile: MTLC 54.2%; first chain instruction 28.5%; five instructions 0%. MTLC and LH-MTLC success percentages. | Static RGB alone: 53.9%, 48.9%, 0.08%, respectively. Static plus gripper RGB: MTLC 51.8%; both RGB-D cameras: MTLC 46.1%. The highest listed single-task score does not identify the best chain behavior. Sensor additions offer no uniform improvement in this implementation; statistical significance and optimal fusion are not established. e09e11e16 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 7. The chain benchmark weights manipulation skills unequally. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use the vertical labels to identify individual tasks and the legend to group them into manipulation families. Long bars near the top correspond to placing in the slider and opening the drawer; short bars at the bottom correspond to lifting particular blocks from the drawer. The horizontal axis is labeled Probability, but the caption explicitly defines the plotted quantities as percentages of subtask appearances. Interpret the scale accordingly rather than as probabilities bounded by one. This is the distribution of the filtered evaluation chains described in Section III-C. It is neither the frequency of tasks in the collected play data nor the policy's success on each task. e12e07e04
What it supports. The diagnostic reveals that aggregate chain performance depends on a particular mixture of subtasks. A policy's overall score can be influenced more by common placements and door operations than by rare drawer lifts. Reader analysis: per-task outcomes would help distinguish broad skill improvement from gains concentrated on frequent evaluation tasks.
Where the evidence stops. This diagnostic is not a model ablation and does not report per-task accuracy. Feasibility filtering shapes the task mixture; equalizing frequencies could change the evaluated challenge and should be labeled as a separate analysis.
7. Analysis & limitations
7.1 What the evidence leaves open
The four scenes use simple unicolored shapes and shared physical structure. Experiments concern simulation; generalization to realistic objects, physical robots or arbitrary new skills remains untested. e03e07e10
The authors observe sensitivity to initial robot position and confusion between red and blue blocks. Their explanations involving causal confusion, language embeddings, augmentation and improved fusion are hypotheses rather than controlled causal demonstrations. e13
Source discrepancies remain unresolved: Section III-B reports over 400 language instructions, while Appendix B gives 389 unique instructions. Figure 6 associates light-bulb on/off with switch up/down; Figure 9 instead specifies down for on and up for off. Figure 8 also lists 54.2% MTLC despite the introduction highlighting up to 53.9%. e17e15e02e11
Figure 7 shows an unequal distribution of subtasks in instruction chains. Aggregate chain success therefore cannot stand in for uniform competence over all 34 tasks. Reported 0.08% values are retained without inferring integer trial counts from the stated 1,000 chains. e12e10e16
7.2 Questions for discussion
- How much of the chain failure comes from language grounding, neutral initial states and transitions between otherwise solvable subtasks? (e07, e13)
- Would gripper-relative control make local camera information more useful under the same data and fusion budget? (e03, e09, e13)
8. Reproducibility audit
8.1 Requirements and known gaps
A faithful baseline requires the documented environment split, sensor combination, absolute-action convention, language split, annotation filter, initial-state protocol and task detectors. Figure 9 gives concrete thresholds, including at least 12 cm slider motion and at least 10 cm drawer travel. e03e06e07e09e14
The supplied implementation section defers architecture details to prior MCIL work. It does not provide a complete training budget, batch size, seed count, evaluation timeout, hardware specification or precise simulator/software versions. Figure 8 omits uncertainty and aggregation details. The PDF describes the benchmark as open-source but does not establish a dataset license or a verified executable release here. e02e09e16
Proposed checks: compare a matched sensor-by-action-frame experiment to test the authors' gripper-camera hypothesis, and measure correct-versus-color-swapped instructions from matched initial states to test language dependence. Record per-task and chain-prefix outcomes; these are proposed experiments, not reproduced results. e03e07e09e13
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Does the gripper camera help specifically with relative actions?
Reader-proposed experiment, not performed: train a two-by-two comparison of static RGB versus static-plus-gripper RGB, crossed with absolute-world versus gripper-relative Cartesian actions. Use identical D training episodes, language labels, training budgets and paired evaluation starts; convert action targets so they describe the same demonstrated end-effector motions. Keep the documented encoder/fusion choices where applicable and record every other change. Evaluate grasping/stacking tasks as well as all chain prefixes, reporting variation across repeated runs. A larger camera benefit with relative control would support the authors' proposed interaction. Similar gains from relative actions with either sensor suite would weaken a camera-specific explanation. e03e07e09e13e14
Check 2: Separate color grounding from initial-pose cues
Reader-proposed experiment, not performed: choose scenes where analogous red-block and blue-block tasks are both feasible. For each fixed scene, evaluate paired held-out instructions that differ only in color, using both neutral and demonstration-derived robot starts. Keep the trained policy and scene objects fixed, and use the corresponding state detectors to record intended-object success and wrong-color manipulation. Include shuffled color instructions as a control and report results by initial-pose condition. If selected objects track the requested color across poses, that supports language grounding. If actions mostly follow the starting pose or remain unchanged when the color word changes, the result would support the authors' shortcut/confusion concern. e07e13e14e05
8.3 Reading coverage
Visual audit: All ten supplied PDF pages were rendered and visually inspected, including the title/author/version block, Figures 1–9, Equation (1), implementation details, evaluation protocols and both appendices. All five final original-PDF crops were separately viewed; complete table borders, headers, task conditions, chart axes and legend are retained. Figure 1's train-to-test arrow was checked against Section III-C, and Figure 2's action frames against Section IV-B. Figure 9's thresholds were checked against Appendix A; the light-switch direction conflict with Figure 6 is disclosed. Figure 7's Probability label is interpreted using its percentage caption. Figure 8's sensor marks and numerical columns were checked directly. Page 8 contains the bibliography, whose cited works were not separately inspected. No external supplements, code or dataset files were part of this pass.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- PDF p. 1: title, authors, affiliations, version and abstract
- PDF pp. 1–2: I. Introduction; II. Related Work
- PDF pp. 2–5: III. CALVIN; A. Environment; B. Dataset; C. Challenge
- PDF p. 6: IV. Baseline Models; A. Multicontext Imitation Learning; B. Implementation Details
- PDF pp. 6–7: V. Experimental Results; VI. Conclusion; Acknowledgement
- PDF p. 8: References
- PDF p. 9: Appendix A. Tasks
- PDF p. 10: Appendix B. Language Annotation Generation
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout. This extraction limitation was addressed by visually inspecting all ten PDF pages and all five final crops.
- Separate supplemental material availability has not been fully verified.
- Identity/version note: the inspected artifact is arXiv:2112.03227v4 [cs.RO], dated 13 July 2022, carrying an IEEE Robotics and Automation Letters preprint heading and an acceptance date of 22 May 2022. Its title matches the catalog. The title page spells the second author Lukas Hermann; the catalog spells the name Lukás Hermann. The four-author identity otherwise agrees, and Mees and Hermann are marked equal contributors. The publisher edition and earlier arXiv revisions were not supplied, so differences from them have not been established.
- No code, linked website, external references, separate dataset files or supplements were inspected; no experiments were run. Reading the bibliography does not constitute reading its cited works.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title/author block, affiliation footnote and arXiv margin stamp
The exact CALVIN title and authors Oier Mees, Lukas Hermann, Erick Rosete-Beas and Wolfram Burgard appear on arXiv:2112.03227v4, 13 July 2022. Mees and Hermann are equal contributors. Affiliations are University of Freiburg, Germany, and University of Technology Nuremberg, Germany.
Go to primary source ↓e02PDF pp. 1–2, Abstract, Figure 1 and Section I; p. 2, Section II
CALVIN is presented as an open-source simulation benchmark combining language, multimodal sensing, continuous robot control and long-horizon tasks; Figure 1 shows A/B/C training and D testing. Section I mentions 24 hours of play, 20K directives and up to 53.9% short-horizon baseline success.
Go to primary source ↓e03PDF p. 3, Section III-A.1–2 and Figures 2–3; p. 4, Section III-C footnote 1
Four PyBullet scenes share robot/desk/camera placement but vary textures and static interaction-element locations. Figure 2 specifies sensor shapes and absolute, relative and joint-action interfaces. Control runs at 30 Hz; object simulator states are provided but not used for baseline perception.
Go to primary source ↓e04PDF pp. 3–4, Section III-B.1, Unstructured Demonstrations
The dataset contains 24 hours of play, approximately six per environment, roughly 2.4M interaction steps and 40M short goal-relabeling windows of 1–2 seconds. Three untrained collectors used HTC Vive teleoperation without downstream-task guidance; task occurrence is uneven.
Go to primary source ↓e05PDF p. 4, Section III-B.2, Language Instructions, and Figure 4
The main text reports over 400 crowd-sourced instructions, procedural episode labeling, language annotations for 1% of interactions, and supplied MiniLM sentence embeddings of dimension 384.
Go to primary source ↓e06PDF p. 10, Appendix B, Language Annotation Generation
Annotations use randomly sampled 64-frame windows with task completion across the full window and no task completed in the first half. A synonym is sampled for qualifying windows. The appendix states 389 unique instructions for 34 tasks, approximately 11 per task, and 1% labeled interactions.
Go to primary source ↓e07PDF pp. 4–5, Section III-C.1–2, Training and Test Environments and Evaluation Metrics; p. 5, Figure 5
Training modes are same environment, all four environments, or three with the fourth held out. MTLC runs ten rollouts per task from unseen demonstration states. LH-MTLC uses 1,000 feasible five-task chains, neutral initialization between sequences and success-gated instruction transitions. Evaluation wording is held out; the unseen scene shares elements with training.
Go to primary source ↓e08PDF p. 6, Section IV-A, Multicontext Imitation Learning, Equation (1) and following paragraphs
Reached states relabel play as goal-conditioned demonstrations for maximum log-likelihood imitation. A seq2seq CVAE encodes latent plans and reconstructs actions conditioned on state and goal; test-time plans guide closed-loop actions. MCIL shares a policy over contextual datasets with their encoders; no learned future-state rollout objective is described.
Go to primary source ↓e09PDF p. 6, Section IV-B, Implementation Details
Adam learning rate is 10^-4 and KL weight is 0.001; windows of 16–32 are padded to 32. The baseline uses absolute world-frame Cartesian actions without image augmentation. It concatenates modality features, fuses RGB/depth channels, uses a 32/64/64-convolution and 128-unit MLP gripper encoder and a pretrained ResNet-18 tactile encoder.
Go to primary source ↓e10PDF p. 7, Figure 8, first three rows (static RGB only), MTLC and LH-MTLC columns; Section V
D→D gives 53.9% MTLC and 48.9/12.9/2.6/0.5/0.08% chain prefixes. A,B,C,D→D gives 35.6% MTLC and 28.2/2.5/0.3/0/0%; A,B,C→D gives 38.6% and 20.2/0.2/0/0/0%. Section V discusses poor long-horizon performance.
Go to primary source ↓e11PDF p. 7, Figure 8, D→D rows across the four sensor groups
Static RGB plus gripper RGB gives 51.8% MTLC and 0.08% five-chain completion; static plus tactile gives 54.2% MTLC, 28.5% first-prefix and 0% five-prefix success; static/gripper RGB-D gives 46.1% MTLC. All rows show complete sensor inclusion/exclusion marks.
Go to primary source ↓e12PDF p. 6, Figure 7 and caption
The chart shows the unequal subtask distribution across 1,000 LH-MTLC instruction chains. Place-in-slider and open-drawer are frequent; drawer lifts are rare. Its axis says Probability, while the caption specifies percentages. It is an evaluation distribution, not a training histogram or per-task success plot.
Go to primary source ↓e13PDF pp. 6–7, Section V, discussion before and after Figure 8
The authors suggest gripper-relative actions may help the gripper camera, report sensitivity to initial robot pose and red/blue confusion, and propose augmentation, language alignment and improved fusion. These mechanisms are not separately validated in the reported experiments.
Go to primary source ↓e14PDF p. 9, Appendix A and Figure 9; p. 3, Section III-A.2
Task completion compares first and final simulator states. Figure 9 defines 34 tasks, including rotation, contact, displacement, lift and gripper-contact conditions. Slider motion requires at least 12 cm; drawer travel at least 10 cm. The state detector can support sparse rewards.
Go to primary source ↓e15PDF p. 5, Figure 6, Turn on/off light bulb and Open/close drawer rows; p. 9, Figure 9, individual light-bulb and drawer rows
The main figure describes switch up/down for light on/off, but the appendix explicitly gives down for on and up for off. The supplied paper does not resolve this directional inconsistency. The main figure's grouped open/close drawer row lists push-in/pull-out, while the appendix explicitly pairs opening with pulling out and closing with pushing in.
Go to primary source ↓e16PDF p. 6, Section IV-B; p. 7, Figure 8 and Section V; pp. 9–10, Appendices A–B
The supplied sections provide selected hyperparameters and annotation/task rules but no complete training budget, batch size, seed count, evaluation timeout, hardware, precise software versions or module-freezing policy. Figure 8 reports percentages without error bars or an aggregation explanation; values such as 0.08% cannot be converted into simple integer counts over 1,000 chains without further information.
Go to primary source ↓e17PDF p. 4, Section III-B.2; p. 10, Appendix B
The main text's over-400 instruction count and the appendix's 389 unique instructions are both reported; the supplied source does not explain the difference.
Go to primary source ↓8.5 Primary sources
CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Tasks ↗
PDF · 8,099 extracted words
Source fingerprint
c8ad4741782fa8fb773377fbf61b8772b86574f255599a229d3f6ad4d878c60e