LUMOS: Language-Conditioned Imitation Learning with World Models
1. Paper overview
In one sentence: LUMOS learns language-guided manipulation by practicing against expert latent trajectories inside frozen learned dynamics, gaining longer task chains while inheriting the world model’s errors. e-identitye-overviewe-planse-rewarde-ablatione-real-table
| At a glance | What to know |
|---|---|
| Research problem | Source description Behavior cloning trains on demonstrated states but encounters its own action-induced states at deployment, allowing errors to compound. LUMOS asks whether a policy can practice corrective behavior offline, using learned dynamics instead of further robot interaction, while remaining steerable by sparse language annotations. Its reward is an intrinsic imitation signal; no observed extrinsic task reward is required. e-probleme-overviewe-reward |
| Core mechanism | Source description Combines a DreamerV2-style world model and DITTO latent-matching reward with HULC-inspired latent plans and language–trajectory contrastive alignment in two training stages. e-overviewe-rssme-planse-reward |
| A key reported result | CALVIN D long-horizon instruction following: LUMOS: 2.34 ± 0.05 tasks; 21.1% ± 0.8 percentage points. Average completed chain length; five-instruction success rate (mean ± standard deviation).. Six hours of environment-D play; 34 subtasks; 1% language annotations. Evaluation: 1,000 chains of up to five instructions, three seeds, and 64 × 64 two-camera inputs for every method. HULC: 2.27 ± 0.05 tasks; 20.0% ± 1.5 percentage points. Derived gains are 0.07 tasks and 1.1 percentage points. This is an adapted-resolution, same-environment comparison, without a reported significance test or cross-environment result. e-calvine-baselines |
| Reading caution | Source description The authors identify uncertain world-model quality and the compute cost of increasing image resolution as bottlenecks. Figure 3 illustrates held-out action-conditioned prediction beyond the training horizon, without a quantitative prediction-error metric or contact-accuracy test. e-limitationse-rollout |
Core contributions
- Source description
Combines a DreamerV2-style world model and DITTO latent-matching reward with HULC-inspired latent plans and language–trajectory contrastive alignment in two training stages. e-overviewe-rssme-planse-reward
- Author claim
The authors argue that multi-step practice reduces covariate shift and enables zero-shot real-robot deployment. The experiments support performance within the evaluated settings but do not directly measure distribution shift. e-probleme-ablatione-real-protocole-limitations
Figure 2. The learned simulator, the policy-training machinery, and the deployed controller serve different roles. Original paper, p. 3 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the panels from left to right. The left panel connects camera encodings to stochastic state z and recurrent state h, with actions feeding the recurrence and an image decoder reconstructing observations. The middle panel adds two routes into the plan space: recognition sees the entire training sequence, while proposal sees the current state and encoded goal. The caption specifies that goals come from either a language annotation or the final latent state. The right panel retains proposal and the action decoder, driven by language and fresh camera observations. Compare these arrows with Equations (1), (3), and (4): learning dynamics, assigning imitation reward, and selecting actions are separate operations. e-overviewe-rssme-planse-rewarde-agent-implementatione-collection
What it supports. The mechanism is a modular policy-learning system. A world model learns predictive state representations, and a separate agent learns to act inside them. Full-trajectory hindsight helps train the proposal, but that privileged trajectory is unavailable at deployment. The action decoder operates on inferred current state, goal, and sampled plan.
Where the evidence stops. Figure 2 draws a plan-to-critic arrow, whereas Equation (4) and Appendix B describe critic inputs as state and goal; that discrepancy remains unresolved. The 30 Hz and 1 Hz annotations are also distinct from Appendix E’s 15 Hz training data and do not establish measured deployment latency.
2. Motivation
2.1 The problem and the proposed response
Behavior cloning trains on demonstrated states but encounters its own action-induced states at deployment, allowing errors to compound. LUMOS asks whether a policy can practice corrective behavior offline, using learned dynamics instead of further robot interaction, while remaining steerable by sparse language annotations. Its reward is an intrinsic imitation signal; no observed extrinsic task reward is required. e-probleme-overviewe-reward
2.2 What this reading follows
Imagine learning to handle a sequence of robot instructions without collecting new trials whenever the policy makes a mistake. LUMOS first learns dynamics from offline play, then lets an actor-critic practice inside that learned environment. Hindsight goals make unannotated windows useful, while latent plans and contrastive alignment connect a small amount of language supervision to behavior. The crucial distinction is between learning a policy through imagined consequences and executing that policy with fresh camera feedback. These original visuals explain the distinction, show the component ablations, and expose a remaining gap between imagined success and physical manipulation. The evidence concerns the supplied arXiv v1 and its particular evaluation settings. e-identitye-overviewe-planse-rewarde-ablatione-real-table
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Not assigned |
| Architecture | Not assigned |
| Prediction paradigm | Not assigned |
| Quadrant | Not assigned |
This table preserves the labels recorded at reading time. The current major category is WAMs. View the current classification.
3.1 Evidence-based assessment
Insufficient evidence to decide
The recorded fields are unassigned, leaving no substantive classification to confirm. Architecture evidence shows a separately trained, frozen action-conditioned world model plus a goal-conditioned actor-critic. This supports modular world-model-based control rather than a single joint future-and-action predictor or inverse-dynamics action extractor. Imagination trains the policy; deployment uses latent-state inference and plan proposal. This reader assessment leaves the snapshot unchanged. e-overviewe-rssme-rewarde-agent-implementation
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Turn an offline demonstration into a place to practice
Start with the distinction between posterior and prior state inference. With a real image, the RSSM posterior can incorporate what the cameras just observed; without another image, the prior predicts the stochastic state from recurrent history. Training brings these distributions together while requiring reconstruction of both camera views. Once the world model is fixed, the actor can generate new action-conditioned trajectories through its prior. Equation (3) then compares policy and expert latent states at corresponding steps. This is why the reward can penalize consequences beyond a single action label. As reader analysis, freezing the model also keeps this comparison space from changing in response to the actor’s objective. It does not guarantee that the fixed space represents every contact or object identity accurately. e-rssme-rewarde-objectivese-limitations
Figure 5. The physical training data come from this instrumented tabletop and human teleoperation. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use the colored labels to separate observation hardware from demonstration equipment. Purple marks the camera mounted near the gripper, blue marks the fixed external camera, and red identifies the VR setup. Appendix E names the cameras as FRAMOS Industrial Depth D435e and Azure Kinect, and the teleoperation device as HTC VIVE Pro. The authors collected three hours of play while recording camera images, proprioception, and absolute commands. Their training pipeline differences successive commands to obtain relative actions, resizes recorded 200 × 200 images to 64 × 64, and downsamples the recorded 30 Hz stream to 15 Hz. e-collectione-real-datae-rssme-overviewe-actions
What it supports. The paper’s zero-shot transfer is from offline learned dynamics and policy to physical execution after collecting real data in this setup. It does not mean training without robot data. The two camera viewpoints define the policy’s visual interface, while the VR equipment supports the human play-data collection process.
Where the evidence stops. A setup photograph does not show deployment success or prove which recorded sensor channels enter the policy. Appendix D’s opposing gripper sign conventions remain unresolved, and Figure 2’s rate labels must be reconciled with the downsampled training stream before reproduction.
5.2 Use hindsight to connect sparse language with alternative behaviors
An unannotated play window still has a useful endpoint: its final latent state can serve as a reached goal. An annotated window additionally supplies a language description of the behavior. LUMOS encodes these goal types for a shared goal-conditioned policy. The recognition transformer observes the full window and encodes its behavior as a categorical latent plan. The proposal network learns a compatible distribution from only the current state and goal, the information available when beginning a task. Contrastive alignment encourages the correct latent-trajectory and language pairing relative to unrelated instructions in the batch. Training omits that contrastive term when a window lacks language. At deployment, the proposal supplies the plan and the actor combines it with the current state and language goal; no future demonstration needs to be observed. e-overviewe-planse-objectivese-agent-confige-agent-implementation
5.3 Separate evidence for the components from evidence for the explanation
The full CALVIN model’s 2.34-task average exceeds the adapted HULC baseline’s 2.27, but that margin alone says little about why it works. Removing intrinsic-reward learning, latent plans, or alignment produces larger losses, supporting their usefulness within this implementation. The No DITTO comparison, however, replaces the learning objective as well as multi-step policy practice, so it cannot identify imagination horizon as the sole cause. Physical results add a different test: policies trained offline do execute useful instructions, while the model-space average exceeds the physical average. The authors attribute that gap to dynamics errors. A reader’s causal interpretation should therefore remain conditional: imagined practice can improve behavior when the learned consequences are useful, yet optimistic consequences can also mislead the policy. e-calvine-ablatione-real-protocole-real-tablee-limitations
5.4 Training and inference
During training
Stage one minimizes reconstruction loss plus posterior–prior KL. The approximately 40-million-parameter world model uses 50-step sequences, batch size 50, and a 2048-dimensional latent state. Random RSSM resets with probability 0.01 compensate for scarce episode resets in long play streams. e-rssme-wm-config
Stage two samples windows of 20–32 steps, padded to 32, with batch size 512. The actor maximizes lambda-returns with plan-KL and contrastive terms weighted 0.1 and 3.0. Unannotated windows omit contrastive loss. Discount is 0.995 and lambda 0.95; critic targets are stopped and use a slowly updated network. e-objectivese-agent-confige-agent-implementation
Language goals use paraphrase-MiniLM-L3-v2 embeddings. The recognition transformer has two blocks and eight attention heads. The actor has eight 256-unit fully connected layers; the critic has eight 1024-unit layers. e-agent-confige-agent-implementation
During inference
Camera observations update the latent state. Language and current state condition the proposal; its sampled plan conditions the action decoder. The robot executes the action and supplies another observation. Full-trajectory recognition is a training privilege. The described deployment specifies no online trajectory-search optimizer. e-overviewe-planse-agent-implementation
Relative training actions are derived from successive recorded absolute commands. The physical gripper requires thresholding, but the appendix gives contradictory opening/closing signs, preventing unambiguous implementation from this text alone. e-actionse-collection
5.5 Implementation flow
- Learn an action-conditioned latent simulator
Separate camera CNNs encode the two views. Fused features feed an RSSM with recurrent state h and categorical stochastic state z. The observation-conditioned posterior trains the observation-free prior, while decoders reconstruct both views. Imagined rollouts replace posterior samples with prior samples. e-rssme-wm-config
- Infer goals and reusable plans
A recognition transformer sees the entire training trajectory. A proposal network sees the current state and goal. KL matching trains the proposal to approximate recognized behavior; categorical plans represent alternative ways to reach a goal. Contrastive alignment pairs latent trajectories with instructions using in-batch negatives. e-planse-agent-config
- Optimize behavior through frozen dynamics
The nonrecurrent actor receives current-state features, goal features, and a sampled plan, and outputs a tanh-transformed Gaussian action distribution. The critic learns bootstrapped returns. Actor gradients pass through imagined dynamics, but world-model parameters remain fixed. Actions affect subsequent latent states, which receive the imitation reward. e-rewarde-objectivese-agent-implementation
6. Experiments & results
LUMOS trains a language-guided manipulation policy by practicing inside a world model learned from offline robot play. Frozen recurrent dynamics support actor-critic learning with expert-latent matching rewards; hindsight plans and language alignment organize behavior. CALVIN and physical tabletop experiments support this combination, with modest gains over adapted HULC and larger component-ablation losses. Transfer means deployment after learning from that environment’s offline data, without online policy fine-tuning.
6.1 Read the original evidence
Figure 3. Action-conditioned prediction continues beyond the training sequence length in this held-out example. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow each time label vertically: the upper row is the real camera view, and the lower row is the world model’s reconstruction at the corresponding step. Then scan horizontally to see how object arrangement changes. The caption specifies five conditioning images even though only selected context frames appear in the graphic. After that context, the model receives actions and predicts the next 195 steps without additional image observations. The numbered panels are selected checkpoints, not every predicted frame. Training used sequences of length 50. Only gripper-camera reconstructions are displayed, so the static-camera stream and the full latent representation remain outside this visual comparison. e-rolloute-rssme-real-tablee-limitations
What it supports. The displayed model sequence preserves recognizable coarse changes in the carrot, bowl, and gripper view across a rollout longer than its training window. This is qualitative evidence for sustained action-conditioned prediction. The authors use it to motivate behavior learning in compact latent space, where the policy can explore consequences without rendering images at every training step.
Where the evidence stops. This example does not provide a numerical error curve, contact-fidelity test, or autonomous task-success measurement. Actions are supplied, and the graphic samples the rollout. Visual resemblance alone cannot explain or eliminate the model-versus-robot success gap in Table II.
Table II. Real execution improves on average, while imagined outcomes remain more optimistic overall. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each row as a task category, and keep the three method columns conceptually separate. LUMOS and HULC are evaluated on the physical robot; World Model reports simulated outcomes of LUMOS action trajectories. The surrounding text specifies 20 rollouts per underlying task from neutral starting positions. Vegetable rows combine carrot and eggplant results, but the evaluation still counts choosing the wrong vegetable as failure. The bottom two rows use different units: average task success is a percentage, while sequential tasks is a chain length from 20 five-instruction chains. The dash means no model-space chain result is supplied. e-real-datae-real-protocole-real-tablee-limitations
What it supports. The reported task average improves from HULC’s 63.39% to LUMOS’s 67.68%, and physical chain length rises from 1.90 to 2.05. Yet the World Model average is 75.89%. Cabinet pan lifting makes the mismatch concrete: 60% in the model versus 35% on the robot. Overall gains coexist with task-specific weaknesses.
Where the evidence stops. The table supplies no uncertainty estimates, and aggregation hides separate vegetable-task rates. The model-space success-judging procedure is not detailed. Its 75.89% cannot be presented as robot execution success, nor does this one tabletop setup establish generalization to new environments.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| CALVIN D long-horizon instruction following Six hours of environment-D play; 34 subtasks; 1% language annotations. Evaluation: 1,000 chains of up to five instructions, three seeds, and 64 × 64 two-camera inputs for every method. | LUMOS: 2.34 ± 0.05 tasks; 21.1% ± 0.8 percentage points. Average completed chain length; five-instruction success rate (mean ± standard deviation). | HULC: 2.27 ± 0.05 tasks; 20.0% ± 1.5 percentage points. Derived gains are 0.07 tasks and 1.1 percentage points. This is an adapted-resolution, same-environment comparison, without a reported significance test or cross-environment result. e-calvine-baselines |
| CALVIN D component ablations Table I protocol, three seeds: replace intrinsic-reward training with MSE behavior cloning, remove both plan networks, or omit alignment. | Full: 2.34 ± 0.05; No DITTO: 1.68 ± 0.02; No latent plan: 1.81 ± 0.01; No alignment: 2.05 ± 0.06. Average completed chain length (mean ± standard deviation). | Every variant underperforms the full system. No DITTO changes both the objective and rollout-learning mechanism; it does not isolate imagination horizon alone. e-ablatione-calvin |
| Real-world individual manipulation Franka Panda tabletop; three hours of play, less than 1% language annotation (about 2,800 windows). Each underlying task receives 20 rollouts; vegetable rows aggregate separately tested carrot and eggplant tasks. | LUMOS: 67.68%; uncertainty not reported. Table-reported average success over tasks. | HULC: 63.39%; derived difference 4.29 percentage points. Superiority is not uniform: lifting the pan from the table is 65% versus HULC’s 75%. Neutral starts discourage inferring the task from initial pose. e-real-datae-real-protocole-real-table |
| Real-world five-instruction chains 20 unique chains of five instructions in the physical tabletop environment. | LUMOS: 2.05; uncertainty not reported. Average number of sequential tasks completed. | HULC: 1.90. Executed robot tasks yield a derived gain of 0.15 tasks after offline training; this does not establish transfer to unseen physical environments. e-real-protocole-real-table |
| Real-world versus learned-world task outcomes LUMOS action trajectories are also simulated in the learned world model beside Table II. | World Model column: 75.89%. Table-reported average task success. | Physical LUMOS: 67.68%; derived gap 8.21 percentage points. Model-space success is not another physical result. The authors attribute the gap to dynamics errors; exact model-space success adjudication is unspecified. e-real-tablee-limitations |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table I. The component removals lose more chain length than the full model gains over HULC. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Columns one through five report successful consecutive instructions, so the rightmost success column is the demanding full-chain endpoint, not an independent fifth-task accuracy. The final column reports average completed chain length. Parentheses are standard deviations across three seeds; the retained note establishes 1,000 chains, environment D, and the common two-camera resolution. Compare LUMOS first with HULC, then with the bottom three rows. No DITTO replaces intrinsic-reward training with MSE action supervision. No latent plan removes recognition and proposal. No alignment removes the contrastive language objective. Appendix C further explains how the image-based baselines were adapted to this resolution. e-calvine-ablatione-baselines
What it supports. LUMOS reaches 2.34 tasks versus HULC’s 2.27, while No DITTO, No latent plan, and No alignment reach 1.68, 1.81, and 2.05. Their derived drops from the full model are 0.66, 0.53, and 0.29 tasks. Five-instruction success is 21.1% for LUMOS and 20.0% for HULC.
Where the evidence stops. No significance test accompanies the small HULC margin. No DITTO changes more than horizon alone. Separately, GCBC’s displayed success rates sum to about 0.865 tasks under the cumulative-chain interpretation, inconsistent with its printed 0.63 average; the original cells are preserved without correction.
7. Analysis & limitations
7.1 What the evidence leaves open
The authors identify uncertain world-model quality and the compute cost of increasing image resolution as bottlenecks. Figure 3 illustrates held-out action-conditioned prediction beyond the training horizon, without a quantitative prediction-error metric or contact-accuracy test. e-limitationse-rollout
Appendix D states −1 opens and +1 closes, then gives the opposite continuous threshold rule. Equation (7), with delta = 0.8, weights the posterior-gradient term more strongly despite motivating faster prior updates. Figure 2 feeds a plan to the critic, whereas Equation (4) and Appendix B list only state and goal. These inconsistencies remain unresolved. e-actionse-wm-confige-overviewe-rewarde-agent-implementation
Table I’s GCBC average length of 0.63 does not reconcile with its cumulative success rates under the chain-length interpretation: those rates sum to about 0.865. The original cells are retained, without inferring corrected results. e-calvin
7.2 Questions for discussion
- Does longer imagined practice improve recovery with the world model, actor capacity, and training budget held fixed?
- Which contact or object-identity errors best predict the gap between imagined and physically executed task success?
8. Reproducibility audit
8.1 Requirements and known gaps
Match the two-view pipeline and altered baselines: 64 × 64 inputs, documented convolution changes, baseline-only random shifts, and identical evaluation chains. Real recordings are downsampled from 30 to 15 Hz; Figure 2 labels 30 Hz action execution and a 1 Hz proposal input path without fully resolving deployment timing. e-baselinese-collectione-overview
Architectures and optimizer settings are supplied, but GPU model/count, wall-clock training cost, total optimizer-update budget, and executable software versions are not. Gripper signs, KL balancing, and model-space task judging need clarification. The illustrated edition’s two reproduction checks are proposed, unperformed experiments. e-wm-confige-agent-confige-agent-implementatione-actionse-real-protocole-limitations
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Isolate the contribution of multi-step practice
Reader-proposed, not performed: fix the CALVIN D dataset, frozen world-model checkpoint, actor capacity, goal relabeling, and plan/alignment settings. Compare actor-critic training with imagination horizons of 1, 8, and 32, matching the total number of simulated transition evaluations and reporting wall-clock cost. Include the MSE behavior-cloning variant as a separate objective control. Evaluate identical instruction chains and paired seeds, reporting average chain length and five-instruction success. Add a standardized small end-effector displacement to test recovery directly. If longer practice improves both chain success and perturbation recovery under these controls, the horizon explanation gains support; failure to do so would narrow what Table I establishes. e-ablatione-calvine-rewarde-agent-confige-agent-implementation
Check 2: Measure when imagined success disagrees with physical outcomes
Reader-proposed, not performed: collect paired held-out evaluations for cabinet pan lifting and easier tabletop placement tasks, retaining separate carrot and eggplant identities. Initialize the world model with the same five context images and replay each physically executed action sequence through it. Define task-success judgments identically for the real record and the decoded model sequence; compare both binary outcomes and their disagreement by horizon. Also compare latent-matching reward for successful and failed executions. Document the resolved gripper mapping and action timing before this comparison. A persistent optimistic bias on contact-heavy failures would support the authors’ model-error explanation; poor reward discrimination despite accurate predicted outcomes would instead implicate the imitation signal. e-rolloute-real-protocole-real-tablee-rewarde-actionse-collectione-limitations
8.3 Reading coverage
Visual audit: All nine supplied PDF pages were rendered and visually inspected after every text chunk was read. Page 1 verifies title, authors, affiliations, and v1 date; pages 2–4 support motivation, architecture, equations, and training/inference distinctions; page 5 supports CALVIN, ablations, and real-data scope; page 6 supports prediction examples, physical evaluation, model-space outcomes, and limitations; page 7 contains references; pages 8–9 support hyperparameters, stop-gradient/KL details, network sizes, baseline adaptations, hardware, actions, and preprocessing. Every final original crop was separately viewed: Figure 2, Table I including its standard-deviation/protocol note, Figure 3, Table II, and Figure 5. Figure 2’s plan-to-critic arrow was checked against Equation (4) and Appendix B, and its rate labels against Appendix E; discrepancies are disclosed rather than repaired. Separate supplements, external code, and videos were not inspected, and no experiments were run.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract; I. Introduction (pp. 1–2)
- II. Related Work (p. 2)
- III. Problem Formulation (pp. 2–3)
- IV. LUMOS: A. World Model Learning; B. Behavior Learning (pp. 3–5)
- V. Experimental Results: A. Experiments in Simulation; B. Experiments in Real-World (pp. 5–6)
- VI. Conclusion and Limitations (p. 6)
- References (p. 7)
- VII. Appendix A–E: world model, agent, baseline modifications, action framework, and data collection (pp. 8–9)
Outside the original text pass
- The inspected title page identifies arXiv:2503.10370v1 [cs.RO], 13 March 2025. Its title and all six authors match the catalog. No other revision or venue edition was supplied or compared; the catalog venue annotation is not independently established by this PDF.
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- The extraction caveat was addressed by visually inspecting all nine original PDF pages, including every figure and table. Five original crops were also inspected.
- External code, dataset downloads, project videos, and linked references were not inspected. No experiments were executed or reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e-identityPDF p. 1, title, author line, affiliation footnote, and arXiv margin stamp
Title: LUMOS: Language-Conditioned Imitation Learning with World Models. Authors: Iman Nematollahi; Branton DeMoss; Akshay L Chandra; Nick Hawes; Wolfram Burgard; Ingmar Posner. Affiliations: University of Freiburg; University of Oxford; University of Technology Nuremberg. The artifact is arXiv:2503.10370v1 [cs.RO], 13 March 2025.
Go to primary source ↓e-problemPDF pp. 1–2, Abstract and Section I, Introduction and contributions
The paper motivates offline policy practice by behavior-cloning covariate shift and claims language-conditioned real-robot transfer without online learning. Latent matching comes from DITTO and latent planning builds on earlier imitation-learning methods.
Go to primary source ↓e-overviewPDF p. 3, Figure 2 and caption; Sections III–IV, opening paragraphs
World-model learning precedes policy learning. Hindsight state goals or language annotations condition training. Inference uses observed latent state, language goal, proposal plan, and action decoder. The graphic draws a sampled-plan arrow into the critic. It labels the action loop and plan-sampling current-state path 30 Hz, and the current-state input to plan proposal 1 Hz.
Go to primary source ↓e-rssmPDF p. 3, Section IV-A, Equation (1); p. 4, Equation (2) and following paragraph
Separate static/gripper camera encoders and decoders surround an action-conditioned DreamerV2 RSSM. Recurrent state and categorical posterior/prior components define latent state. Reconstruction and KL train the model; the prior replaces the observation-conditioned posterior for imagination.
Go to primary source ↓e-plansPDF p. 4, Section IV-B, Latent Plan Encoding and Semantic Alignment
Plan recognition sees the entire sequence; proposal conditions on initial state and goal. KL aligns categorical plans. Contrastive cosine alignment pairs language and latent trajectories with in-batch negatives.
Go to primary source ↓e-rewardPDF p. 4, Section IV-B, Reward, Equation (3), Action and value models, Equation (4)
Intrinsic reward is the expert/policy latent-state dot product divided by the squared maximum norm. The actor emits a tanh-transformed Gaussian; actor and critic optimize imitation returns in frozen world-model dynamics. The reward equation has no explicit action-error term.
Go to primary source ↓e-objectivesPDF p. 4, Equation (5) and Learning Objective; p. 5, Equation (6) and opening paragraph
The critic regresses stopped lambda-targets using a slowly updated target network. Actor learning differentiates through dynamics to maximize value and minimize plan KL and contrastive losses. Unannotated windows omit contrastive loss.
Go to primary source ↓e-calvinPDF p. 5, Section V-A and Table I, GCBC/HULC/LUMOS rows
Environment D supplies six hours of play and 34 subtasks, with 1% language annotation. All methods use 64 × 64 two-camera inputs, 1,000 chains and three seeds; parentheses denote SD. LUMOS/HULC average lengths are 2.34 (0.05)/2.27 (0.05), fifth-step rates 21.1% (0.8)/20.0% (1.5). GCBC cells are 56.7%, 20.9%, 7.3%, 1.6%, 0.04%, with average length 0.63 (0.4).
Go to primary source ↓e-ablationPDF p. 5, Section V-A, ablation paragraphs and Table I bottom three rows
No DITTO replaces intrinsic-reward learning with MSE behavior cloning; No latent plan removes proposal and recognition; No alignment omits alignment. Average lengths (SD) are 1.68 (0.02), 1.81 (0.01), and 2.05 (0.06). Fifth-step success is 8.7%, 11.0%, and 17.2%, respectively.
Go to primary source ↓e-real-dataPDF p. 5, Section V-B, opening paragraph
Physical experiments use a Franka Emika Panda and three hours of teleoperated tabletop play. Less than 1% is language annotated, described as approximately 2,800 random windows; the dataset contains over 22 tasks.
Go to primary source ↓e-real-protocolPDF p. 6, Section V-B, paragraph beginning We trained our language-conditioned policy
Each task receives 20 rollouts against HULC from neutral starts. Carrot and eggplant tasks are tested separately, then aggregated as vegetable rows. Picking the wrong vegetable is a failure. Chain evaluation uses 20 unique chains of five instructions.
Go to primary source ↓e-real-tablePDF p. 6, Table II, task rows and final two rows
LUMOS/HULC/World Model average success is 67.68%/63.39%/75.89%. Physical average chain length is 2.05/1.90, with a dash for World Model. Lifting the pan from the table is 65%/75%/70%; lifting it from the cabinet is 35%/35%/60%. No uncertainty is supplied.
Go to primary source ↓e-rolloutPDF p. 6, Figure 3 and caption; Section V-B first paragraph on p. 6
A held-out example uses five context images and predicts 195 steps given actions, despite training horizon 50. Only gripper-camera reconstructions are shown. Real/model frame pairs are qualitative evidence without a numerical prediction-error metric.
Go to primary source ↓e-limitationsPDF p. 6, final Section V-B paragraph and Section VI
The authors simulate LUMOS action trajectories in the learned model and attribute model/real outcome differences to dynamics inaccuracies. They identify difficulty assessing model quality a priori and the compute cost of higher-resolution modeling. Model-space success adjudication is not detailed.
Go to primary source ↓e-wm-configPDF p. 8, Appendix A, Equation (7), and Table III
The approximately 40M-parameter world model has a 1024-dimensional recurrent state plus 32 categorical variables with 32 classes, giving latent size 2048. Batch/sequence lengths: 50/50; beta: 0.3; delta: 0.8; reset probability: 0.01; learning rate: 3e-4. Equation (7) places delta on KL(q || sg(p)), labeled posterior regularizer, while prose motivates faster prior updates. Hardware, runtime and total-update counts are absent.
Go to primary source ↓e-agent-configPDF p. 8, Appendix B, Goal Encoder, Plan Recognition and Proposal, Equation (8), and Table IV
Language uses paraphrase-MiniLM-L3-v2. Recognition has two transformer blocks and eight heads; proposal has four 2048-unit fully connected layers. Batch: 512; windows: 20–32; plan KL scale: 0.1; contrastive scale: 3.0; delta: 0.8; discount: 0.995; lambda: 0.95; actor/critic learning rates: 2e-4/3e-4; slow critic interval: 100.
Go to primary source ↓e-agent-implementationPDF p. 9, Appendix B continuation, Action Decoder and Critic, lambda-target equation and window sampling
Actor inputs concatenate current state, goal, and plan. Its eight 256-unit fully connected layers are nonrecurrent with about 1M parameters. The critic has eight 1024-unit layers and is described as conditioned on current and goal latent states. Lambda-return recursion bootstraps values with lambda 0.95. Sampled windows of 20–32 steps are padded to 32.
Go to primary source ↓e-baselinesPDF p. 5, Section V-A resolution paragraph; p. 9, Appendix C
Baselines use 64 × 64 inputs. Convolution kernel/stride settings change from 8/4 to 4/2, 4/2 to 2/2, and 3/1 to 2/2. Baselines use 0–3-pixel shifts with bilinear interpolation; LUMOS uses no image augmentation. These are adaptations of original higher-resolution baseline settings.
Go to primary source ↓e-actionsPDF p. 9, Appendix D, 7-DoF Action Framework
Actions contain three relative translations, three Euler-angle changes relative to the robot base, and gripperAction. Motion components lie in [−1,1]. The discrete description gives −1 for open and +1 for close; the continuous threshold rule instead gives positive for open and nonpositive for close.
Go to primary source ↓e-collectionPDF p. 9, Appendix E, Data Collection Details; Figures 5–6 and captions
Three hours of Panda play use HTC VIVE Pro teleoperation. Cameras are Azure Kinect (static) and FRAMOS Industrial Depth D435e (gripper), recording 200 × 200 RGB. Proprioception and absolute commands are recorded. Training derives relative commands by differencing, resizes images to 64 × 64 and downsamples 30 Hz to 15 Hz. Figure 5 locates cameras and VR equipment.
Go to primary source ↓8.5 Primary sources
LUMOS: Language-Conditioned Imitation Learning with World Models ↗
PDF · 8,288 extracted words
Source fingerprint
7f857a03bb3d46542ed3d7b2d3a45f45a4aa338873c718fa8a801a35cd497baf