LUMOS: Language-Conditioned Imitation Learning with World Models
Classification
View four quadrants- Major category
- WAMs
- Quadrant
- Outside quadrants
- Architecture
- Pending verification
- Prediction paradigm
- Other mechanisms
- Subcategories
- Policy post-training & WM-RL
- Source review status
- Not assigned
Category review. A frozen action-conditioned RSSM supports imagined actor-critic optimization; language-conditioned plans and latent-state inference produce robot behavior during deployment. The described action mechanism is external planning, model-assisted policy optimization, geometric tracking, or video-conditioned control; the source does not establish joint future/action generation or an IDM action decoder. Reading evidence
Contribution
LUMOS trains a language-guided manipulation policy by practicing inside a world model learned from offline robot play. Frozen recurrent dynamics support actor-critic learning with expert-latent matching rewards; hindsight plans and language alignment organize behavior. CALVIN and physical tabletop experiments support this combination, with modest gains over adapted HULC and larger component-ablation losses. Transfer means deployment after learning from that environment’s offline data, without online policy fine-tuning.
Abstract
An abstract has not been added yet.
Affiliations
University of Freiburg; University of Oxford; University of Technology Nuremberg