MoWM: Mixture-of-World-Models for Embodied Planning via Latent-to-Pixel Feature Modulation
1. Paper overview
In one sentence: MoWM adds predicted latent dynamics to detailed video-model features before decoding actions, improving CALVIN chain completion while leaving the cause of that improvement only partly isolated. e-identitye-motivatione-architecturee-maine-ablatione-real
| At a glance | What to know |
|---|---|
| Research problem | Author claim Language-conditioned manipulation needs both accurate spatial detail and useful future-motion information. The authors argue that video reconstruction preserves distracting appearance, while compressed latent prediction can discard manipulation-critical detail. MoWM tests whether combining these representations improves executable action prediction. e-motivation |
| Core mechanism | Source description A two-stage architecture independently trains instruction-conditioned pixel and latent world models, then freezes them while training feature fusion and an inverse-dynamics action decoder. e-architecturee-pixele-latent |
| A key reported result | CALVIN ABC→D five-task instruction chains: MoWM: 0.943 / 0.873 / 0.812 / 0.750 / 0.675; average length 4.10. Stage 1–5 success rates (fractions); reported average completed task length. Train exclusively on language-annotated data from scenes A/B/C; evaluate unseen scene D. Table 1 compares fine-tuned methods; VPP uses a single-view implementation. Pi0: 0.938 / 0.850 / 0.767 / 0.681 / 0.599; 3.92. VPP: 0.909 / 0.815 / 0.713 / 0.620 / 0.518; 3.58. Best values among the listed methods. The fifth-stage difference from Pi0 is 7.6 percentage points, approximately 12.7% relative (reader arithmetic), consistent with the long-horizon claim. No uncertainty is reported; average-length inconsistencies are preserved below. e-setupe-maine-gains |
| Reading caution | Reader analysis Table 1 prints average lengths 4.10 and 3.92 for MoWM and Pi0, although their displayed stage rates sum to 4.053 and 3.835 (reader arithmetic). Those differ beyond ordinary rounding if rates represent cumulative chain completion. Preserve both columns; the aggregation discrepancy is unresolved. e-main |
Core contributions
- Source description
A two-stage architecture independently trains instruction-conditioned pixel and latent world models, then freezes them while training feature fusion and an inverse-dynamics action decoder. e-architecturee-pixele-latent
- Reader analysis
Concatenation, projection and a learned residual connection implement the proposed feature modulation. CALVIN comparisons and a fusion ablation support this particular combination; motion-focused explanations remain partly interpretive. e-fusione-maine-ablatione-diagnostic
Figure 1. Two predictive branches supply one action decoder through learned feature fusion. Original paper, p. 3 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start at the image and instruction on the left, then follow the green pixel branch and yellow latent branch into the central modulation block. Both branches contribute features; the action decoder on the right also receives noisy actions. Read the training labels as two stages: the encoder snowflakes accompany trainable world-model blocks in stage one, while the caption explicitly freezes the world models during stage two. The diagram compresses the internal fusion operation. Equations (4)–(6) expand it into dimension alignment, concatenation, projection and a residual connection from the pixel features. e-architecturee-featurese-fusion
What it supports. The architecture supports a modular inverse-dynamics interpretation: future-state predictors supply information to a separate learned action generator. The second-stage training can optimize how the predictions are used without retraining the world models. The arrows show feature conditioning, not a search that ranks action candidates by predicted rewards.
Where the evidence stops. The figure omits the pixel residual. Decoder prose calls its input fused features, whereas Eq. (7) uses the final residual-enhanced features. This edition follows Eqs. (6)–(7); the implementation discrepancy remains unresolved.
2. Motivation
2.1 The problem and the proposed response
Language-conditioned manipulation needs both accurate spatial detail and useful future-motion information. The authors argue that video reconstruction preserves distracting appearance, while compressed latent prediction can discard manipulation-critical detail. MoWM tests whether combining these representations improves executable action prediction. e-motivation
2.2 What this reading follows
A robot folding cloth or moving a block needs to anticipate motion while retaining enough visual detail to act precisely. MoWM approaches this through two predictive representations. A video diffusion model supplies detailed intermediate features; a transformer forecasts features from a V-JEPA 2 encoder. The method aligns and combines those streams, then uses them to condition action denoising. This reading follows the supplied v3 paper from that information flow to the CALVIN comparisons, the fusion ablation and the physical folding example. The central distinction is between measured action success, reconstructed future imagery and hypotheses about which representation carries useful dynamics. e-identitye-motivatione-architecturee-maine-ablatione-real
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | WAMs |
| Architecture | Dual-system |
| Prediction paradigm | IDM |
| Quadrant | Q4 · Dual-system × IDM |
3.1 Evidence-based assessment
Supports the recorded classification
Architecture evidence supports the recorded Dual-system × IDM classification: independently trained predictive world models feed a distinct action decoder, and Section 2.2 explicitly chooses inverse dynamics. End-to-end action-module training does not make this one jointly predictive model. V-JEPA 2 provides the latent encoder; MoWM's predictor is trained with supervised next-feature L1 loss. e-architecturee-latente-fusion
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Train the future representations before training the controller
MoWM first learns two different prediction problems from aligned embodied sequences. The SVD branch is adapted with instruction conditioning and learns to predict diffusion noise for future video. The latent branch starts with V-JEPA 2 ViT-g frame representations and learns a teacher-forced next-state predictor under L1 loss. Matching the temporal sampling makes the two streams refer to the same physical progression. Figure 1 then freezes the world models for the action-training stage. This sequencing matters because action supervision can learn how to combine the predictions without changing the predictors themselves. The latent RGB decoder described in the experiments is a separate visualization tool; producing those reconstructed images is not specified as an intermediate requirement for action generation. e-pixele-latente-architecturee-visualization
Figure 3. The two feature spaces produce visibly different reconstructions of instruction-conditioned futures. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Each example starts with the initial observation at the left. Read time from left to right within each row, then compare corresponding positions vertically: ground truth above, latent-feature visualization in the middle and pixel-feature visualization below. The examples concern placing an object in the slider and turning the pink block. The middle rows are not native RGB outputs of the latent predictor. The authors train a separate convolutional decoder to make those states viewable. Consequently, appearance differences must be read through that extra decoder as well as through the underlying representation. e-visualizatione-diagnostice-features
What it supports. The examples make the paper's proposed complementarity tangible: latent reconstructions show evolving scene structure despite appearance changes, while pixel predictions retain detailed imagery. They motivate combining representations. They do not measure whether every predicted contact is correct or establish that the action policy uses the visible motion cues.
Where the evidence stops. The visualization decoder is trained with image MSE and is used for analysis. Its reconstruction errors can influence apparent color or motion quality. These selected examples contain no quantitative prediction metric or action-success comparison.
5.2 Understand modulation as a feature path with a residual
At action inference, the pixel model supplies intermediate U-Net features from a single denoising step. These are resized and concatenated across scales; a second predicted sequence comes from the latent model. Learned projections align their embedding dimensions before concatenation and another projection produce fused features. Equation (6) then adds a gated fused term to the projected pixel representation. A reader's interpretation is that the residual leaves a direct path for fine spatial information while allowing the latent stream to alter the action condition. The displayed equations do not explicitly select moving pixels or impose a motion mask. That distinction matters when interpreting the paper's language about suppressing irrelevant content. Equation (7) conditions action denoising on the final representation, despite nearby prose naming the earlier fused one. e-featurese-fusion
5.3 Separate action improvement from an explanation of that improvement
CALVIN provides the strongest quantitative evidence: the fused model leads the listed baselines and its own pixel-only variant on the reported chain metrics. The ablation is closer to the architectural question than comparisons across unrelated model families, but it still lacks a latent-only model and a capacity-matched control. Figures 3 and 5 offer a plausible explanation through temporal information, yet a decoder stands between latent features and the images we see. A reader therefore cannot infer policy dependence on motion simply from attractive reconstructed trajectories. The cloth-folding photographs answer a different question by showing a physical execution. They do not supply repeated-trial reliability. A useful reproduction would connect these evidence types by manipulating the latent temporal information and measuring the resulting executed-action success. e-maine-ablatione-visualizatione-diagnostice-real
5.4 Training and inference
During training
Stage one uses the same embodied data and aligned temporal sampling for both world models. The pixel model minimizes squared noise-prediction error; the latent model uses teacher-forced L1 next-state prediction. Figure 1 marks the encoders as frozen and its caption explicitly freezes both world models for stage two, whose action denoising loss trains the planning module. e-pixele-latente-architecturee-fusion
The latent predictor has 24 layers and about 400M parameters. Training uses four H20 GPUs for about seven hours: 75 epochs of 300 steps, batch four per GPU (16 global), AdamW, cosine scheduling and weight decay 0.04. Stage two uses four H20 GPUs for 13 hours, 7,000 steps, batch size 28, AdamW, learning rate 1e-4 and weight decay 0.05. Whether 28 is global is unstated. e-training
During inference
Section 3.2 describes one forward pass per world model to obtain future features, followed by multiple action denoising steps. Its latent training formulation predicts one next state from history; the implementation of a multi-step latent forecast is not resolved. No complete observation-refresh or chunk-execution schedule is specified. e-latente-featurese-fusione-real
The AgileX setup reduces control frequency from 50 to 30 Hz and predicts 30-timestep, one-second chunks from joint angles, images and text. The paper reports 110 ms average inference latency 'for 50 steps' without clearly defining those steps or the timing boundary. e-real
5.5 Implementation flow
- Predict in two representation spaces
The pixel branch adapts Stable Video Diffusion with text cross-attention. The latent branch encodes frames using V-JEPA 2 ViT-g, concatenates text tokens with current/past state tokens, and predicts the next state using transformer blocks with SwiGLU feed-forward layers. e-pixele-latent
- Align intermediate features
For action prediction, the pixel branch uses one denoising step. Features from multiple U-Net upsampling layers are bilinearly resized to a common spatial grid and concatenated. The latent branch supplies predicted state features. Separate learned projections map both streams to the same embedding dimension. e-features
- Fuse and decode
Concatenate the aligned latent and pixel features, project them, and add a gated fused term to the projected pixel feature. A Diffusion Policy action decoder performs multi-step action denoising conditioned on the resulting representation. This is a learned future-state-to-action mapping, without a described reward-based search over candidate actions. e-fusione-architecture
6. Experiments & results
MoWM conditions an action diffusion decoder on predictions from two frozen world models: an SVD-based video model and a transformer predicting V-JEPA 2 features. Learned concatenation and a pixel-feature residual combine their representations. CALVIN results favor this fusion; decoded prediction images and a real-robot folding example provide narrower qualitative evidence. The report separates those findings from claims about why fusion works.
6.1 Read the original evidence
Table 1. MoWM leads the listed methods on the reported CALVIN chain metrics. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the five numbered columns as success rates at successive stages of a five-task evaluation chain, and keep the separate average-length column as printed. The setup trains on language-annotated scenes A, B and C and evaluates unseen scene D. Compare MoWM with Pi0 for the strongest listed VLA baseline, then with the single-view VPP implementation for a video-world-model baseline. The gap to Pi0 is small in the first column and larger in the fifth. The method categories describe different model families; the table does not establish equal pretraining data or compute budgets. e-setupe-maine-gains
What it supports. MoWM reports fifth-stage success of 0.675 versus Pi0's 0.599 and VPP's 0.518. The Pi0 comparison is a reader-calculated 7.6-percentage-point gain, or about 12.7% relative. The printed average lengths are 4.10, 3.92 and 3.58 respectively. These are simulator action outcomes, distinct from judging generated images.
Where the evidence stops. MoWM's five displayed rates sum to 4.053, not the printed 4.10; Pi0's sum is 3.835, not 3.92. The paper does not reconcile these aggregates or supply uncertainty estimates. Both sets of reported values are retained.
Figure 4. The real-robot evidence is an illustrated cloth-folding execution. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the photographs in the direction of the black arrows, from the spread garment toward the later folded configuration. These are images of physical manipulation, rather than reconstructions from a world model. Section 4.1 describes two Piper arms, an overhead camera and two wrist cameras, with 600 training trajectories. The deployment paragraph adds joint-angle inputs and one-second action chunks at 30 Hz. Those details explain the control setting, but the photographs do not reveal the intervening action values, observation refreshes or how the system handles an unsuccessful grasp. e-reale-setup
What it supports. The figure documents the authors' successful T-shirt-folding example and extends the evidence beyond simulated CALVIN scenes. It shows that the method was used for an actual bimanual manipulation task. It supplies neither a measured success probability nor a controlled comparison against other policies on physical hardware.
Where the evidence stops. No trial count, quantitative folding score or real-robot baseline is reported. The stated 110 ms latency 'for 50 steps' has an unclear timing boundary. Neither the photo spacing nor its arrows establish controller timing.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| CALVIN ABC→D five-task instruction chains Train exclusively on language-annotated data from scenes A/B/C; evaluate unseen scene D. Table 1 compares fine-tuned methods; VPP uses a single-view implementation. | MoWM: 0.943 / 0.873 / 0.812 / 0.750 / 0.675; average length 4.10. Stage 1–5 success rates (fractions); reported average completed task length | Pi0: 0.938 / 0.850 / 0.767 / 0.681 / 0.599; 3.92. VPP: 0.909 / 0.815 / 0.713 / 0.620 / 0.518; 3.58. Best values among the listed methods. The fifth-stage difference from Pi0 is 7.6 percentage points, approximately 12.7% relative (reader arithmetic), consistent with the long-horizon claim. No uncertainty is reported; average-length inconsistencies are preserved below. e-setupe-maine-gains |
| CALVIN feature-fusion ablation Table 2 compares concatenation, cross-attention and pixel-only features under the paper's CALVIN evaluation. | Concatenation: 0.943 / 0.873 / 0.812 / 0.750 / 0.675; 4.10. Stage 1–5 success rates; reported average completed task length | Cross-attention: 0.936 / 0.836 / 0.748 / 0.665 / 0.573; 3.80. Pixel-only: 0.927 / 0.831 / 0.741 / 0.652 / 0.560; 3.70. Concatenation gains 11.5 percentage points at stage five over pixel-only (reader subtraction). This supports feature combination in the tested setup; it does not isolate motion information from extra capacity or show latent-only performance. e-setupe-ablation |
| Real-world T-shirt folding AgileX with two Piper arms and three cameras: one overhead and two wrist-mounted; trained on 600 trajectories. | Figure 4 presents a successful sequence folding a T-shirt into a rectangle. Qualitative execution demonstration | No quantitative real-robot baseline, trial denominator or success rate is supplied. Evidence of a demonstrated physical execution, not a measured reliability estimate or broad real-world generalization result. e-setupe-real |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 2. Concatenation outperforms cross-attention and pixel-only features in the reported ablation. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Begin with the bottom row, which removes the latent feature contribution. Moving to cross-attention adds latent features through an attention-based fusion route; moving to the top row uses concatenation followed by projection. Keep the columns aligned with Table 1's chain metrics. All three rows use the MoWM naming, so the bottom row is the within-framework comparison to use when judging the added latent stream. It is not numerically identical to Table 1's VPP row. No latent-only row appears, even though the surrounding prose refers to two no-fusion variants. e-ablatione-maine-setupe-diagnostic
What it supports. The fifth-stage score rises from 0.560 with pixel-only features to 0.675 with concatenation, an 11.5-percentage-point difference calculated from the table. Cross-attention reaches 0.573. This supports the tested fusion design, while the ordering alone cannot determine whether dynamics information, extra capacity or optimization differences explain the gain.
Where the evidence stops. The prose's 11.2% average gain does not match the approximately 9.2% relative gain obtained from the displayed mean stage rates. There are no seed/error estimates, latent-only control or explicit capacity-matched control.
Figure 5. A selected rollout illustrates the authors' concern about static pixel predictions. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow the upper latent row and lower pixel row from left to right. The red box is an original annotation around repeated-looking frames in the pixel row; it labels that interval as having limited dynamics. Compare the changing arm configuration in the upper row with the relative persistence below. Unlike Figure 3, this crop does not provide a ground-truth row for judging which motion is correct. Section 4.3 uses the example to argue that latent features prioritize temporal structure, while also acknowledging appearance deviations in reconstructed latent frames. e-diagnostice-visualizatione-ablation
What it supports. The selected sequence illustrates a specific failure mode: a future prediction can preserve a plausible scene while changing too little over time. That observation motivates adding a temporal representation to pixel features. The quantitative fusion ablation is the separate evidence that the combined representation improves action outcomes in this setup.
Where the evidence stops. One annotated sequence does not measure how often pixel predictions stall or whether latent motion is accurate. The reconstruction decoder is another source of distortion. Claims about suppressing irrelevant pixels remain an interpretation of the visuals and ablation.
7. Analysis & limitations
7.1 What the evidence leaves open
Table 1 prints average lengths 4.10 and 3.92 for MoWM and Pi0, although their displayed stage rates sum to 4.053 and 3.835 (reader arithmetic). Those differ beyond ordinary rounding if rates represent cumulative chain completion. Preserve both columns; the aggregation discrepancy is unresolved. e-main
Section 4.3's claimed 11.2% average gain over pixel-only does not match the ratio of displayed mean stage rates, about 9.2%. The prose mentions 'both no-fusion variants', but Table 2 contains only one. Neither table reports confidence intervals or seed variation; Table 2's pixel-only row also differs from Table 1's VPP. e-maine-ablation
Decoder prose names Φ_fused as its condition, whereas Eq. (7) uses Φ_final after the residual. This report follows Eqs. (6)–(7), while retaining the discrepancy. Figure 1 omits the internal residual and combines training-stage symbols in one drawing. e-architecturee-fusion
Latent RGB reconstructions use a separately trained decoder, so their color deviations and motion patterns cannot alone establish which information the policy uses. Cross-attention optimization explanations are author hypotheses. Adaptive fusion and large-scale unannotated pretraining for zero-shot transfer are future work. e-visualizatione-diagnostice-future
7.2 Questions for discussion
- Does temporal shuffling remove the fusion advantage when capacity and action-decoder training are held constant?
- Can raw evaluation chains reconcile the reported stage success rates and average lengths?
8. Reproducibility audit
8.1 Requirements and known gaps
Reproduction requires SVD/VPP and V-JEPA 2 ViT-g initialization, matched language-annotated CALVIN data, temporal alignment, both predictive objectives and frozen world models during action training. Stage-one latent learning rate, exact forecast horizon and sampling intervals, selected U-Net layers, feature dimensions, diffusion schedules and software/checkpoint versions are not fully specified. e-pixele-latente-featurese-training
Reader-proposed checks: reproduce Table 2 with matched seeds, capacity and decoder budgets, including a shuffled-latent control; then test controlled background changes while holding task geometry fixed. Report raw chain outcomes, uncertainty and latency to distinguish useful dynamics from added features or appearance robustness. e-ablatione-diagnostice-main
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Does aligned latent prediction cause the fusion gain?
Reader-proposed experiment: repeat the CALVIN ABC→D ablation with identical data, frozen world-model checkpoints, action-training steps and evaluation chains across several seeds. Compare concatenation, cross-attention, pixel-only with a matched projection-parameter budget, and concatenation with temporally shuffled latent features. Retain the same action decoder and denoising budget. Log every chain outcome so stage rates and average length can be independently reconciled. The dynamics interpretation predicts a reproducible advantage for aligned latent forecasts over both shuffled features and the capacity control. If those controls match the full model, the proposed causal explanation weakens even if the original ranking is reproduced. e-setupe-architecturee-traininge-maine-ablatione-latent
Check 2: Does the representation resist appearance changes without losing task detail?
Reader-proposed experiment: evaluate matched CALVIN episodes after changing only background/table appearance, keeping object identity, pose, instruction and dynamics fixed. Compare full MoWM with the pixel-only ablation on executed chain success and object-motion prediction error. Use the same fixed latent visualization decoder across conditions, and separately reconstruct encoded ground-truth states to estimate decoder distortion. The proposed complementary-feature mechanism predicts a smaller action-performance drop for fusion while preserving sensitivity to task-relevant object changes. Robust-looking latent reconstructions without improved action robustness would fail to connect the visualization argument to control; similar gains on an object-detail control would require a different explanation. e-motivatione-visualizatione-diagnostice-ablatione-setup
8.3 Reading coverage
Visual audit: Visually inspected the title/version/author page, motivation and architecture, all method equations and training/setup pages, both quantitative tables, all five original figures, and the diagnostic/conclusion page. All six final crops were individually inspected; the architecture and main-table crops were tightened and inspected again. The page set includes all source pages underlying retained method, numerical, deployment and proposed-reproduction claims. Pages 11–12 contain references read in the complete text pass. No separate supplement was supplied and no code or experiments were inspected or run.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10. Appendix coverage: not present.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Title page and abstract (p. 1)
- 1 Introduction (pp. 1–2)
- 2 Related Works, including 2.1 and 2.2 (pp. 2–3)
- 3 Methodology, including 3.1 and 3.2 and Eqs. (1)–(7) (pp. 3–5)
- 4.1 Experimental Setup (pp. 5–6)
- 4.2 Main Results (pp. 6–9)
- 4.3 Ablation Study (pp. 9–10)
- 5 Conclusion and Future Work (p. 10)
- References (pp. 10–12)
Outside the original text pass
- Identity: the supplied title page identifies arXiv:2509.21797v3, dated 10 February 2026. Its title and all seven authors match the catalog. The catalog submission date is 26 September 2025; earlier versions were not supplied, so this report describes v3 and does not establish inter-version scientific changes.
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- The extraction limitation above was addressed by visually inspecting PDF pages 1–10, all five figures and both tables, and every final crop. References on pages 11–12 were read as text.
- Separate supplemental material availability has not been fully verified.
- No appendix is present in the supplied 12-page PDF. No separate supplement was supplied.
- Code and external links were not inspected; no experiments were reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e-identityPDF p. 1, title/author/affiliation block and arXiv margin
Title matches the catalog; arXiv:2509.21797v3 is dated 10 February 2026. Authors: Yangcheng Yu, Xin Jin, Yu Shang, Xin Zhang, Haisheng Su, Wei Wu, Yong Li. Affiliations: Tsinghua University; Manifold AI; Shanghai Jiao Tong University.
Go to primary source ↓e-motivationPDF pp. 1–2, Abstract and Section 1
The authors motivate combining detailed pixel features with compressed, motion-aware latent features for action decoding.
Go to primary source ↓e-architecturePDF p. 3, Figure 1 and caption; Section 2.2 final paragraph
Diagram shows separate encoders and predictive world models feeding modulation and action denoising. Encoder snowflakes and stage-one world-model flames indicate staged training; caption freezes the world models in stage two. Section 2.2 explicitly adopts future-state-to-action inverse dynamics.
Go to primary source ↓e-pixelPDF p. 4, Section 3.1, pixel-world-model paragraph and Eq. (1)
SVD, following VPP, is adapted to embodied instruction-conditioned video generation with text cross-attention and squared noise-prediction loss.
Go to primary source ↓e-latentPDF p. 4, Section 3.1, latent-world-model paragraph, Eq. (2) and final paragraph
V-JEPA 2 ViT-g encodes frames; a transformer with SwiGLU predicts next-state tokens from text and state histories using teacher forcing and L1 loss. Both world models share embodied datasets and aligned temporal sampling.
Go to primary source ↓e-featuresPDF pp. 4–5, Section 3.2, Eqs. (3)–(4)
The action pipeline describes one forward pass per world model, single-step pixel denoising, multiple U-Net upsampling features resized to a common grid, and separate projections of pixel and latent sequences to a shared embedding dimension.
Go to primary source ↓e-fusionPDF p. 5, Section 3.2, Eqs. (5)–(7) and adjacent paragraphs
Concatenation and linear projection form fused features; a learned gating matrix and pixel residual form final features. Action diffusion minimizes noise error. Prose calls the condition fused features, but Eq. (7) explicitly conditions on final features.
Go to primary source ↓e-setupPDF p. 5, Section 4.1 Datasets and Baselines; p. 6, VPP baseline and Section 4.2 opening
CALVIN trains on language-annotated A/B/C and tests D with five-subtask chains. Baselines are described as fine-tuned; VPP uses one view. Real data comprise 600 cloth-folding trajectories using two Piper arms and three cameras.
Go to primary source ↓e-trainingPDF p. 6, Section 4.1, Implementations of MoWM
Latent predictor: 24 layers, approximately 400M parameters, four H20 GPUs, about seven hours, 75 epochs × 300 steps, batch four per GPU/16 global, 12 workers, AdamW, cosine schedule, weight decay 0.04. Action stage: four H20 GPUs, 13 hours, 7,000 steps, batch 28, AdamW at 1e-4 and weight decay 0.05.
Go to primary source ↓e-mainPDF p. 7, Table 1, MoWM/Pi0/VPP rows, all success columns and Avg. Len
MoWM reports 0.943/0.873/0.812/0.750/0.675 and 4.10; Pi0 0.938/0.850/0.767/0.681/0.599 and 3.92; VPP 0.909/0.815/0.713/0.620/0.518 and 3.58. No uncertainty columns accompany the values.
Go to primary source ↓e-gainsPDF p. 6, Section 4.2 final paragraph; p. 8, Section 4.2 long-horizon paragraph
The prose reports 5.7% and 13.4% mean-stage relative improvements over the strongest VLA/world-model baselines and a 12.7% fifth-stage relative improvement over the strongest baseline.
Go to primary source ↓e-visualizationPDF p. 8, Figure 3 and Section 4.2 qualitative future-state prediction paragraph
Figure 3 juxtaposes initial observations, ground truth, latent reconstructions and pixel predictions for two instructions. Latent RGB visualization uses a separate approximately 8.04M-parameter CNN decoder, trained for 15,000 steps on 3,200 CALVIN images of 200×200×3 with MSE, on one H20 for about 30 minutes.
Go to primary source ↓e-realPDF p. 8, Figure 4 and caption; p. 9, Section 4.2 Real-world robot manipulation task validation
Figure 4 depicts cloth folding. Deployment uses 14 joint angles, images and task descriptions, reduces 50 Hz to 30 Hz, and predicts 30-timestep/one-second chunks. The text gives 110 ms latency 'for 50 steps' without a clear timing definition or quantitative success evaluation.
Go to primary source ↓e-ablationPDF p. 9, Table 2 and Section 4.3
Concat/cross-attention/pixel-only rows respectively report stage-five rates 0.675/0.573/0.560 and average lengths 4.10/3.80/3.70. Their full stage rows are 0.943/0.873/0.812/0.750/0.675; 0.936/0.836/0.748/0.665/0.573; 0.927/0.831/0.741/0.652/0.560. Prose claims 11.2% mean improvement and refers to two no-fusion variants although only one is tabulated.
Go to primary source ↓e-diagnosticPDF p. 9, Figure 5 and caption; p. 10, Section 4.3 continuation
A red box identifies repeated pixel-prediction frames as limited dynamics. Authors interpret latent reconstructions as retaining dynamics despite color deviations and propose optimization/alignment difficulty as possible reasons cross-attention underperforms.
Go to primary source ↓e-futurePDF p. 10, Section 5 final paragraph
Adaptive task-dependent fusion and large-scale unannotated-video pretraining enabling broader zero-shot transfer are proposed future directions.
Go to primary source ↓8.5 Primary sources
MoWM: Mixture-of-World-Models for Embodied Planning via Latent-to-Pixel Feature Modulation ↗
PDF · 6,151 extracted words
Source fingerprint
9ca4304a7a9b31ad17ef4e4ac3aa03d9dca00917bbfc7ccb662364ba57cf58d4