LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels
1. Paper overview
In one sentence: LeWM trades pixel reconstruction and elaborate anti-collapse losses for Gaussian-regularized latent prediction, enabling compact planning with task-dependent control and representation limits. identitytraining-objectiveplanning-objectivecontrol-resultsefficiencyprobes-pusht
| At a glance | What to know |
|---|---|
| Research problem | Source description A prediction-only JEPA can minimize its loss by mapping every image to the same embedding. The paper asks whether reward-free, reconstruction-free learning from pixels can avoid this collapse without frozen pretrained encoders, stop-gradient targets, or a complicated collection of weighted auxiliary losses. motivationtraining-objective |
| Core mechanism | Reader analysis LeWM jointly trains an approximately 5M-parameter encoder and 10M-parameter predictor using prediction error plus SIGReg. The main simplification concerns loss balancing; architecture, dropout, and planning choices still matter. architecturetraining-objectiveother-ablations |
| A key reported result | Push-T goal-conditioned control: 96 in Figure 6; 96.0 ± 2.83 in Table 5. Success rate (%). Figure 6; offline-trajectory start/goal pairs, goal 25 steps later, execution budget 50. Table 5 separately uses three training seeds and the same 50 evaluation trajectories. Figure 6: PLDM 78; DINO-WM pixels-only 74; DINO-WM+proprioception 92. An 18-percentage-point gain over PLDM. Table 5 labels a 92.0 ± 1.63 baseline simply DINO-WM, leaving its modality ambiguous; its caption calls dispersion variance, so ± is retained without relabeling it SD or CI. control-resultscontrol-protocolseed-variance |
| Reading caution | Source description The authors identify short planning horizons, sufficient offline coverage, difficulty fitting a high-dimensional Gaussian in low-diversity environments, and dependence on action labels. Inverse dynamics is future work, not the current action-extraction mechanism. limitations |
Core contributions
- Reader analysis
LeWM jointly trains an approximately 5M-parameter encoder and 10M-parameter predictor using prediction error plus SIGReg. The main simplification concerns loss balancing; architecture, dropout, and planning choices still matter. architecturetraining-objectiveother-ablations
- Author claim
The authors report competitive simulated control, faster planning than DINO-WM, and latent physical structure. Their stability evidence combines training curves, seed variation, and targeted design ablations. control-resultsefficiencyprobes-pushtseed-variancetraining-curves
Figure 1. Predict the next embedding while preventing the encoder from collapsing. Original paper, p. 1 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start at the two frame images on the left. The shared encoder maps the current observation to z_t and the next observation to z_{t+1}; action a_t enters the predictor alongside the current representation. The predicted next embedding is compared with the encoded next frame using MSE. Both encoder branches remain trainable, as confirmed by Equation (3) and Algorithm 3. Follow the red SIGReg branches to the inset: latent samples are projected along random directions, then their one-dimensional distributions are encouraged toward a Gaussian target. These branches regularize encoded representations, while the prediction branch learns action-conditioned temporal structure. architecturetraining-objectivesigreg-theoryimplementation
What it supports. The two losses address different failure modes. Prediction error encourages a representation that can forecast transitions, while SIGReg penalizes the constant-embedding shortcut. This permits joint learning from pixels without a frozen pretrained encoder. The regularizer acts during training; it is not a reward or an action-selection module.
Where the evidence stops. The inset is a conceptual distribution diagram. Appendix A's joint-distribution argument is asymptotic over projections; the finite training procedure does not guarantee that every control-relevant variable is preserved.
2. Motivation
2.1 The problem and the proposed response
A prediction-only JEPA can minimize its loss by mapping every image to the same embedding. The paper asks whether reward-free, reconstruction-free learning from pixels can avoid this collapse without frozen pretrained encoders, stop-gradient targets, or a complicated collection of weighted auxiliary losses. motivationtraining-objective
2.2 What this reading follows
Imagine learning to push an object from recorded images and actions without being told a reward function. LeWM learns which latent state should follow an action, while a Gaussian regularizer prevents the encoder from making every image look identical. At deployment, a separate optimizer searches for action sequences whose predicted endpoint resembles an encoded goal image. The resulting system performs strongly on Push-T and plans quickly, but its advantages vary across environments. This reading follows the training and planning diagrams, then separates executed simulated control, supervised physical probes, and mechanism ablations. The reviewed source is the June 3, 2026 arXiv v3 PDF. identitytraining-objectiveplanning-objectivecontrol-resultsefficiencyprobes-pusht
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Foundational work |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
This table preserves the labels recorded at reading time. The current major category is WAMs. View the current classification.
3.1 Evidence-based assessment
Supports the recorded classification
The foundational world-model/visual-representation classification is supported. Encoder and dynamics predictor learn together, but actions condition future prediction and are selected by an external optimizer. This does not establish joint future/action generation or inverse-dynamics control, so the recorded Not applicable WAM architecture/paradigm/quadrant labels are appropriate. architectureplanning-objectivelimitations
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Why predicting the next frame's embedding needs two objectives
The encoder supplies both the predictor's input and its training target. That makes the representation adaptable to the environment, but also creates an escape route: constant embeddings make next-state prediction trivial. LeWM counters this with SIGReg, applied to embeddings at each time step across a batch. Random unit-direction projections turn high-dimensional distribution matching into tractable univariate normality tests. Equation (3) combines this term with prediction error, and gradients pass through the entire model. The projection MLP after the encoder's [CLS] token matters because the final ViT LayerNorm would constrain the distribution being regularized. The stated defaults are 1,024 projections and λ=0.1. Calling λ the one effective loss hyperparameter expresses the observed insensitivity to projection count; it does not remove the need to choose predictor size, dropout, or a planner. architecturetraining-objectivesigreg-theoryimplementationother-ablations
Figure 4. The world model predicts consequences; CEM searches for actions that approach the goal. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the upper path from the initial image toward the terminal predicted embedding. The repeated predictor boxes reuse the learned dynamics model to evaluate a candidate action sequence; they are not separate policies. On the right, the goal image is encoded into the same representation space. The cost compares that goal embedding with the final predicted embedding. Follow the dotted path downward and back to the actions: it depicts solver updates to the candidate sequence, with model weights held fixed. CEM implements this loop by sampling plans, retaining the lowest-cost elites, and updating its sampling distribution. planning-objectivecemimplementationarchitecture
What it supports. LeWM remains in use at inference because every candidate plan is evaluated through latent rollouts. Appendix D specifies a five-block horizon spanning 25 environment steps. The implemented controller executes that entire optimized sequence before obtaining feedback and replanning, so its demonstrated feedback cadence is coarser than one-step MPC.
Where the evidence stops. Section 3.2 describes the general first-K-actions MPC scheme, while Appendix D specifies full-horizon execution. The figure shows optimization, not the environment-feedback loop. Its caption also mispoints to Figure 2 for training; the training schematic is Figure 1.
5.2 How a learned forward model becomes a controller
A forward prediction model answers what follows an action; it does not by itself decide which action should occur. LeWM supplies that missing decision through trajectory optimization. Encode the current observation and goal, sample action sequences, roll each sequence forward in latent space, and score its endpoint using squared distance to the goal embedding. CEM retains 30 elites from 300 samples and updates the sampling distribution. Model parameters stay fixed throughout this search. Appendix D specifies five model steps, each grouping five environment actions, and executes the whole resulting plan before replanning. This separates the paper's general MPC description from its actual reported cadence. It also explains why decoder quality and control success are different measurements: candidate actions are ranked in latent space, and the visualization decoder is unnecessary for this decision. planning-objectivecemimplementationdecoder
5.3 What the control and physics evidence can establish together
A useful interpretation is to treat control, probing, and surprise as complementary tests with different failure modes. Figure 6 evaluates whether planned actions reach a goal under the stated budget. Table 1 asks whether a supervised readout can recover physical variables from embeddings. Figure 8 asks whether sudden perturbations produce prediction error. None logically implies the other. Two-Room is especially revealing: LeWM retains position information in the probes while trailing PLDM in planning, leaving dynamics or search as plausible bottlenecks. Decoded rollouts and Cube probes also show that broad spatial structure can survive while rotational detail remains weak. My reading is that these experiments support useful task-specific latent dynamics and selective perturbation sensitivity, while leaving long-horizon transfer and general physical reasoning unestablished. control-resultscontrol-protocolprobes-pushtprobes-othersurprisedecoderlimitations
5.4 Training and inference
During training
Defaults are 1,024 SIGReg projections and regularization weight 0.1. Batches contain 128 four-frame subtrajectories at 224×224 resolution; frame skip five groups five consecutive actions into each action block. training-objectiveimplementation
Each world model trains for ten epochs. Push-T supplies 20,000 expert episodes averaging 196 steps; Two-Room supplies 10,000 heuristic episodes averaging 92. Cube and Reacher each supply 10,000 200-step episodes, collected with a benchmark heuristic and SAC policy respectively. datasets
During inference
Freeze the learned model, encode the goal, and rank candidate action sequences by terminal latent squared distance. CEM samples 300 candidates and retains 30 elites per iteration. Appendix D specifies up to 30 iterations for Push-T and ten elsewhere, refining Appendix B's blanket 30-iteration description. planning-objectivecemimplementation
The horizon is five action blocks, or 25 environment steps. Although Section 3.2 describes executing the first K actions before feedback, Appendix D explicitly executes the entire optimized horizon before replanning. A visualization decoder is diagnostic only and is absent from the planning objective. planning-objectiveimplementationdecoder
5.5 Implementation flow
- Encode each frame
A ViT-Tiny with patch size 14, 12 layers, 3 attention heads, and hidden dimension 192 produces a [CLS] representation. A one-layer MLP with Batch Normalization projects it beyond the final LayerNorm constraints. architecture
- Predict action-conditioned dynamics
A six-layer transformer with 16 heads and 0.1 dropout uses zero-initialized AdaLN action conditioning, temporal causal masking, and an output projector. History is three frames for Push-T/Cube and one for Two-Room; Reacher history is not specified. architectureimplementation
- Prevent a trivial latent solution
Teacher-forced next-embedding MSE updates both predictor and encoder, including the target embedding. Step-wise SIGReg matches random one-dimensional projections to a standard Gaussian using the Epps–Pulley statistic. No pretrained component, EMA, or stop-gradient is used. training-objectivesigreg-theoryimplementation
6. Experiments & results
LeWorldModel (LeWM) learns a compact, action-conditioned latent world model directly from offline image–action trajectories. Next-embedding prediction supplies dynamics supervision; SIGReg discourages representation collapse by matching random latent projections to a Gaussian target. A separate CEM solver chooses actions by predicting their consequences and matching a goal embedding. Strong Push-T performance and fast planning coexist with weaker results on Two-Room and OGBench-Cube than selected baselines. Physical probes and perturbation tests diagnose representations, rather than establish general physical reasoning.
6.1 Read the original evidence
Figure 6. Strong Push-T and Reacher results coexist with task-dependent weaknesses. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Compare methods within each panel before comparing environments. Red bars denote LeWM; the explicitly labeled DINO-WM+prop bars add proprioceptive information, whereas ordinary DINO-WM is evaluated without it under the main-text protocol. The vertical axis is success rate, with higher values better. Reacher includes fewer baselines, so absent bars are unreported comparisons. Appendix F.1 samples the start and reachable goal from the same offline trajectory, placing the goal 25 steps ahead and allowing 50 executed steps. These bars therefore measure a particular goal-conditioned control protocol, not arbitrary-goal transfer or the fixed-FLOP experiment shown separately in Figure 3. control-resultscontrol-protocolseed-varianceefficiencydatasets
What it supports. On Push-T, LeWM reaches 96% against PLDM's 78% and pixels-only DINO-WM's 74%, an 18-point improvement over PLDM. The pattern reverses elsewhere: DINO-WM reaches 86% on Cube against LeWM's 74%, and 100% on Two-Room against 87%. The evidence supports competitive performance rather than uniform dominance.
Where the evidence stops. The figure does not define its error bars. Table 5 labels a 92% Push-T result simply DINO-WM, although this figure assigns 92% to DINO-WM+prop and 74% to pixels-only DINO-WM. That appendix modality ambiguity remains unresolved.
Table 1. Recoverable physical state is informative, but probe quality is not equivalent to planning quality. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Choose a physical property first, then compare the three model rows under a single probe family. The Linear columns test direct linear accessibility; the MLP columns permit nonlinear decoding of information that may be entangled in the embedding. MSE measures prediction error and decreases with better performance, whereas Pearson r measures correlation and increases. Keep the two metrics separate: a high correlation need not imply equally small numerical errors. Finally, compare the linear and MLP columns for the same property to see whether a more flexible readout can recover information that a linear probe misses. probes-pushtprobes-otherarchitecturetraining-objective
What it supports. For block angle, LeWM's linear MSE is 0.187 with r=0.902, improving on PLDM's 0.446 and 0.745 while trailing DINO-WM's 0.050 and 0.979. LeWM's MLP block-angle r rises to 0.990. The table supports accessible physical information, with performance depending on the variable and readout.
Where the evidence stops. Probe training/test splits, target normalization, and the meaning of the reported ± terms are not specified. A probe uses physical-variable supervision for diagnosis; these numbers neither add privileged state to LeWM's default training nor prove causal physical understanding.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Push-T goal-conditioned control Figure 6; offline-trajectory start/goal pairs, goal 25 steps later, execution budget 50. Table 5 separately uses three training seeds and the same 50 evaluation trajectories. | 96 in Figure 6; 96.0 ± 2.83 in Table 5. Success rate (%) | Figure 6: PLDM 78; DINO-WM pixels-only 74; DINO-WM+proprioception 92. An 18-percentage-point gain over PLDM. Table 5 labels a 92.0 ± 1.63 baseline simply DINO-WM, leaving its modality ambiguous; its caption calls dispersion variance, so ± is retained without relabeling it SD or CI. control-resultscontrol-protocolseed-variance |
| Control beyond Push-T Figure 6; each environment uses goals 25 trajectory steps ahead and a 50-step execution budget. | Two-Room 87; Reacher 86; OGBench-Cube 74. Success rate (%) | PLDM: 97, 78, 65 respectively. DINO-WM: 100, 79, 86 respectively. LeWM improves on these baselines in Reacher but does not dominate across tasks. These are simulated control evaluations. control-resultscontrol-protocoldatasets |
| Planning efficiency Figure 3: timing averaged over 50 runs; separate fixed-FLOP control comparisons. | 0.98 seconds; fixed-FLOP Push-T 90 and Cube 74. Planning seconds; success rate (%) | DINO-WM: 47 seconds; fixed-FLOP Push-T 13 and Cube 48. About 48× faster in the reported setup. Fixed-FLOP results must not be mixed with Figure 6, and the PDF does not specify timing hardware or the full FLOP-matching recipe. efficiencyimplementation |
| Push-T physical-state probing Table 1; linear and MLP probes of learned embeddings. Probe split and target normalization are not specified. | Linear block-angle MSE 0.187 ± 0.359, r=0.902; MLP MSE 0.021 ± 0.139, r=0.990. MSE (lower); Pearson r (higher) | PLDM linear: 0.446 ± 0.625, r=0.745; DINO-WM linear: 0.050 ± 0.101, r=0.979. LeWM improves on PLDM but trails DINO-WM for this property. Reported ± terms lack a defined statistical interpretation. probes-pushtprobes-other |
| Adding reconstruction supervision Table 7; Push-T planning with versus without decoder loss. | Without decoder loss: 96.0 ± 2.83. Success rate (%) | With decoder loss: 86.0 ± 7.54. A ten-point reduction in the reported means supports omitting reconstruction in this setup; loss weighting and compute controls are insufficiently described for a universal conclusion. reconstruction-ablation |
| Violation-of-expectation detection Figure 8 and Appendix F.3; Two-Room, Push-T, and Cube trajectories with teleportation, color change, or no perturbation. | The authors report teleportation-related increases with p<0.01 in all three environments. Prediction-error surprise; paired t-test | Color-change increases are described as weaker and nonsignificant. Evidence concerns sensitivity to these synthetic perturbations. Test sample counts, exact p-values, and a matched perceptual-change control are not provided. surprise |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 15. The strongest robustness evidence concerns projection count, not every design choice. Original paper, p. 24 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read these as three separate interventions. The left panel changes embedding dimension and includes both PLDM and LeWM; its x-axis samples 8, 24, 96, 192, and 384 dimensions. The middle changes SIGReg's number of random projections from 64 to 1,024. The right changes quadrature resolution through the number of integration knots. All panels report Push-T success, but their vertical ranges differ, so visually similar line slopes are not directly comparable. The shaded region appears in the knots panel only. The default architecture and training loss are held in the background of these targeted ablations. sigreg-ablationstraining-objectiveimplementation
What it supports. LeWM maintains high success across the tested projection counts, supporting the decision to leave that SIGReg setting fixed. Representation dimension is less forgiving: very small embeddings lose substantial control performance. Integration resolution also matters at the lowest setting, with four knots visibly worse than the stronger settings to its right.
Where the evidence stops. The prose gives an embedding threshold 'around 184', although 184 is not plotted. The figure supports a broad saturation trend, not an exact threshold. Its knots curve also warrants a narrower robustness claim than complete insensitivity.
Table 7. Adding reconstruction loss reduces reported Push-T planning success in this comparison. Original paper, p. 25 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. The intervention is whether a decoder loss contributes to world-model training. Read the first row as the standard LeWM objective, combining next-embedding prediction and SIGReg. The second row adds reconstruction supervision. This distinction matters because the paper also uses a separate visualization decoder to display latent rollouts; those diagnostic pictures do not mean the default model was trained to reconstruct images. The single reported metric here is Push-T success rate, with higher better. Preserve the printed ± values when recording the result, and avoid assigning them a confidence-interval or standard-deviation interpretation that the table does not state. reconstruction-ablationdecodertraining-objective
What it supports. Success falls from 96.0 ± 2.83 without decoder loss to 86.0 ± 7.54 with it, a ten-percentage-point difference in the reported means. This favors reconstruction-free training in the tested configuration. The authors suggest that reconstructing additional visual detail may distract the representation from information useful for planning.
Where the evidence stops. The comparison does not fully specify reconstruction weight or match additional decoder compute. It therefore cannot establish that reconstruction is generally harmful, or identify irrelevant visual detail as the unique cause of the lower success rate.
7. Analysis & limitations
7.1 What the evidence leaves open
The authors identify short planning horizons, sufficient offline coverage, difficulty fitting a high-dimensional Gaussian in low-diversity environments, and dependence on action labels. Inverse dynamics is future work, not the current action-extraction mechanism. limitations
Two-Room probes recover position well despite weaker planning, so its control deficit cannot simply be equated with absent position information. Cube rotational and velocity probes also expose uneven representation quality. probes-othercontrol-results
Push-T ablations support a projection-count plateau, but few integration knots and very small embeddings perform worse. Figure 16 is nonmonotonic in λ; a generic logarithmic bisection guarantee for tuning success is not established. sigreg-ablationsregularizer-weight
Smooth loss curves and three seeds provide bounded stability evidence. PLDM's selected coefficients disable its IDM and temporal-covariance losses, so fluctuations in those plotted components cannot alone establish competing optimization gradients. training-curvesbaselinesseed-variance
7.2 Questions for discussion
- Does Gaussian regularization improve planning because it prevents collapse, shapes the goal metric, or both?
- Would the Two-Room gap persist with more frequent feedback while keeping training and planning compute controlled?
8. Reproducibility audit
8.1 Requirements and known gaps
A reproduction needs the documented offline datasets, frame/action grouping, trainable encoder/projectors, step-wise SIGReg, and exact CEM execution cadence. The PDF does not give the training optimizer, learning-rate schedule, GPU model, complete probe configuration, or an explicit train/test trajectory split. implementationdatasetscontrol-protocolprobes-other
Proposed checks: compare SIGReg against prediction-only training while measuring both embedding collapse and control; separately vary the number of executed action blocks at fixed model and planning horizon to test sensitivity to feedback cadence. training-objectiveplanning-objectiveimplementation
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Separate collapse prevention from useful dynamics
Reader-proposed experiment, not performed: train the same Push-T encoder and predictor on the same trajectory split with the published prediction-plus-SIGReg objective versus prediction loss alone. Use identical initialization seeds, optimizer settings, update budgets, and CEM evaluation; document the optimizer because the PDF does not specify it. Evaluate at least three seeds on the same 50 reachable start/goal pairs, reserving entire trajectories from training. Track embedding variance/effective rank, held-out next-embedding error, and control success. Low prediction error accompanied by collapsed embeddings and failed control in the prediction-only condition would support the anti-collapse mechanism. If SIGReg preserves diversity without improving control, it would separate distribution matching from learning useful dynamics. training-objectivesigreg-theoryimplementationdatasetsseed-variancecontrol-protocol
Check 2: Test whether feedback cadence explains planning failures
Reader-proposed experiment, not performed: freeze a trained Two-Room model and keep the five-block planning horizon and terminal cost fixed. Compare executing one action block before replanning with the reported five-block execution cadence, using paired initial states/goals and the same 50-step environment budget. First keep per-call CEM settings identical; then repeat with a matched total number of candidate rollout evaluations to separate better feedback from extra search compute. Record success and predicted-versus-observed endpoint error after execution. If more frequent feedback improves success under matched compute without changing probe accuracy, that would implicate rollout error or delayed correction rather than missing position information. Report a null result if the improvement disappears after compute matching. implementationplanning-objectivecontrol-protocolprobes-othercontrol-results
8.3 Reading coverage
Visual audit: All seven supplied text chunks were read individually, covering all 28 PDF pages, including references and Appendices A–I. All 19 figures and all 10 tables were visually inspected on the declared pages. The pass also inspected the title/version/affiliations, equations, CEM and training algorithms, baseline coefficients, dataset descriptions, evaluation protocols, and reproduction-relevant implementation details. Figure 1 arrows and regularization branches were cross-checked against Eqs. (1)–(3) and Algorithm 3; Figure 4 was checked against Eqs. (4)–(5) and Appendix D. Six original crops were inspected individually, with two table boundaries corrected and re-inspected to remove caption fragments. References-only pages 10–12 were read as text but not rendered. Separate supplements and linked code remain unverified; no experiments were run.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract and title block (p. 1)
- 1 Introduction (p. 2)
- 2 Related Work (pp. 3–4)
- 3 Method: LeWorldModel; 3.1 Learning the Latent World Model; 3.2 Latent Planning (pp. 4–6)
- 4 Latent Planning Performance; 4.1–4.3 (pp. 6–7)
- 5 Quantifying Physical Understanding; 5.1–5.2 (pp. 7–9)
- 6 Conclusion, including Limitations & Future Work (p. 9)
- References (pp. 10–13)
- Appendix A SIGReg and B Cross-Entropy Method (pp. 13–14)
- Appendix C Baselines, C.1–C.4 (pp. 14–17)
- Appendix D Implementation details and E Environment & Dataset (pp. 17–18)
- Appendix F Evaluation Details, F.1–F.3 (pp. 19–22)
- Appendix G Ablations (pp. 22–26)
- Appendix H Temporal Latent Path Straightening (pp. 26–27)
- Appendix I Training Curves (pp. 26–28)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- The text-extraction image omission was addressed by inspecting the retained PDF: all 19 figures and all 10 tables were visually reviewed. Separate supplements remain unverified.
- Identity/version note: the title and ordered five-author list match the catalog. The inspected artifact is arXiv:2603.19312v3 [cs.LG], dated 3 June 2026. The catalog submittedDate is 2026-03-13; that is not the observed revision date. Earlier revisions and their changes were not supplied or compared.
- Code and linked resources were not inspected; no experiments were reproduced. No external sources were acquired.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
identityPDF p. 1, title block, author affiliations, equal-contribution footnote, and arXiv margin stamp
The observed title matches the supplied primary title. Authors in order are Lucas Maes, Quentin Le Lidec, Damien Scieur, Yann LeCun, and Randall Balestriero; the first two contributed equally. Affiliations are Mila & Université de Montréal, New York University, Samsung SAIL, and Brown University. Stamp: arXiv:2603.19312v3 [cs.LG], 3 Jun 2026.
Go to primary source ↓motivationPDF p. 2, Section 1, representation-collapse discussion; p. 1, Abstract and Figure 1
Prediction alone admits constant embeddings. The authors motivate joint pixel-based JEPA training without frozen pretrained encoders, EMA, stop-gradient, reward supervision, or image reconstruction.
Go to primary source ↓architecturePDF pp. 4–5, Section 3.1, Model Architecture; p. 1, Figure 1
ViT-Tiny encoder: roughly 5M parameters, patch 14, 12 layers, 3 heads, hidden dimension 192; [CLS] plus one-layer MLP/BatchNorm projection. Predictor: roughly 10M parameters, 6 layers, 16 heads, 10% dropout, zero-initialized AdaLN action conditioning, causal history masking, and a matching output-projector design.
Go to primary source ↓training-objectivePDF p. 5, Section 3.1, Eqs. (1)–(3); p. 17, Algorithm 3
Teacher-forced next-embedding squared error plus λSIGReg trains all components with gradients through targets. Defaults are M=1024 projections and λ=0.1. Algorithm 3 applies regularization step-wise. The displayed pseudocode contains incomplete call syntax and is not a complete executable specification.
Go to primary source ↓sigreg-theoryPDF p. 13, Appendix A, Eq. (6), EP integral, and Cramer–Wold statement
Random unit directions reduce distribution matching to univariate Epps–Pulley characteristic-function discrepancies. Appendix A states Gaussian joint-distribution matching in an asymptotic projection limit and approximates the integral with quadrature.
Go to primary source ↓planning-objectivePDF p. 4, Figure 4; pp. 5–6, Section 3.2, rollout equation and Eqs. (4)–(5)
Initial and goal observations are encoded; action-conditioned latent rollouts are scored by terminal squared goal distance. World-model parameters stay fixed while actions are optimized. Section 3.2 describes executing the first K planned actions and then replanning.
Go to primary source ↓cemPDF pp. 13–14, Appendix B and Algorithm 2
CEM samples Gaussian candidate plans, selects lowest-cost elites, and updates distribution mean and variance. Appendix B gives 300 samples, 30 iterations, and 30 elites, and notes nonconvexity and high-dimensional search limitations.
Go to primary source ↓implementationPDF pp. 17–18, Appendix D, preprocessing, Predictor Architecture, and Planning solver
Batch 128, four 224×224 frames, frame skip five and action blocks of five. History is three for Push-T/Cube and one for Two-Room; Reacher history is omitted. CEM uses 300 samples, 30 elites, initial variance one, at most 30 iterations for Push-T and ten elsewhere. All five optimized blocks, spanning 25 environment steps, are executed before replanning. Training optimizer, learning rate, hardware model, and library versions are not specified here.
Go to primary source ↓datasetsPDF p. 18, Appendix E, entries a–d
Two-Room: 10,000 noisy-heuristic episodes, average length 92. Push-T: 20,000 expert episodes, average length 196. Single-cube OGBench: 10,000 heuristic episodes of 200 steps. Reacher: 10,000 SAC episodes of 200 steps. Each world model trains for ten epochs; these are continuous-control benchmark environments.
Go to primary source ↓control-protocolPDF p. 19, Appendix F.1; p. 6, Section 4.1 Baselines
Start states and goals are sampled from the same offline trajectory, with goals 25 steps later and 50 executed-step budgets for all four environments. No explicit train/test trajectory partition or numerical success tolerance is supplied in F.1. DINO-WM excludes proprioception unless explicitly labeled otherwise.
Go to primary source ↓control-resultsPDF p. 7, Figure 6, four labeled environment panels and caption
LeWM/PLDM/DINO-WM success rates are Two-Room 87/97/100, Reacher 86/78/79, Push-T 96/78/74, Cube 74/65/86. Push-T DINO-WM+prop is 92. Cube GCBC is 84. Figure 6 does not define its error-bar statistic.
Go to primary source ↓efficiencyPDF p. 3, Figure 3 and caption; p. 7, Section 4.2 continuation
Timing averaged over 50 runs is 0.98 s for LeWM versus 47 s for DINO-WM. Fixed-FLOP success is 90 versus 13 on Push-T and 74 versus 48 on Cube. The text describes 48× speedup; the caption cites roughly 200× fewer tokens. Timing hardware and the numerical FLOP budget are not supplied.
Go to primary source ↓probes-pushtPDF p. 8, Section 5.1 and Table 1, Agent Location, Block Location, and Block Angle rows
Table 1 reports linear/MLP MSE and Pearson r. Block-angle linear LeWM/PLDM/DINO-WM MSE: 0.187±0.359/0.446±0.625/0.050±0.101, with r 0.902/0.745/0.979. LeWM MLP block-angle MSE is 0.021±0.139, r 0.990; MLP block-location MSE is 0.001±0.006, r 0.999. The ± convention is not defined.
Go to primary source ↓probes-otherPDF pp. 20–21, Appendix F.2, Tables 3–4 and probing description
Linear and nonlinear probes measure physical-variable MSE and Pearson correlation. Two-Room LeWM and PLDM linear position MSE both equal 0.008 with r 0.996. Table 4 shows weaknesses in LeWM joint velocity and rotational recovery relative to DINO-WM. Probe architecture details, split, target normalization, and uncertainty definitions are not provided.
Go to primary source ↓decoderPDF p. 8, Figure 7 and Decoding Latent Space; p. 18, Appendix D, Decoder (Visualization Only); pp. 19–20, Figures 9–10
A separate diagnostic transformer decoder maps the 192-dimensional [CLS] embedding to images. Reconstruction does not train the default world model. Decoded open-loop rollouts retain broad scene layout while fine end-effector orientation degrades.
Go to primary source ↓surprisePDF p. 9, Figure 8 and Section 5.2; pp. 21–22, Appendix F.3 and Figure 11; p. 23, Figures 12 and 14
Surprise is prediction error under no perturbation, abrupt object-color change, or teleportation. Figure 8 reports paired-test p<0.01 for physical perturbations in all three environments and weaker nonsignificant color effects. Appendix F.3 specifies perturbation policies and which objects change, but not test sample counts or exact p-values.
Go to primary source ↓sigreg-ablationsPDF p. 24, Figure 15 and Appendix G, Embedding dimensions / Number of projections paragraphs
Push-T plots vary embedding dimension 8/24/96/192/384, projections 64/256/512/1024, and integration knots 4/8/12/17/32. Performance is strong across projection counts, but low embedding dimensions and four knots fare worse. The text's threshold 'around 184' is not one of the plotted dimensions.
Go to primary source ↓seed-variancePDF p. 24, Table 5, caption and all rows
Three training seeds evaluated on the same 50 Push-T trajectories yield DINO-WM 92.0±1.63, PLDM 78.0±5.0, LeWM 96.0±2.83. Caption says mean and corresponding variance. The table does not mark DINO-WM's proprioception setting, unlike Figure 6.
Go to primary source ↓reconstruction-ablationPDF p. 25, Table 7 and Appendix G, Decoder paragraph
Push-T success without decoder loss is 96.0±2.83, with decoder loss 86.0±7.54. The authors suggest reconstruction may favor unnecessary visual details; reconstruction weighting, added compute controls, and a table-specific uncertainty definition are not supplied.
Go to primary source ↓other-ablationsPDF p. 24, Table 6; p. 25, Table 8; p. 26, Table 9
Predictor size, encoder backbone, and dropout affect Push-T control. Table 9 reports dropout 0.0/0.1/0.2/0.5 success 78±6.54/96.0±2.83/85.33±5.74/66.67±4.11. Thus one effective SIGReg coefficient does not mean all architecture and regularization settings are immaterial.
Go to primary source ↓solver-ablationPDF p. 26, Table 10
Push-T LeWM success: CEM 96.0±2.83, SGD 26±4.32, RMSProp 67.33±2.49, Adam 84±7.12. PLDM reaches 78.0±5.0 with CEM and 80±3.27 with Adam. Solver optimization budgets and step-size settings are not detailed in the table.
Go to primary source ↓regularizer-weightPDF p. 25, Figure 16 and caption; p. 24, Weight of SIGReg regularization paragraph
The sampled λ-success curve peaks near 0.09 and drops at 0.5; the plotted default λ=0.1 is approximately 80%, despite a caption claiming above 80% throughout [0.01,0.2]. The authors propose bisection but do not supply assumptions guaranteeing optimization of this nonmonotonic success curve.
Go to primary source ↓training-curvesPDF p. 7, Section 4.3 Training Curves; pp. 26–28, Appendix I and Figures 18–19
LeWM prediction and SIGReg losses fall and plateau. PLDM shows component-dependent fluctuations, including IDM, covariance over time, and temporal smoothness. These plots describe observed optimization trajectories, not a proof of training convergence.
Go to primary source ↓straighteningPDF pp. 26–27, Appendix H, Eq. (9), and Figure 17
Temporal straightness averages cosine similarities of successive latent velocities. LeWM's paths become straighter without an explicit straightening term; the proposed explanation in terms of unconstrained temporal structure is a hypothesis.
Go to primary source ↓limitationsPDF p. 9, Section 6, Limitations & Future Work
The authors identify short horizons, offline-data coverage requirements, low-diversity difficulty for the Gaussian prior, and dependence on labeled actions. Large-scale video pretraining and inverse dynamics are proposed future directions.
Go to primary source ↓baselinesPDF pp. 14–17, Appendix C, especially Eq. (8), hyperparameter search paragraph, and Table 2
DINO-WM freezes DINOv2 features. PLDM has prediction plus six weighted regularizers in the stated formulation; after a 256-configuration Push-T search, Table 2 lists α=18, β=12, γ=0.2, ζ=0.7, ν=0, μ=0. The latter two disable temporal-covariance and IDM contributions. GC-RL and GCBC use DINOv2 patch representations.
Go to primary source ↓8.5 Primary sources
LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels ↗
PDF · 11,904 extracted words
Source fingerprint
553a01501ac74ec56721cfdb936d3825ac941734133a1fd761f614c01b6887e9