Learning Latent Action World Models In The Wild
1. Paper overview
In one sentence: Regularized continuous latent actions turn natural-video prediction into a reusable planning interface, but that interface still needs visual context and supervised action mapping. e02e03e04e09e11e12
| At a glance | What to know |
|---|---|
| Research problem | Source description Natural videos provide abundant action-free observations but no shared embodiment. An inverse dynamics model can explain a transition by encoding motion, environmental noise, or even the future image. The problem is to constrain that information while retaining complex changes such as a person entering a scene, then connect the learned space to interpretable control. e02e03e04 |
| Core mechanism | |
| A key reported result | Goal-conditioned arm displacement planning: Noisy/high: 0.10 m. Mean Δxyz (m), lower is better. DROID-trained controller; recorded Franka Panda validation trajectories; 64 sampled episodes from 16 videos, four frames at 4 fps, horizon 3. Noisy/mid 0.11; sparse/high 0.13; discrete/high 0.14; V-JEPA 2-AC 0.15; V-JEPA 2 + WM 0.05. Table S1 places the best noisy model ahead of V-JEPA 2-AC but behind the specialized comparator. The measured quantity compares planned and recorded displacement; it is not a closed-loop manipulation success rate. No uncertainty is supplied. e12e14 |
| Reading caution | Source description Static regularization cannot adapt to each video's action complexity. Frozen representations may hinder prediction. Direct planning in the latent space remains open; Appendix B's qualitative sampling analysis shows mismatch between inferred and sampled high-capacity continuous latents. e18 |
Core contributions
Figure 2. A future-informed action channel connects inverse dynamics to prediction. Original paper, p. 3 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start at the two frame stacks. Both pass through the encoder marked with a snowflake, indicating frozen weights. Their representations feed the inverse dynamics block gφ, which produces z. Follow the upward arrow from z into the predictor pψ; the left encoded observation also feeds that predictor. The right encoded observation supplies the target for the prediction loss. The lateral E branch and the blue inset describe information constraints: noisy, sparse or quantized actions. Figure 2 is schematic; Appendix A states the combined prediction-plus-regularization objective. The future branch belongs to training and IDM diagnostics. Later planning instead obtains latent actions through the real-action controller. e03e04e05e11e21
What it supports. The learned object is a conditional world model with an inferred action interface. Freezing the encoder fixes the representation space, while jointly training inverse and forward dynamics makes the latent explain transitions useful for prediction. The information constraint is central because inverse dynamics has access to the answer.
Where the evidence stops. The diagram does not show a policy that predicts executable actions. The noisy regularizer's printed negative KL sign conflicts with the prior-matching description and additive loss; its intended optimization sign remains unresolved in the supplied version.
2. Motivation
2.1 The problem and the proposed response
Natural videos provide abundant action-free observations but no shared embodiment. An inverse dynamics model can explain a transition by encoding motion, environmental noise, or even the future image. The problem is to constrain that information while retaining complex changes such as a person entering a scene, then connect the learned space to interpretable control. e02e03e04
2.2 What this reading follows
What should an action mean in a video containing moving people, a moving camera and objects entering view? This paper learns an answer from transitions rather than action labels. Its inverse dynamics model sees the future during training, while a constrained latent channel asks the predictor to retain only useful transition information. The resulting actions can transfer motion across different objects. Turning them into a planner requires another step: a controller trained with real action labels and visual context. Read the figures as a sequence of tests of capacity, transfer and control, while keeping reconstruction quality separate from the paper's recorded-trajectory planning metrics. e02e03e04e09e11e12
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Foundational work |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
This table preserves the labels recorded at reading time. The current major category is WAMs. View the current classification.
3.1 Evidence-based assessment
Supports the recorded classification
The foundational-work classification and latent-action-pretraining/world-simulator subcategories fit this study. The catalog's Not applicable quadrant is treated as a scope choice. Architecturally, a frozen encoder, separate IDM and forward predictor, and later controller are distinct components. Joint training does not establish one backbone jointly predicting future states and executable actions. Inverse dynamics learns the latent interface; controller-conditioned prediction supports inference-time planning. e03e05e11e12
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Why observing the future creates both the action signal and the shortcut
Consider two encoded observations of a person entering an initially empty room. Predicting the second from the first alone is ambiguous; inverse dynamics removes that ambiguity by observing both and supplying a latent action. The forward model learns how to use that action together with the past. Yet a channel rich enough to carry the entire second representation would make reconstruction easy without discovering useful action structure. The paper therefore changes the channel's information constraint while holding the surrounding prediction setup fixed. Sparse and noisy channels retain more complex transitions than the tested quantization scheme. This does not mean unrestricted capacity is optimal: the later controller must produce a latent from known actions and past context, without the IDM's future observation. The training advantage and downstream difficulty originate in the same information asymmetry. e03e04e05e07e11e16
5.2 What a successful round trip says about an action
The cycle starts with a transformation inferred on one video, tests whether it can change another, and asks whether the change can be inferred again. That structure explains why Figure 7 and Table 2 belong together: the images show an interpretable direction of motion, while the table measures reconstruction after the round trip. Reader interpretation: this is evidence of recoverable transformations rather than a complete test of causal or semantic actions. A latent that merely moved image content could still be cycle-consistent. The scene-cut experiment checks the more extreme shortcut of copying the entire future, but it cannot certify the absence of all leakage. Figure 8 further narrows the interpretation by showing camera-relative locality. The learned interface is useful across objects precisely because it is not tied to one consistent robot or human embodiment. e08e09e10
5.3 How supervised action mapping turns the predictor into a planner
The natural-video model does not initially know what a robot displacement command means. The controller supplies that connection by predicting inferred latent actions from real actions and the previous visual representation. Visual context matters because the same physical command can correspond to different image changes across camera viewpoints. During planning, CEM proposes real action sequences; the controller translates each action using the current predicted state, and the world model rolls the state forward. The optimizer compares the final predicted representation with the goal representation. Evaluation then uses a different quantity: cumulative planned displacement versus the recorded displacement for the arm task, or trajectory errors for navigation. This distinction explains how a model with less attractive decoded rollouts can still select better plans. It also makes clear where downstream action supervision enters a system pretrained on unlabeled videos. e11e12e13e14e16e20
5.4 Training and inference
During training
World model and IDM train jointly with teacher forcing on YoutubeTemporal-1B: 16-frame clips at 4 fps, 30,000 iterations, batch 1,024. Muon uses learning rate 0.02 and AdamW 6.25×10⁻⁴, with 10% warmup, cosine decay and weight decay 0.04. Appendix A reports approximately 12 hours on 64 H100 GPUs. e05
The visualization decoder is a separate causal ViT-L trained with L1 and perceptual losses. Controllers train on DROID or RECON for 3,000 iterations, batch 256, using eight-frame clips duplicated to sixteen and AdamW learning rate 10⁻³. e06e11
During inference
IDM-conditioned rollouts require observed future frames and serve as an idealized diagnostic. For planning, the controller replaces that unavailable inference path: candidate real actions and recursively predicted states generate latent actions and future states. CEM minimizes distance to the encoded goal. e07e11e12
DROID uses horizon 3, 300 candidates, 10 elites and 15 CEM iterations. RECON uses horizon 8 at 4 fps, 120 candidates and one iteration, restricting trajectories to a straight-line action divided over the horizon. These are different protocols. e12e13
5.5 Implementation flow
- Encode and infer transitions
Frozen, frame-causal V-JEPA 2-L produces states. The inverse dynamics model observes consecutive states, including the future, and infers a default 128-dimensional latent action. The ViT-L world model uses past representations and frame-wise AdaLN-zero action conditioning to predict the next representation. e03e05
- Restrict the information channel
Sparse actions combine an L1 constraint with norm and variance/covariance/mean terms against degeneracy. Noisy actions use a Gaussian-prior formulation; discrete actions use a codebook with unused-code resets. These alternatives constrain information, without guaranteeing that it describes only controllable causes. e04e05e08
- Learn the real-action interface
The later controller minimizes L2 error against inferred latent targets. Two self-attention blocks process the previous frame representation, followed by cross-attention with actions embedded by a three-layer MLP. A linear projection produces the latent action. Frame duplication compensates for tokenization that groups two frames per latent. e11
6. Experiments & results
A frozen video encoder supports jointly learned inverse dynamics and latent-conditioned future prediction. On natural videos, constrained continuous actions capture richer changes than the tested quantization scheme. A separately supervised controller makes this space usable for short-horizon planning, but latent predictability, visual quality and planning accuracy are different objectives.
6.1 Read the original evidence
Figure 7. Motion passes from a person to a ball and back. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each row from left to right across the dashed context/prediction boundary. The source row supplies a man moving left. In the first transfer, latent actions inferred from that source condition predictions of a different video containing a ball; the ball's earlier motion changes toward the transferred direction. The second transfer is a new inference step: actions are inferred from the generated ball sequence and then applied back to the original human video. It is not a second independent copy of the source action. This order matches the cycle protocol in Figure 5b. The paper credits its authors for the human footage and Riochet and colleagues for the ball video. e09e10
What it supports. The example supports transfer of a spatial transformation across visually different objects. Returning the motion to the human also illustrates why re-inference is informative: the first transfer leaves a transformation that the IDM can recover. The next table tests this cycle quantitatively over video pairs.
Where the evidence stops. These are decoded predictions, not observations of a controlled physical system. One selected example cannot establish semantic action equivalence. Figure 8 indicates that the learned transformations depend on image location, so transfer should not be interpreted as an embodiment-independent command such as walking.
Table 2. Small cycle penalties must be read alongside absolute prediction quality. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Each dataset has an Original column, using actions inferred directly from the source, and a Transfer column, after applying actions to another video and re-inferring them. Values are LPIPS over two seconds; smaller is better. Read within a row to assess the cycle penalty, then compare final Transfer values between rows. For example, sparse/high on Kinetics moves from 0.19 to 0.24, whereas discrete/low moves from 0.32 to 0.33. The smaller discrete increase accompanies worse final reconstruction. The blue multipliers are the paper's reported ratios; they need not equal a quotient recomputed from the rounded displayed cells. e08e09
What it supports. High-capacity sparse actions retain lower final error than their low-capacity counterparts on both datasets: 0.24 versus 0.31 on Kinetics and 0.23 versus 0.29 on RECON. A representation can therefore lose more relative fidelity during a cycle yet still explain more of the resulting video.
Where the evidence stops. Random-video transfer has no uniquely specified semantic ground truth. Cycle consistency alone could reward shortcuts; the separate scene-cut test in Table 1 argues against complete future-frame copying, while Figure 6 still shows partial future information. Evaluation sample counts and uncertainty are not supplied here.
Table S1. Better-looking rollouts and better displacement plans select different capacities. Original paper, p. 20 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Treat the panels as separate measurements. The left compares decoded rollout quality under future-informed IDM actions and controller-predicted actions; the main text identifies controller quality with LPIPS. The right evaluates planned cumulative displacement against recorded ground truth, in meters. Compare the noisy rows: the mid-capacity controller has lower rollout error than the high-capacity controller, but high capacity has slightly lower planning error. Appendix A defines the latter evaluation as 64 sampled episodes from 16 validation videos, with horizon three. The bottom two rows are action-conditioned world-model comparators; use these printed table values for numerical comparisons rather than reading approximate baselines from Figure 11. e11e12e14e16e22
What it supports. Noisy/high reports 0.10 m displacement error, compared with 0.15 m for V-JEPA 2-AC and 0.05 m for V-JEPA 2 + WM. Noisy/mid has the better controller LPIPS, 0.11 versus 0.12, but planning error is 0.11 m. The useful action interface does not erase the gap to the strongest comparator.
Where the evidence stops. The metric is an offline comparison with recorded displacement, not a manipulation success rate or a closed-loop robot trial. The table supplies no confidence intervals. Its V-JEPA 2-AC value also differs from the dashed baseline's position in Figure 11; this edition preserves the table's number.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Goal-conditioned arm displacement planning DROID-trained controller; recorded Franka Panda validation trajectories; 64 sampled episodes from 16 videos, four frames at 4 fps, horizon 3. | Noisy/high: 0.10 m. Mean Δxyz (m), lower is better | Noisy/mid 0.11; sparse/high 0.13; discrete/high 0.14; V-JEPA 2-AC 0.15; V-JEPA 2 + WM 0.05. Table S1 places the best noisy model ahead of V-JEPA 2-AC but behind the specialized comparator. The measured quantity compares planned and recorded displacement; it is not a closed-loop manipulation success rate. No uncertainty is supplied. e12e14 |
| Goal-conditioned navigation planning RECON-trained controller, unseen-video goal-image evaluation; horizon 8 at 4 fps, straight-line planning with one CEM iteration. | Noisy/high: RPE 0.40, ATE 1.40. RPE and ATE, lower is better | NoMaD: 0.52 / 1.93; NWM: 0.35 / 1.13 (RPE / ATE). Table S2 supports improvement over NoMaD, not parity with NWM. The supplied protocol does not state a navigation evaluation count or uncertainty; the table does not label metric units. e13e15 |
| Scene-cut future-copying diagnostic Kinetics videos with artificially exchanged endings; IDM receives the transition including the new future. | High-capacity sparse: 0.20 / 0.50. LPIPS without / with scene change, lower is better | High-capacity noisy: 0.21 / 0.54; high-capacity discrete: 0.29 / 0.68. Large error increases argue against perfect next-frame copying in this test. They do not establish zero leakage or causal disentanglement; Figure 6 shows some future content entering the prediction. e08 |
| Action transfer cycle consistency Infer on video A, apply on B, re-infer and apply back on A; two-second predictions on Kinetics and RECON. | High-capacity sparse: Kinetics 0.19 / 0.24; RECON 0.20 / 0.23. Original / transferred LPIPS, lower is better | Low-capacity sparse: Kinetics 0.26 / 0.31; RECON 0.24 / 0.29. High capacity retains better absolute reconstruction after transfer. Random-video transfer has no uniquely defined semantic target, so cycle consistency is evidence of reusable transformations rather than a semantic action accuracy measure. e09 |
| Effect of domain-specific pretraining data Separate Appendix D sweep mixing DROID and YoutubeTemporal-1B before controller training. | Noisy model: 0.14 at 0% DROID, 0.09 at 10%, and 0.06 at 75% or 90%. Δxyz, lower is better | Sparse model: 0.14 at 0% and 0.13 at 10%. The benefit depends on regularization and data mixture. These sweep configurations are not interchangeable with Table S1. Table S3 also contains an unresolved controller-LPIPS anomaly. e19e23 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 4. Continuous constraints offer a wider range of predictive capacity. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the horizontal axis first: smaller in-the-wild prediction error means greater operational latent capacity. Each colored rectangle spans the performance reached while varying that family's constraint, rather than an uncertainty interval around one run. The gray dashed line is the unconstrained reference; the red dashed line is prediction without action conditioning. Sparse and noisy variants cover broad ranges, whereas the discrete range stays near the unconditioned reference. Each latent-conditioned variant uses inverse dynamics to obtain actions from observed transitions. This makes the figure a test of what information the channel can express, before asking whether a controller can predict the same latent actions. e04e05e07e16
What it supports. The tested continuous spaces can accommodate richer transitions as their constraints relax. Vector quantization offers much less flexibility in this setup. This motivates examining continuous latents further, but the leftmost error is not automatically the best setting for downstream control: the controller must infer actions without seeing the future.
Where the evidence stops. Capacity here is defined through prediction error, not a measured information-theoretic quantity. The rectangles are sweep ranges, not confidence intervals. The result concerns the tested architectures and codebook sweep and does not establish that every discrete action representation must fail.
Figure 11. The capacity that improves reconstruction need not minimize planning error. Original paper, p. 11 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Compare columns within each dataset row. The left column measures decoded controller rollouts, while the right measures planning error: displacement for DROID and RPE for RECON. The shared horizontal coordinate is in-the-wild IDM prediction error, so moving left corresponds to greater latent capacity, not weaker performance on that pretraining diagnostic. Colors identify constraint families. A controller-LPIPS minimum does not necessarily line up with a planning minimum at the same horizontal position. This is why the paper evaluates both outcomes rather than choosing a latent space from reconstruction alone. The dashed baselines belong to each task's own planning protocol and should not be compared across rows. e07e11e12e13e14e15e16e22
What it supports. The sweeps show that the most constrained and most expressive settings need not provide the most useful controller predictions. Planning is also nonmonotonic, and its preferred setting can differ. A reproduction should measure each stage separately: future-informed latent prediction, controller approximation and goal-based action selection.
Where the evidence stops. RECON LPIPS values in this plot conflict with Table S2, and the DROID V-JEPA 2-AC baseline differs from Table S1. The supplied source gives no reconciliation. Use this faithful crop for qualitative trends and the explicitly cited tables for numerical results; no uncertainty bands are shown.
7. Analysis & limitations
7.1 What the evidence leaves open
Static regularization cannot adapt to each video's action complexity. Frozen representations may hinder prediction. Direct planning in the latent space remains open; Appendix B's qualitative sampling analysis shows mismatch between inferred and sampled high-capacity continuous latents. e18
Rollout quality does not reliably rank planners. Scaling improves IDM prediction, but longer training produces the clearest planning improvement; more pretraining data has no consistent planning trend in the reported sweep. e16e17
Controllers can fail when the robot arm begins outside the image. Without visual representations, the displayed action-only controller leaves the arm static. These examples limit claims of a universal action interface. e20
Reporting ambiguities remain: the noisy regularizer has a printed negative KL sign despite prior-matching prose and an additive training loss; Figure 11 and Table S2 disagree on RECON rollout values; Table S3 prints 0.9 for noisy/100%-DROID controller LPIPS. No corrections are inferred. e21e22e23
7.2 Questions for discussion
- How much of cycle consistency reflects camera-local motion rather than reusable semantic actions?
- Which capacity should be chosen when the best rollout model differs from the best planner?
- Would viewpoint-balanced controller training preserve control when the arm starts outside the frame?
8. Reproducibility audit
8.1 Requirements and known gaps
Reproduction requires the frozen encoder, natural-video data, predictor/IDM, decoder for LPIPS, and labeled DROID/RECON controllers. Preserve frame duplication and each planner's budget. Appendix A supplies optimizer settings and regularization sweeps, but not a complete IDM architecture, explicit Low/Mid/High-to-coefficient mapping for every table, or full evaluation seeds and split identifiers. e03e05e06e11e12e13e14e15
Proposed checks: compare capacity settings with identical controller data and planning budgets; separately compare visual-context and action-only controllers across camera viewpoints. Measure latent prediction, decoded error and displacement error independently, including uncertainty across seeds. These are reader proposals, not completed experiments. e11e14e16e20
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Does the best latent predictor select the best planner?
Reader-proposed check, not run: after resolving the printed noisy-loss sign, train matched noisy models at three coefficients from Appendix A, such as 5×10⁻³, 5×10⁻⁵ and 10⁻⁶. Hold encoder, training data, updates and controller data fixed. Compare IDM error, controller LPIPS and Δxyz on the same DROID evaluation draws, using horizon 3, 300 candidates, 10 elites and 15 CEM iterations. Repeat across seeds and report uncertainty. The discriminating observation is a changed capacity ranking between controller LPIPS and displacement error; identical rankings across repeated evaluations would weaken the claimed practical mismatch. e05e11e12e14e16e21
Check 2: Does visual context disambiguate the real-to-latent action map?
Reader-proposed check, not run: keep the pretrained world model and chosen latent capacity fixed, then compare controllers receiving correct previous-frame features, shuffled features from another viewpoint, or actions alone. Match training data, update budgets and parameter counts as closely as possible. Evaluate the same action sequences across camera viewpoints, separating clips with the arm initially visible from those with it outside the frame. Measure latent-target error, predicted motion and planned displacement. Correct context should outperform shuffled context on viewpoint changes if the camera-relative explanation is operative. Equal performance would weaken that explanation; failure only for initially invisible arms would instead expose missing scene information. e10e11e12e20
8.3 Reading coverage
Visual audit: Visually inspected the title/version/author block, all main-paper figures on the declared pages, Tables 1–2 and S1–S3, Appendix A's training and planning equations, the sampling diagnostic in Figure S1, and controller failures in Figures S12–S13. All six final crops were individually viewed and retain original pixels, labels and legends. Architecture and diagnostic arrows were checked against their captions and the method; the training-loss sign and plot/table discrepancies are disclosed. All eight text chunks, including Appendices A–H, were read. Figure S2–S11 images, earlier revisions, a separate venue edition, code and external supplements remain outside the visual/source pass. Every PDF page supporting a retained method, numerical, training, evaluation or proposed-reproduction detail is included above.
PDF pages inspected for this edition: 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 17, 18, 19, 20, 21, 34, 36, 37. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract; 1 Introduction; 2 Related works (pp. 1–4)
- 3 Problem setting; 4 Experimental details (pp. 4–6)
- 5 Performance of information regularizations; 6 What kind of actions do we learn? (pp. 6–9)
- 7 Leveraging latent action world models for planning; 8 Scaling models and data (pp. 9–12)
- 9 Limitations and future work; 10 Conclusion; 11 Acknowledgments (p. 12)
- References (pp. 13–16)
- Appendix A Training and evaluation protocols (pp. 17–18)
- Appendix B Sampling latent actions; C Detailed planning results (pp. 19–20)
- Appendix D Robot manipulation vs in-the-wild videos; E Qualitative impact of regularization strength (pp. 21–26)
- Appendix F Additional IDM rollouts; G Additional human action transfer results; H Qualitative performance of the controllers (pp. 27–37)
Outside the original text pass
- Identity/version scope: the title and all six authors match the catalog. The inspected artifact is arXiv:2601.05230v2, dated 20 January 2026. Earlier revisions and the catalog's ICML 2026 venue edition were not supplied; their differences cannot be verified.
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- Every supplied text chunk was read, including all appendix text and captions. Original images were visually inspected on PDF pages 1, 3–12, 17–21, 34, 36 and 37. Figures S2–S11 were read through their text and captions but their page images were not inspected.
- No external sources or code were inspected, and no training, planning or robot experiments were reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title block and arXiv margin
Title: Learning Latent Action World Models In The Wild. Authors: Quentin Garrido, Tushar Nagarajan, Basile Terver, Nicolas Ballas, Yann LeCun, Michael Rabbat. Affiliations: FAIR at Meta; Inria; NYU. Margin identifies arXiv:2601.05230v2 [cs.AI], 20 Jan 2026.
Go to primary source ↓e02PDF p. 1, Abstract and Introduction; p. 4, Section 3
Natural video lacks action annotations and a common embodiment; inverse dynamics must separate useful transition information from exogenous noise and future-state copying.
Go to primary source ↓e03PDF p. 3, Figure 2 and caption; p. 5, Section 4
Figure 2 routes frozen encoded observations to inverse dynamics and a latent-conditioned forward model. Section 4 identifies frozen V-JEPA 2-L, joint IDM/predictor training and frame-wise AdaLN-zero.
Go to primary source ↓e04PDF pp. 4–5, Section 3, Sparsity / Noise addition / Discretization
Sparse constraints include norm and VCM terms; the noisy formulation matches a standard Gaussian; quantization uses codebook resets. The methods regulate latent information.
Go to primary source ↓e05PDF pp. 5–6, Section 4; p. 17, Appendix A, Latent action training
Default ViT-L predictor, 128-dimensional actions, teacher forcing, YoutubeTemporal-1B clips, optimizer schedule and full regularized loss are specified. Appendix A reports 12 hours on 64 H100 GPUs and coefficient/codebook sweeps.
Go to primary source ↓e06PDF p. 6, Section 4, decoder paragraph; p. 17, Appendix A, Decoder training
A separate ViT-L decoder uses causal attention and L1 plus perceptual losses to visualize predictions and compute perceptual metrics.
Go to primary source ↓e07PDF p. 6, Figure 4 and Section 5
IDM rollouts use future frames. Prediction error serves as an operational capacity measure; sparse and noisy sweeps span wider ranges than the tested discrete codebooks.
Go to primary source ↓e08PDF p. 7, Figure 5a, Future leakage, Figure 6 and footnote 3; p. 8, Table 1
Scene cuts test whether a latent can copy the next frame. Table 1 high-capacity rows report sparse 0.20→0.50, noisy 0.21→0.54, discrete 0.29→0.68 LPIPS. Figure 6 retains some future information but reconstructs poorly.
Go to primary source ↓e09PDF p. 7, Figure 5b and transfer protocol; p. 8, Figure 7, Table 2 and discussion
Cycle consistency re-infers transferred actions and measures two-second LPIPS back on the source video. Table 2 reports original and transfer columns for Kinetics and RECON. Figure 7 demonstrates human-to-ball motion transfer and return.
Go to primary source ↓e10PDF p. 9, Figure 8, caption and Section 6 conclusion
Only the person near the source motion's position is animated in the two-person example, motivating the camera-relative, spatial-locality interpretation.
Go to primary source ↓e11PDF p. 9, Section 7 Controller training and Figure 9; pp. 17–18, Appendix A Controller training
The controller uses L2 latent targets, previous-frame representations and real actions. Appendix A details two self-attention blocks, cross-attention, a three-layer action MLP, projection, frame duplication and training settings.
Go to primary source ↓e12PDF p. 10, Section 7 Planning performance; p. 18, Appendix A DROID protocol, Equations (1)–(3)
Franka Panda recorded-video evaluation uses 64 episodes sampled from 16 validation videos, horizon 3 and four frames at 4 fps. CEM uses 300 candidates, 10 elites and 15 iterations; reported displacement compares cumulative planned and dataset translations.
Go to primary source ↓e13PDF p. 18, Appendix A, Planning protocol for RECON
RECON uses 120 candidates, one CEM iteration, straight-line trajectories, horizon 8 and 4 fps. Evaluation uses ATE and RPE, with further details referred to NWM.
Go to primary source ↓e14PDF p. 20, Appendix C, Table S1, both panels
Noisy Low/Mid/High controller errors are 0.13/0.11/0.12 and planning errors 0.49/0.11/0.10 m. Sparse/high is 0.13 m, discrete/high 0.14 m, V-JEPA 2-AC 0.15 m and V-JEPA 2 + WM 0.05 m. No uncertainty columns are present.
Go to primary source ↓e15PDF p. 20, Appendix C, Table S2, planning panel
Noisy/high reports ATE 1.40 and RPE 0.40; NoMaD 1.93 and 0.52; NWM 1.13 and 0.35. No uncertainty or unit labels are supplied in this table.
Go to primary source ↓e16PDF pp. 9–10, Section 7 rollout/planning discussion; p. 11, Figure 11
Controller LPIPS and planning error are plotted against in-the-wild prediction error for each regularizer. Intermediate capacity improves visual prediction, but planning minima need not coincide with rollout minima.
Go to primary source ↓e17PDF p. 11, Figure 12; p. 12, Section 8
Model size, training duration and data quantity affect IDM prediction. Planning benefits most clearly from longer training; model-size effects differ by regularizer and data quantity shows no significant planning trend.
Go to primary source ↓e18PDF p. 12, Section 9; p. 19, Appendix B and Figure S1; p. 20, Appendix B closing sentence
Authors identify static constraints, frozen representations and direct latent planning as limitations. Sampling uses codebooks, Gaussian priors or SGLD, with qualitative UMAP mismatch for less-constrained continuous latents.
Go to primary source ↓e19PDF p. 21, Appendix D, Table S3, planning rows and discussion
DROID mixture weights are 0, 10, 25, 50, 75, 90 and 100%. Noisy planning error changes from 0.14 at 0% to 0.09 at 10% and 0.06 at 75%/90%; sparse changes from 0.14 to 0.13 at 10%.
Go to primary source ↓e20PDF p. 34, Appendix H; p. 36, Figure S12; p. 37, Figure S13
Qualitative examples show failure to generate an initially invisible arm and static predictions when the controller has no visual-state representation.
Go to primary source ↓e21PDF p. 5, Noise addition equation and prose; p. 17, Appendix A full loss
The noise equation visibly prints a negative beta times KL divergence, while prose describes prior matching and Appendix A adds latent regularization to prediction error. The supplied paper does not resolve this optimization-sign ambiguity.
Go to primary source ↓e22PDF p. 11, Figure 11 lower-left and upper-right panels; p. 20, Tables S1–S2
The RECON Controller LPIPS panel spans roughly 0.38–0.48, while Table S2 controller values span 0.21–0.26. Figure 11's V-JEPA 2-AC line also lies below the 0.15 reported in Table S1. Exact results here follow the identified tables, without reconciling the displays.
Go to primary source ↓e23PDF p. 21, Table S3, Noisy / Controller LPIPS / 100% DROID column
The printed entry is 0.9, despite adjacent values near 0.10 and prose about improved rollouts. This apparent anomaly is preserved without guessing a correction.
Go to primary source ↓8.5 Primary sources
Learning Latent Action World Models In The Wild ↗
PDF · 14,387 extracted words
Source fingerprint
b005b53ccba93bcc68471d7c9fb88b1504849b8e63f11395d18d6535fec2bca4