Latent Action Pretraining Through World Modeling
1. Paper overview
In one sentence: LAWM uses a temporary world model to teach an imitation policy latent action chunks from video, trading extra pretraining and imperfect visual supervision for stronger downstream robot adaptation. e-identitye-overviewe-losse-libero90e-suitese-scalinge-budget
| At a glance | What to know |
|---|---|
| Research problem | Source description Robot action labels require costly teleoperation and encode a particular embodiment. LAWM asks whether instruction-conditioned human or robot video can teach transferable action priors before labeled downstream adaptation. Here, 'unlabeled' means lacking robot action labels: language instructions remain inputs. e-overviewe-latent |
| Core mechanism | Source description A two-stage, architecture-agnostic framework jointly learns latent action chunks and predictive dynamics, demonstrated with BAKU or Diffusion Policy and DreamerV3. e-overviewe-architecture |
| A key reported result | LIBERO four-suite manipulation: 92.70 / 96.30 / 94.20 / 88.00; reported average 92.80. Success rate (%). Human-video-pretrained Diffusion Policy, then supervised adaptation to Spatial/Object/Goal/Long; 10 tasks per suite and 50 demonstrations per task. The paper states three seeds and 30 evaluation trials per LIBERO task. Scratch Diffusion Policy: 81.87 average; villa-X: 90.10; UniVLA: 88.70. The reported average gain over scratch is 10.93 percentage points. External baselines come from other publications. LAWM does not lead every suite; even the bold Goal cell is below pi0's printed 95.00. e-protocole-suites |
| Reading caution | Source description Authors acknowledge costly joint training, pixel-loss shortcuts and single-view world modeling that omits proprioception and gripper state. Higher-fidelity video generation is not their objective. e-limitationse-qualitative |
Core contributions
- Source description
A two-stage, architecture-agnostic framework jointly learns latent action chunks and predictive dynamics, demonstrated with BAKU or Diffusion Policy and DreamerV3. e-overviewe-architecture
- Author claim
The authors attribute improved transfer to learning more universal dynamics instead of overfitting source-robot actions. Their comparisons support transfer benefits in the tested settings but do not isolate that explanation. e-protocole-libero90
Figure 2. Video prediction supplies the pretraining signal; robot supervision adapts the surviving policy. Original paper, p. 2 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow the left panel from the current image and language instruction into the blue imitation model. Its yellow outputs are the latent sequence z, which feeds the red world model alongside video frames. The input and target frame strips are shifted by one time step. On the right, the world model disappears and robot state joins the observation inputs; yellow outputs now represent explicit actions a. Read these arrows together with Eqs. (1)–(7): the policy produces latents from the present, while the world model's observation-conditioned posterior participates in training. Appendix B explains that the second camera and proprioceptive inputs are zeroed during pretraining. e-overviewe-latente-losse-finetunee-architecturee-head-conflict
What it supports. The transferred object is a pretrained imitation policy, not a deployed video planner. Future-frame reconstruction trains its latent outputs, and labeled downstream demonstrations give those outputs robot-action meaning. This supports the catalog's latent-action-pretraining placement without implying that joint optimization creates one unified controller.
Where the evidence stops. The diagram and Eq. (2) expose a frame sequence, while the prose describes rollout from one frame. The exact observation-conditioning schedule remains underspecified. Section 3.2 and Appendix B also disagree about final-layer reinitialization.
2. Motivation
2.1 The problem and the proposed response
Robot action labels require costly teleoperation and encode a particular embodiment. LAWM asks whether instruction-conditioned human or robot video can teach transferable action priors before labeled downstream adaptation. Here, 'unlabeled' means lacking robot action labels: language instructions remain inputs. e-overviewe-latent
2.2 What this reading follows
A human manipulation video contains visible changes but no robot motor commands. LAWM makes those changes a training target: a policy predicts latent action chunks, and a jointly trained world model uses them to predict future frames. The policy later learns explicit robot actions from demonstrations, while the world model is discarded. Read the architecture first, then separate the evidence from robot-video pretraining, human-video transfer and comparisons with external models. The appendix adds two useful boundaries: control performance saturates as image reconstruction improves, and fewer robot demonstrations can suffice only after additional video pretraining. This edition reviews the supplied June 2026 arXiv v2. e-identitye-overviewe-losse-libero90e-suitese-scalinge-budget
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Foundational work |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
This table preserves the labels recorded at reading time. The current major category is WAMs. View the current classification.
3.1 Evidence-based assessment
Supports the recorded classification
The recorded foundational-work/latent-action-pretraining placement is supported. LAWM contains distinct policy and world-model modules, and discards the latter for downstream action prediction. Joint training does not establish a unified One Model controller. Its latent chunks are predicted from present observations and language, rather than extracted by inverse dynamics; the catalog's architecture, prediction-paradigm and quadrant exclusions are reasonable for this pretraining contribution. e-overviewe-latente-finetunee-architecture
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Why a forward prediction loss can train an action-producing policy
Start with what the policy can observe: one image x_t and an instruction c. Equation (1) maps them to a sequence of latent actions, so future images are targets for learning rather than inputs used to extract the policy's actions. The world model then uses the sequence to predict visual transitions. Its RSSM maintains deterministic and stochastic state, with an observation-conditioned posterior and predictive prior. Equation (7) combines reconstruction error with KL regularization and updates both modules. This creates a learning signal for the policy even when the video contains no robot commands. Reader interpretation: the formulation does not identify a unique physical meaning for each latent coordinate. Seven latent dimensions matching a robot's action dimension do not by themselves establish seven interpretable motor controls. e-latente-losse-protocol
5.2 What actually transfers when the world model disappears
During pretraining the full imitation-policy architecture is present, but missing proprioception and the second camera are represented by zero inputs. Shared image-encoder weights let the policy reuse what it learned from one view when finetuning uses two. After pretraining, the world model is discarded and the policy is optimized against demonstrated robot-action chunks. BAKU uses transformer-based fusion, while Diffusion Policy conditions a denoising U-Net. The transfer is therefore in the policy's learned parameters and representations; online control does not depend on a generated-video search. One implementation choice cannot be settled from the source: Section 3.2 says to reinitialize the final latent-action layer, whereas Appendix B says matching seven-dimensional actions permit reuse of the whole policy. A reproduction must report which version of that choice it implements. e-architecturee-finetunee-head-conflicte-protocol
5.3 How far the evidence connects reconstruction to control
Reader analysis: the strongest practical evidence is the downstream policy comparison, not a visually plausible rollout or a large correlation coefficient. Figure 4 tests alternative pretraining strategies within a policy family; Table 2 adds a broader comparison whose external rows have different provenance. The capacity sweep then qualifies the proposed mechanism: as PSNR improves, control initially improves but saturates at the largest models. The CCA diagnostic also needs restraint. It chooses projections that maximize linear correlation and reports only the first coefficient; LAWM leads three action groups but trails villa-X on Remove, and no independent fitting/evaluation split is specified. Together these observations support useful transfer in the reported tasks. They leave open whether accurate action-dependent dynamics, generic visual representation learning, or additional training resources chiefly produce that benefit. e-libero90e-suitese-scalinge-ccae-qualitative
5.4 Training and inference
During training
ResNet-18 encodes images; CLIP encodes text. BAKU fuses tokens with a GPT-style transformer, while Diffusion Policy conditions a 1D U-Net. The full policy exists during pretraining, with second-view and proprioceptive inputs zeroed; those modalities become informative during finetuning. e-architecture
Default world-model size is 50M parameters; latents have seven dimensions. BAKU predicts chunks of 10 and Diffusion Policy 16. Table 5 specifies 50 pretraining epochs, 105 finetuning epochs, batch 256 and AdamW at 1e-4 learning rate with 1e-6 weight decay. e-protocole-hyperparameters
During inference
The adapted policy receives observations and predicts explicit action chunks; Diffusion Policy uses DDIM denoising. Learned video prediction is a training mechanism. Neither online imagined-trajectory scoring nor world-model planning is part of the described deployed policy. e-finetunee-architecturee-qualitative
The text does not specify how many predicted actions execute before refreshing observations, or give a control frequency and denoising-step count. Chunk prediction alone therefore does not establish the exact feedback schedule. e-finetunee-architecturee-hyperparameters
5.5 Implementation flow
- Predict an action prior from the present
The imitation policy maps the current image and instruction to n latent actions. It does not infer these actions by taking a future image as a policy input. Their usefulness is learned through future-frame prediction. e-latent
- Make the chunk predict visual transitions
DreamerV3's RSSM updates a deterministic state using the preceding state and latent action, predicts a categorical stochastic state and reconstructs a frame. Its observation-conditioned posterior regularizes the predictive prior. Policy and world-model image encoders are separate. e-latente-architecture
- Adapt to robot supervision
Remove the world model and optimize the policy with ground-truth action chunks. Appendix B lists all policy components as finetuned. Section 3.2 says the final layer is reinitialized; Appendix B instead retains it for matching seven-dimensional actions. The supplied text does not resolve this discrepancy. e-finetunee-head-conflict
6. Experiments & results
LAWM turns next-frame prediction into a pretraining signal for an imitation policy's action chunks. A jointly trained world model gives the policy a video-derived prior, then disappears before supervised robot adaptation. Human-video pretraining improves the reported LIBERO-suite average, but cross-publication comparisons and inconsistent implementation descriptions constrain interpretation.
6.1 Read the original evidence
Figure 4. Robot-video world modeling exceeds supervised pretraining; human-video transfer is competitive but does not always exceed it. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each panel as a comparison within one policy family. The blue bars are scratch and supervised BridgeData pretraining; the red bars use world modeling on Something-Something human videos or BridgeData robot videos. Black labels are success rates. The two red annotations compare against the two blue baselines: their numerical differences are percentage points despite the plotted percent symbols. BAKU's human-video result, 92.6, is below supervised pretraining at 92.7; Diffusion Policy's human-video result equals its supervised baseline at 91.9. The robot-video world-model variant has the highest printed score in both panels. Both vertical axes begin at 80 with a break. e-libero90e-protocol
What it supports. For BridgeData pretraining, BAKU rises from 92.7% with action supervision to 93.7% with world modeling, and Diffusion Policy rises from 91.9% to 93.0%. The evidence supports matching or improving supervised priors in these settings. It does not support a universal claim that human-video pretraining always wins.
Where the evidence stops. The broken axes magnify small differences. The paper states three seeds but shows no error bars here. Its explanation that dynamics transfer better than source-robot actions remains a hypothesis rather than an isolated mechanism test.
Table 1. Human-video pretraining improves a small physical robot evaluation, especially moving and stacking. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Compare the two rows column by column before reading the average. The pretrained policy improves pick-and-place task 1 from 80 to 90, leaves tasks 2 and 3 unchanged at 100 and 90, raises moving from 70 to 90 and stacking from 80 to 100. Appendix C identifies the numbered tasks as putting both cups on the plate, placing the blue cup in a bowl on a plate, and removing the pineapple from a pot to place it between cups. The experiment uses a Realman arm and labeled demonstrations for every downstream task after human-video pretraining. e-reale-real-setupe-protocole-architecture
What it supports. The five-task average increases by 10 percentage points, from 84% to 94%. This is evidence about executed physical manipulation, separate from the world model's reconstructed video. The task breakdown shows that the average improvement is concentrated in three tasks, with two already-strong tasks unchanged.
Where the evidence stops. The custom evaluation covers five tasks. Section 5 states three seeds and 10 real-world trials without explaining aggregation or uncertainty. Camera descriptions differ: Appendix B mentions a gripper view, while Appendix C specifies two fixed views.
Table 2. The highest reported average accompanies a large Long-suite gain, with heterogeneous external comparisons. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the bottom two rows, which compare the paper's Diffusion Policy with and without latent pretraining. The Long column changes from 70.50 to 88.00, and the reported average from 81.87 to 92.80. Then move upward: villa-X has a lower average of 90.10 but higher Spatial and Object scores, 97.50 and 97.00. Keep the pretraining-method column visible because these rows use different sources of supervision. The caption says external baseline scores come from their publications and LAPA was reproduced by UniVLA's authors. Inspect actual values instead of treating bold or underlined cells as an infallible ranking. e-suitese-protocol
What it supports. LAWM's 92.80% reported average is 10.93 percentage points above the scratch Diffusion Policy row and 2.70 above villa-X's reported average. Long-horizon manipulation contributes strongly to that improvement. The table supports a promising benchmark result, while leaving the contribution of differing data and training budgets unresolved.
Where the evidence stops. External rows are not a uniform rerun under matched resources. A formatting inconsistency also matters: LAWM's Goal score 94.20 is bold, although robot-action-pretrained pi0 reports 95.00. Preserve the numbers and disregard that implied ranking.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| LIBERO four-suite manipulation Human-video-pretrained Diffusion Policy, then supervised adaptation to Spatial/Object/Goal/Long; 10 tasks per suite and 50 demonstrations per task. The paper states three seeds and 30 evaluation trials per LIBERO task. | 92.70 / 96.30 / 94.20 / 88.00; reported average 92.80. Success rate (%) | Scratch Diffusion Policy: 81.87 average; villa-X: 90.10; UniVLA: 88.70. The reported average gain over scratch is 10.93 percentage points. External baselines come from other publications. LAWM does not lead every suite; even the bold Goal cell is below pi0's printed 95.00. e-protocole-suites |
| LIBERO-90 pretraining comparison Same downstream benchmark; compare scratch, supervised BridgeData and world-model pretraining. | BridgeData world modeling: BAKU 93.7; Diffusion Policy 93.0. Success rate (%) | Supervised BridgeData: 92.7/91.9; scratch: 91.4/85.7; human-video world modeling: 92.6/91.9. Robot-video world modeling exceeds supervised pretraining by 1.0/1.1 percentage points. Human-video pretraining slightly trails or matches supervised pretraining here. No uncertainty is shown. e-protocole-libero90 |
| Five physical tabletop tasks BAKU on a Realman arm; human-video pretraining and 50 teleoperated demonstrations per downstream task. Section 5 states three seeds and 10 real-world evaluation trials without detailing aggregation. | 94 average; pick-and-place 90/100/90, moving 90, stacking 100. Success rate (%) | Without latent pretraining: 84 average; 80/100/90, 70, 80. A 10-percentage-point average gain is physical execution evidence, restricted to this small custom setup. e-reale-real-setupe-protocol |
| World-model capacity on LIBERO-Long BAKU; human-video pretraining with different world-model sizes. | Success 86.7 at 1M, 90.0 at 25M, and a text-reported 91.0 at both 50M and 100M; PSNR 17.65 at 1M and 23.75 at 100M. Success rate (%) and reconstruction PSNR (dB) | BAKU without latent pretraining: 89.0%. Visual prediction improves with capacity, while reported control success saturates. This correlation does not isolate reconstruction quality as the cause. e-scaling |
| Reduced demonstrations on LIBERO-Long Human-video-pretrained Diffusion Policy; vary finetuning demonstrations per task. | 77.0 with 20 demonstrations; 46.0 with five and 88.0 with 50. Success rate (%) | Scratch policy with all 50 demonstrations: 70.5. Twenty demonstrations use 40% of the full robot-data budget and exceed this baseline by 6.5 percentage points; pretraining data and compute remain additional resources. e-budgete-compute |
| Latent/action alignment diagnostic CCA grouped by Put/Move/Remove/Take. | 0.9154 / 0.9599 / 0.9098 / 0.9541. First canonical correlation coefficient | villa-X: 0.7863 / 0.9174 / 0.9955 / 0.9270. LAWM leads three groups but loses Remove. Maximized linear correlation is neither causal identification nor evidence that every latent dimension is controllable; fitting/evaluation split is unspecified. e-cca |
| Successful task completion efficiency Diffusion Policy across four LIBERO suites; successful episodes. | 81. Reported average environment steps (lower is better) | Scratch: 102. Fewer interaction steps do not measure inference latency, and conditioning on success can change which episodes contribute. e-efficiency |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 5. A larger world model improves reconstruction, but the policy's reported success plateaus. Original paper, p. 12 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the three panels as linked views of the same capacity sweep. The left varies world-model size in millions and measures downstream BAKU success. The dashed blue line is the unpretrained BAKU baseline at 89%. The middle replaces the outcome with reconstruction PSNR in decibels. The right plots success against PSNR, connecting the visual-prediction and control measurements. The source reports success increasing from 86.7% at 1M to 90.0% at 25M and 91.0% at 50M. These are BAKU experiments on LIBERO-Long after Something-Something pretraining; they should not be substituted for the Diffusion Policy results in Table 2. e-scaling
What it supports. The smallest world model underperforms the baseline, whereas larger ones surpass it. PSNR rises from 17.65 dB at 1M to 23.75 dB at 100M, but the text reports 91.0% success at both 50M and 100M. Better image prediction therefore does not guarantee a proportional control improvement.
Where the evidence stops. The plotted 100M point sits slightly below 50M, while the text rounds both to 91.0%; no more precise control values are established. Capacity changes several factors together, and the sweep supplies no uncertainty bands or causal isolation.
Figure 6. Twenty labeled demonstrations after video pretraining exceed the full-budget scratch baseline. Original paper, p. 13 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Trace the red points as the number of expert demonstrations per task increases. The horizontal axis changes only the downstream labeled-data budget for the pretrained policy. The dashed line does not represent a scratch policy separately trained at every horizontal position: it is the single baseline trained with all 50 demonstrations, scoring 70.5%. The appendix reports 46.0% for five demonstrations, 77.0% for 20 and 88.0% for 50. The crossover at 20 demonstrations is therefore a comparison with a larger labeled-data budget, after the red model has already received human-video pretraining. Keep that extra source of experience in the interpretation. e-budgete-compute
What it supports. With 20 demonstrations, the pretrained policy uses 40% of the standard 50-demonstration budget and exceeds the scratch baseline by 6.5 percentage points. This is a concrete reduction in the robot-demonstration requirement for the reported LIBERO-Long setting, rather than evidence that the whole training process uses fewer resources.
Where the evidence stops. The figure does not show a scratch learning curve at matched smaller budgets, repeated subset variability or uncertainty bands. A matched-budget reproduction should include those controls and account separately for the additional video data and pretraining compute.
7. Analysis & limitations
7.1 What the evidence leaves open
Authors acknowledge costly joint training, pixel-loss shortcuts and single-view world modeling that omits proprioception and gripper state. Higher-fidelity video generation is not their objective. e-limitationse-qualitative
Tables omit dispersion despite the stated seeds. Appendix B describes a gripper-mounted second view, whereas the physical setup specifies two fixed views. Figure 2/Eq. (2) show frame sequences while the prose emphasizes rollout from one frame; the exact observation-conditioning schedule needs clarification. e-protocole-suitese-reale-architecturee-real-setupe-latent
7.2 Questions for discussion
- Does the transfer gain survive matching both pretraining compute and downstream demonstration budgets?
- Can an intervention on latent actions change predictions and downstream success independently of background reconstruction?
- How much do action-head reuse and the unspecified observation-conditioning schedule affect the reported results?
8. Reproducibility audit
8.1 Requirements and known gaps
Reproduction needs the actual BridgeData and 10% Something-Something subsets, language/video preprocessing and robot datasets. Appendix D reports one A100 and roughly 48 hours pretraining, 24 hours LIBERO-90 finetuning, three hours per suite and 20 minutes for the physical dataset; these are author-reported times. e-computee-real-setup
Resolve final-layer reuse, camera placement, KL weighting, video sampling, diffusion steps and software versions before claiming an exact reproduction. Proposed checks below can test matched-budget transfer and whether the reconstruction signal needs informative latent actions. e-head-conflicte-real-setupe-architecturee-losse-hyperparameterse-computee-budget
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Fill in the matched-budget demonstration curves
Reader-proposed, not executed: on LIBERO-Long, compare scratch Diffusion Policy and LAWM at 5, 10, 20 and 50 demonstrations per task, using identical sampled subsets and evaluation initializations. Repeat the subset sampling and report task-level and seed-level uncertainty. Fix downstream optimizer updates, architecture, camera inputs and action-head treatment; separately record the pretrained model's human-video and compute budget. Include the paper's 20-versus-50 comparison, but also test 20 versus 20. If LAWM's matched-budget advantage disappears across repeated subsets, the published crossover alone would not establish a robust low-data transfer advantage. e-budgete-protocole-hyperparameterse-head-conflicte-compute
Check 2: Test whether informative latent actions drive the benefit
Reader-proposed, not executed: hold the human-video subset, policy, 50M world-model capacity, training updates and downstream demonstrations fixed. Compare standard LAWM pretraining with a control that replaces the world model's predicted latent-action inputs with zeros; also evaluate a trained standard model with latent chunks shuffled across held-out clips. Measure held-out frame error and PSNR, then finetune the standard and zero-input policies under one explicitly documented action-head rule. If shuffling leaves predictions unchanged and the zero-input control matches downstream success, the claim that useful action-dependent prediction supplies the transferable prior would be weakened. Keep pixel quality and robot success as separate outcomes. e-latente-losse-scalinge-limitationse-head-conflicte-protocol
8.3 Reading coverage
Visual audit: Visually inspected the title/authors/version on p. 1; all Figures 1–9 and Tables 1–5 on their declared pages; Eqs. (1)–(8) and the method on pp. 2, 4–5; evaluation and CCA definitions on p. 6; results and limitations on pp. 7–8; and all appendix evidence on pp. 12–17, including architecture, hyperparameters, physical setup and compute. Six final original crops were inspected at their saved resolutions. The scaling crop was expanded to retain the top tick labels. Figure 2's frame-sequence arrows were checked against its caption and equations; unresolved conditioning and action-head descriptions remain explicit. Figure 4's percent annotations are interpreted as percentage-point differences. Table 2's misleading Goal boldface and Figure 5's rounded plateau are disclosed. Page 3 and reference pages 9–11 were read in the complete text but not visually rendered; no retained technical claim depends on them. Separate supplements and external publications remain uninspected.
PDF pages inspected for this edition: 1, 2, 4, 5, 6, 7, 8, 12, 13, 14, 15, 16, 17. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- 1 Introduction
- 2 Related Work (2.1 Imitation Learning; 2.2 World Models; 2.3 Latent Action Methods)
- 3 Method (3.1 Latent Action Pretraining; 3.2 Action Finetuning)
- 4 Experiments (4.1 Benchmarks and Environments; 4.2 Baselines and Experimental Setup)
- 5 Results & Discussion
- 6 Limitations
- 7 Conclusion
- References
- Appendix A Additional Experiments (A.1–A.4.2)
- Appendix B Implementation and Training Details
- Appendix C Benchmarks and Environments Details
- Appendix D Compute Resources
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- The extraction's figure-image omission was addressed by inspecting the supplied PDF: all nine figures and five tables were visually reviewed. All five text chunks, covering all 17 pages including references and appendices, were read.
- Identity/revision note: the inspected title and four authors match the catalog, but the supplied artifact is arXiv:2509.18428v2, dated 13 June 2026; the catalog records initial submission on 22 September 2025. The original edition was not supplied or compared, so changes between versions are not established.
- No code, external baseline publications or separate supplements were inspected; no experiments were reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e-identityPDF p. 1, title, author/affiliation block and arXiv margin
Title and authors match the catalog: Bahey Tharwat, Yara Nasser, Ali Abouzeid, Ian Reid. The artifact identifies arXiv:2509.18428v2 [cs.RO], 13 Jun 2026. Tharwat, Abouzeid and Reid list Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE; Nasser lists Alexandria University, Alexandria, Egypt.
Go to primary source ↓e-overviewPDF pp. 1–2, Abstract, Introduction and Figure 2/caption
LAWM pretrains an imitation policy and world model jointly from videos without robot action labels, then discards the world model and finetunes the policy using labeled robot demonstrations. The framework accepts different policy and world-model architectures.
Go to primary source ↓e-latentPDF p. 4, Section 3.1, Eqs. (1)–(6); p. 2, Figure 2
The policy predicts a latent action chunk from the current image and language instruction. The DreamerV3 RSSM uses deterministic recurrent and categorical stochastic states; its posterior receives the next observation and its prior predicts from recurrent state. Figure 2 and Eq. (2) show shifted frame sequences, while the prose describes sequential rollout from the initial frame.
Go to primary source ↓e-lossPDF p. 4, Section 3.1, Eq. (7)
Both policy and world-model parameters are trained using average squared next-frame reconstruction error plus a beta-weighted posterior/prior KL regularizer. A numerical beta and detailed KL implementation are not specified in this equation or the supplied implementation appendix.
Go to primary source ↓e-finetunePDF pp. 4–5, Section 3.2, Eq. (8)
Finetuning removes the world model, supplies image views, language and proprioception, and supervises action chunks with the architecture's action loss. Page 5 says the implementation reinitializes the final latent-action layer.
Go to primary source ↓e-architecturePDF pp. 15–16, Appendix B
Policies use ResNet-18 image encoding shared across camera views, CLIP text encoding and an MLP for proprioception. BAKU uses a GPT-style fusion transformer and MLP head; Diffusion Policy conditions a 1D U-Net and uses DDIM denoising. Pretraining zeros the second view and proprioceptive inputs. Policy and world model have separate image encoders.
Go to primary source ↓e-head-conflictPDF p. 5, Section 3.2, first paragraph; p. 16, Appendix B, finetuning paragraph
Section 3.2 says the final latent-action layer is reinitialized. Appendix B instead says the robot setup retains the complete pretrained policy because both latent and robot actions have seven dimensions. Appendix B says all listed policy components are finetuned.
Go to primary source ↓e-protocolPDF p. 6, Section 4.2 E1–E2 and Section 5 opening/R1; pp. 16–17, Appendix C
Experiments compare BAKU and Diffusion Policy. Defaults are a 50M DreamerV3, seven-dimensional latents, and chunks of 10 for BAKU or 16 for Diffusion Policy. The stated evaluation uses three random seeds, 30 trials per LIBERO task and 10 trials in the real-world setup. Each of the four LIBERO suites has 10 tasks and 50 demonstrations per task. R1 hypothesizes that supervised pretraining overfits source-robot actions while world modeling learns dynamics that transfer more broadly.
Go to primary source ↓e-libero90PDF p. 7, Figure 4, both panels and Section 5 LIBERO-90 paragraph
BAKU success rates are 91.4% from scratch, 92.7% with supervised BridgeData pretraining, 92.6% with human-video world modeling and 93.7% with BridgeData world modeling. Corresponding Diffusion Policy values are 85.7%, 91.9%, 91.9% and 93.0%. Figure 4 displays a broken vertical axis starting at 80 and provides no error bars.
Go to primary source ↓e-realPDF p. 7, Table 1 and real-world results paragraph; p. 5, Figure 3/caption
Human-video-pretrained BAKU reports pick-and-place success of 90/100/90%, moving 90% and stacking 100%, averaging 94%; the unpretrained baseline reports 80/100/90%, 70% and 80%, averaging 84%. Figure 3 illustrates cup placement and stacking execution.
Go to primary source ↓e-real-setupPDF p. 17, Appendix C, Real-world Setup
The physical platform is a 6-DoF Realman arm with a 1-DoF gripper. Five tasks each receive 50 demonstrations collected with a Meta Quest VR controller. This section says scenes use two fixed camera viewpoints, unlike the static-plus-gripper configuration described in Appendix B.
Go to primary source ↓e-suitesPDF p. 8, Table 2, caption and Diffusion Policy / villa-X / UniVLA / pi0 rows
LAWM Diffusion Policy scores Spatial 92.70, Object 96.30, Goal 94.20, Long 88.00 and average 92.80%. The no-pretraining row is 88.10/81.40/87.50/70.50, average 81.87%. villa-X averages 90.10 and UniVLA 88.70. External baseline scores are taken from publications; LAPA was reproduced by UniVLA authors. LAWM's Goal score is bold although the robot-action-pretrained pi0 row reports 95.00.
Go to primary source ↓e-ccaPDF p. 6, Section 4.2 E3, Eqs. (9)–(10); p. 8, R3 and Table 3
CCA maximizes linear correlation between latent and ground-truth action representations. First coefficients for Put/Move/Remove/Take are LAWM 0.9154/0.9599/0.9098/0.9541 versus villa-X 0.7863/0.9174/0.9955/0.9270. The source does not identify an independent fit/test split or sample counts for this diagnostic.
Go to primary source ↓e-limitationsPDF p. 8, Section 6
Authors identify joint-training cost, MSE's preference for pixel accuracy and background shortcuts, and single-view world modeling without proprioception or gripper states as limitations. Perceptual, optical-flow and masked-modeling objectives are suggested future directions.
Go to primary source ↓e-scalingPDF pp. 12–13, Appendix A.1 and Figure 5/caption
BAKU is pretrained on Something-Something v2 with different world-model sizes and finetuned on LIBERO-Long. Text reports PSNR rising from 17.65 dB at 1M parameters to 23.75 dB at 100M. Success rises from 86.7% at 1M to 90.0% at 25M and 91.0% at both 50M and 100M; baseline is 89.0%. The plotted 100M point is slightly below 50M, while prose rounds both to 91.0%. Only 10% of the human-video dataset is used.
Go to primary source ↓e-budgetPDF p. 13, Appendix A.2, Figure 6/caption
On LIBERO-Long, human-video-pretrained Diffusion Policy reports 46.0% success with five demonstrations, 77.0% with 20 and 88.0% with 50. The dashed baseline uses all 50 demonstrations and scores 70.5%; it is not a separate baseline at every demonstration count. No uncertainty bands are shown.
Go to primary source ↓e-efficiencyPDF p. 14, Appendix A.3, Table 4
Average environment steps for successful task completion are reported as 81 with latent pretraining and 102 from scratch. Per-suite values are 89/49/54/133 versus 94/62/82/170 for Spatial/Object/Goal/Long. The prose calls this a 20% reduction; this metric concerns successful episodes.
Go to primary source ↓e-qualitativePDF pp. 14–15, Appendix A.4, Figures 7–8/captions
Selected image pairs illustrate high cosine similarity of latent actions. Predicted and ground-truth drawer-opening/closing rollouts are shown; authors state high-fidelity video generation is not their objective because the world model is unused after pretraining.
Go to primary source ↓e-hyperparametersPDF p. 16, Table 5
Table 5 gives 50 pretraining epochs and 105 finetuning epochs, batch size 256, AdamW, learning rate 1e-4, weight decay 1e-6, gradient clipping 100, image resolution 128 and task embedding dimension 512. Latent/action dimension is seven; finetuning proprioception dimension is five. It switches off world modeling and enables the gripper camera and proprioception during finetuning.
Go to primary source ↓e-computePDF p. 17, Appendix D
Experiments use one A100 GPU by default and 10% of Something-Something v2. Reported approximate times are 48 hours for pretraining on either dataset, 24 hours for LIBERO-90 finetuning, three hours per LIBERO suite and 20 minutes for the custom real-world dataset. Hardware memory, software versions and exact video subset selection are not provided.
Go to primary source ↓8.5 Primary sources
Latent Action Pretraining Through World Modeling ↗
PDF · 7,425 extracted words
Source fingerprint
e3bffa022f6396d5ec5171e2fdaf9c79ee237a3065ddea9ebfe61c8e8ddfb20e