Latent Action Pretraining from Videos
1. Paper overview
In one sentence: LAPA learns discrete visual-change targets for policy pretraining, improving transfer from actionless videos while leaving robot grounding and precise grasping dependent on supervised adaptation. e01e02e03e04e05e10e12e13
| At a glance | What to know |
|---|---|
| Research problem | Source description Robot action labels constrain scalable pretraining. Human manipulation videos supply interactions but lack robot controls and differ in embodiment and viewpoint. LAPA learns motion-related targets from images; language descriptions remain necessary for policy pretraining, and action labels remain necessary for robot fine-tuning. e02e04e05 |
| Core mechanism | |
| A key reported result | Real-world manipulation across three generalization settings: LAPA: 50.09 partial-credit; 35.19 strict. Mean partial-credit score and strict task completion (%). Open-X pretraining; 450 Franka fine-tuning demonstrations; 54 rollouts covering knocking, covering and sink placement. OpenVLA: 43.87 partial-credit; 27.78 strict. Reported aggregate advantages are 6.22 and 7.41 percentage points, respectively. Partial credit rewards intermediate progress; neither result establishes superiority on every task. e07e10e12e13 |
| Reading caution |
Core contributions
Figure 2. Visual changes become policy targets before labeled robot demonstrations ground the output. Original paper, p. 3 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow the panels from left to right as training stages. The first encoder receives both the current image, x1, and future image, x2. Its codebook supplies z1, while the decoder also receives x1 to reconstruct the future. In the middle panel, the VLM receives only the current image and instruction and learns to predict the inferred latent label. The final panel changes the prediction target to robot action a1. Section 3.3 clarifies an easily missed implementation detail: the latent action head is discarded and a new robot action head is trained. The long bottom arrow therefore describes adaptation, not an inference-time chain through all three panels. e02e03e04e05
What it supports. The method moves action supervision out of pretraining while preserving it for robot deployment. This creates a route from human manipulation videos to a robot policy without requiring a human video's motion to be labeled directly in the robot's control coordinates. The useful transfer is carried by the adapted policy parameters.
Where the evidence stops. The compact diagram does not show freezing choices or head replacement. Sections 3.2–3.3 supply those details. The latent label is not directly executable, and the middle panel still requires a language description paired with the video.
2. Motivation
2.1 The problem and the proposed response
Robot action labels constrain scalable pretraining. Human manipulation videos supply interactions but lack robot controls and differ in embodiment and viewpoint. LAPA learns motion-related targets from images; language descriptions remain necessary for policy pretraining, and action labels remain necessary for robot fine-tuning. e02e04e05
2.2 What this reading follows
A video shows what changed without specifying the motor command that caused it. LAPA makes that gap useful: a quantizer compresses pairs of frames into discrete latent actions, and a vision-language model learns to predict those labels from an image and instruction. Robot demonstrations then adapt the policy to executable controls. Read the results with two distinctions in mind. Actionless pretraining still uses language descriptions and later robot supervision. Also, the paper's headline real-world score includes partial credit; strict completion and grasping expose a more uneven picture. The supplied ICLR 2025 version includes detailed appendix results that make these boundaries visible. e01e02e03e04e05e10e12e13
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Foundational work |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
This table preserves the labels recorded at reading time. The current major category is Components of WAMs. View the current classification.
3.1 Evidence-based assessment
Supports the recorded classification
The recorded foundational latent-action-pretraining classification is supported. The robot policy is a VLA trained using a separately learned inverse-dynamics representation. Its qualitative policy-plus-decoder rollout uses two models, while robot execution does not require world-model rollout planning. Calling the policy monolithic does not establish a One Model world/action architecture or joint future/action prediction; leaving the quadrant not applicable is appropriate. e02e03e05e20
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 1. Learn the change before learning the command
The first stage solves an inverse problem: two observations reveal a transition, and the encoder must summarize it in a small discrete representation. The decoder receives current-frame appearance directly, so the latent path can focus on information useful for reconstructing the change. That is an architectural incentive, not a guarantee of action semantics. In particular, a changing camera can explain a large visual difference without any change in the robot's intended command. Appendix E's camera-motion examples make this limitation concrete. The frame gap also defines what a token must summarize: Appendix F uses different physical intervals for robot and human videos. Reader interpretation: the learned action vocabulary is tied to the temporal and visual statistics of its training data, not a universal inventory of motor primitives. e03e18e19
Figure 8. The future frame supervises a compressed transition representation; current-frame information also reaches the decoder. Original paper, p. 15 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the two frame branches on the left. Each image is patch-embedded and spatially processed before temporal processing. Appendix A then differences the resulting continuous embeddings and quantizes that difference; the drawing abbreviates these operations as the causal-transformer and discretization path. Trace the horizontal bypass carefully: it originates from current-frame patches and crosses over the future-frame branch. The decoder combines this context with latent information. Equation (3), rather than an explicit marker in the schematic, specifies stop-gradient on current-frame patches and cross-attention with latent keys and values. Likewise, Equation (2) specifies NSVQ using both the continuous difference and its selected codebook embedding, information abbreviated by the NSVQ box. e03e19
What it supports. Reconstruction supplies the learning signal for latent actions without motor labels. The architecture gives the decoder appearance context directly and a compressed representation of change through the latent path. Reader interpretation: this encourages a motion-related representation, but reconstruction alone does not guarantee that every encoded change is robot-controllable.
Where the evidence stops. The figure omits explicit subtraction and stop-gradient markers and labels the output x2; Appendix A denotes the reconstruction as hat-x2. Use Eqs. (1)–(4) for these details. Human-video diagnostics also show camera-motion codes, limiting a purely motor interpretation.
5.2 2. Transfer a prediction task, then replace its output
Once the quantizer has learned a codebook, its encoder labels the pretraining videos. A separate vision-language model learns to predict those labels from the current image and instruction. The vision encoder stays frozen while the language model and latent head learn the prediction task. For deployment, the paper discards that head and trains a newly initialized robot-action head using labeled demonstrations, again updating the language model. The information carried forward is therefore broader than a fixed dictionary translating latent tokens into end-effector commands. This explains why the overview's final arrow should be read as a training transition. A separate optional demonstration keeps the latent policy and decoder together to generate imagined rollouts. That demonstration does not establish the performance of a world-model planner controlling the real robot. e03e04e05e20
5.3 3. Ask which part of the interaction improved
The real-world aggregate favors LAPA, but the evaluation deliberately rewards more than final completion. Appendix B assigns partial credit to reaching, grasping and intermediate placement progress, with different rubrics for each task. Table 16 shows that the aggregate advantage also holds under strict completion, which strengthens the transfer finding. Table 15 then supplies a crucial boundary: LAPA reaches the sink-placement target more often but completes that task less often than OpenVLA. Reader interpretation: a better visual or language-conditioned prior can coexist with insufficient grasp precision. The human-video table shows a related separation between completion, grasping and moving. These results motivate evaluating the entire interaction and its stages separately. Differences in backbone, optimization and task distributions still prevent attributing every gain solely to the latent representation. e06e10e12e13e14e24
5.4 Training and inference
During training
Both pretraining stages use the same video dataset. During latent pretraining and action fine-tuning, freeze the vision encoder and update the language model. Main non-Language-Table experiments use four latent tokens from an eight-entry vocabulary. e02e04e05e17
Table 3 lists Bridgev2 60k and Open-X 970k trajectories; real-world fine-tuning totals 450 demonstrations. Robot frame pairs span 0.6 seconds, versus 2.4 seconds for human videos. Dataset-count inconsistencies remain unresolved below. e07e18
During inference
The fine-tuned policy conditions on the image and instruction to output robot actions. Language Table additionally generates language directions before controls. The separate quantizer decoder is used for imagined closed-loop rollouts, not for the reported physical policy's action extraction. e05e08e20
The broccoli rollout uses predicted images as feedback and no action fine-tuning. It illustrates generative consistency only; selecting candidate plans and using test-time search are proposed extensions. e20
5.5 Implementation flow
- Compress a visual transition
Spatial and causal temporal transformers encode two frames. Their embedding difference is quantized into a token sequence. A CNN controls sequence length; NSVQ and early code replacement stabilize learning. The decoder reconstructs the future from current-frame patches and the latent representation. e03
- Learn to predict the latent label
The trained encoder labels videos as an inverse dynamics model. LWM-Chat-1M, a 7B VLM, predicts these tokens from the current observation and instruction through a new single-layer MLP head. Future frames supply training targets, not deployment inputs. e03e04e06
- Ground the policy in robot controls
Discard the latent head and initialize a robot action head. Fine-tune on equal-population action bins. This adapts policy parameters rather than retaining a fixed latent-to-robot decoder. e05
6. Experiments & results
LAPA turns visual changes into discrete training targets for a vision-language-action policy, then uses labeled robot demonstrations to learn executable actions. It improves transfer from actionless videos, including human videos, while retaining weaknesses in precise grasping. Its separate decoder supports qualitative imagined rollouts; the robot results evaluate the fine-tuned policy (e02, e05, e12, e13, e14, e20).
6.1 Read the original evidence
Table 2. Open-X LAPA has the highest reported partial-credit average in each generalization column. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the columns as three evaluation conditions, not three robot tasks. The first recombines objects seen during fine-tuning; the second uses objects unseen during fine-tuning; the third uses new instructions referring to seen objects. Each column averages knocking, covering and sink placement. The rows group pretraining sources, so comparisons within the Open-X pair are easier to interpret than comparisons across all rows at once. The rightmost AVG is the paper's task-specific partial-credit score expressed as a percentage. It should not be read as the fraction of trials fully completed. Appendix B describes the intermediate rewards, and Table 16 separately supplies strict completion. e07e10e11e06
What it supports. The reported Open-X comparison is 57.8 versus 46.2 for unseen combinations, 43.9 versus 42.1 for unseen objects, and 48.5 versus 43.4 for unseen instructions, favoring LAPA. Human-video LAPA's 34.0 average also exceeds Bridge OpenVLA's 30.8, while remaining below Bridge LAPA's 36.8.
Where the evidence stops. Unseen here is relative to fine-tuning; pretraining exposure is not ruled out. OpenVLA and LAPA also differ in backbone and fine-tuning recipe. The table provides no uncertainty by generalization column, so its rankings do not establish statistical significance.
Table 16. Strict completion supports the aggregate improvement while exposing a lower absolute success level. Original paper, p. 27 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Compare the two rows within a model before comparing columns. The first row averages task-dependent partial credit over the real-world trials; reaching or partially completing an interaction can contribute. The second counts complete task success. Then compare the adjacent Open-X OpenVLA and LAPA columns, keeping the pretraining source fixed. The total spans 54 trials per model across three tasks and three generalization conditions. This table gives greater numerical precision than the rounded main-text summary, explaining why the report uses 50.09 rather than 50.1. It also keeps human-video LAPA separate from models pretrained on robot videos. e10e12e13
What it supports. LAPA's Open-X advantage is 6.22 percentage points in partial credit, from 43.87 to 50.09, and 7.41 points in strict completion, from 27.78 to 35.19. The aggregate improvement therefore survives removing partial credit. It still does not imply that LAPA is better at every constituent manipulation skill.
Where the evidence stops. Table 15 gives the counterexample: Open-X LAPA completes fewer sink-placement trials than OpenVLA despite better reaching. Table 16 provides no uncertainty estimates; its two rows use different success definitions and should not be pooled.
Table 12. Human-video transfer improves average task success, with different patterns for intermediate skills. Original paper, p. 22 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the three horizontal blocks separately. The top reports completion for stacking green onto yellow, placing a carrot on a plate, a spoon on a towel, and an eggplant in a basket. The lower blocks report grasping and moving diagnostics; they are not additional independent tasks to average into the top score. All models here undergo robot-labeled SIMPLER fine-tuning after human-video pretraining. The LAPA (10%) column reduces human pretraining data, rather than the downstream label budget. Compare that column with full-data LAPA to examine data scaling, then compare LAPA with VPT and UniPi to examine different ways of exploiting the same video source. e07e08e14e17
What it supports. LAPA reaches 52.1% average completion versus 45.8% for VPT and 0.7% for UniPi. Full-data LAPA improves the average over its 10%-data version's 50.0%, but Spoon2Towel goes the other way, 50.0 versus 66.6%. Average grasping also favors VPT over LAPA, 68.7 versus 66.7%. Transfer quality is therefore skill-dependent.
Where the evidence stops. The table omits uncertainty across training seeds and does not establish a scaling law from two data fractions. Its 100 fine-tuning trajectories were collected from successful robot-policy rollouts; human-video pretraining does not remove this robot-supervision requirement.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Real-world manipulation across three generalization settings Open-X pretraining; 450 Franka fine-tuning demonstrations; 54 rollouts covering knocking, covering and sink placement. | LAPA: 50.09 partial-credit; 35.19 strict. Mean partial-credit score and strict task completion (%) | OpenVLA: 43.87 partial-credit; 27.78 strict. Reported aggregate advantages are 6.22 and 7.41 percentage points, respectively. Partial credit rewards intermediate progress; neither result establishes superiority on every task. e07e10e12e13 |
| Pick an object and place it in the sink Open-X models; same real-world protocol; 18 rollouts. | LAPA: 45.83 / 27.78 / 83.33. Partial-credit / strict / reaching rates (%) | OpenVLA: 54.17 / 50.00 / 66.67. Better reaching coexists with worse completion. Early grasping is the authors' failure explanation, not a controlled causal finding. e10e13 |
| Human-video transfer to SIMPLER Something-Something V2 pretraining; 100 labeled teacher rollouts for fine-tuning; four tasks, 24 evaluation rollouts each. | LAPA: 52.1. Average task success (%) | VPT 45.8; UniPi 0.7; Scratch 34.4. Bridge-pretrained LAPA separately reaches 57.3. Positive transfer requires downstream robot supervision. The human and Bridge figures use different pretraining sources. e07e08e14e15 |
| Human-video data scaling Same SIMPLER evaluation; full human pretraining set versus 10%. | Full-data LAPA 52.1; 10%-data LAPA 50.0. Average task success (%) | Spoon2Towel reverses the average: 50.0 versus 66.6. The average improves modestly; the table supplies no seed uncertainty and does not establish a general scaling law. e14e17 |
| Language Table in-domain transfer 181k simulated pretraining trajectories; 1k labeled fine-tuning trajectories; five tasks, 250 rollouts per seen/unseen setting. | LAPA seen 62.0±8.7; unseen 49.6±9.5. Average success ± reported StdErr (%) | Table 1: Scratch 15.6±9.2 / 15.2±8.3; ActionVLA 77.0±3.5 / 58.8±6.6. Latent pretraining helps but trails action supervision here. Cross-task unseen also favors VPT over LAPA, 60.8 versus 54.8. e07e08e09 |
| Temporal-window ablation Bridgev2 quantization and SIMPLER fine-tuning; default H=3 at 5 Hz. | H=3 and H=5 are similar; H=10 reduces performance. Average success trend in Figure 15(a) | The longest tested frame gap performs worse than the intermediate gaps. This supports sensitivity to large visual changes; a quantizer-capacity explanation remains an author hypothesis. e18 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 15. The frame gap and downstream label budget shape the benefit of latent pretraining. Original paper, p. 25 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. In panel (a), H is the gap in frames used to define a visual transition during quantization. Appendix F gives Bridgev2's default H=3 at 5 Hz, corresponding to 0.6 seconds. Compare the middle settings with H=10: the curve is nearly flat between three and five, then drops for the longest gap. In panel (b), the solid blue LAPA curve and dashed orange Scratch curve show what happens as the fine-tuning data scale increases. The paper evaluates both panels on SIMPLER. The horizontal data-scale axis shows 10, 30, 50 and 100 but does not explicitly print a unit; the caption and appendix identify it as fine-tuning data scale. e18
What it supports. The latent target's temporal granularity matters: intermediate frame gaps retain performance, while the longest tested gap degrades it. LAPA remains above Scratch at every plotted fine-tuning data scale. These observations support a useful pretrained prior over the tested settings, rather than an assumption that any frame separation produces equally learnable actions.
Where the evidence stops. The authors attribute the long-gap drop to large visual changes challenging a 300M-parameter quantizer, but this plot alone does not isolate capacity as the cause. No error bars appear, and Scratch's data-scaling curve is not monotonic.
7. Analysis & limitations
7.1 What the evidence leaves open
Fine-grained grasping and inference latency remain limitations. Past observations are omitted, and applications beyond manipulation are untested. Camera-motion latents need not correspond to controllable robot actions. e19e21
OpenVLA comparisons change backbone and fine-tuning recipe as well as supervision. The 30–40× efficiency claim compares 272 H100-hours with 21,500 A100-hours using an assumed hardware speed ratio; it is not a matched timing experiment or an itemized end-to-end cost. e06e16
Unresolved source differences: Table 3 gives 442k real Language Table trajectories versus 440k in prose, and 200k human videos versus 220K in Section 4.5. Table 1/Table 5 in-domain seen averages differ for UniPi (22.0/22.4), VPT (44.0/43.6), and ActionVLA (77.0/76.8). Figure 5(c)'s task setting is inconsistent between caption and prose. e07e09e17
The real-world test is small, unseen objects may have appeared during pretraining, and pairwise win summaries do not establish statistical significance. Task-frequency explanations use lexical matching rather than a controlled distribution intervention. e10e23e24
7.2 Questions for discussion
- Would additional quantizer capacity recover grasp precision without losing transfer? (e13, e18)
- How much of the OpenVLA comparison survives matched backbones, optimization and training seeds? (e06, e16)
8. Reproducibility audit
8.1 Requirements and known gaps
Reproduction needs the LWM backbone, a trained quantizer and codebook, instruction-paired videos, exact frame intervals, robot demonstrations and matched evaluation poses. SIMPLER's 100 successful teacher trajectories must preserve the stated pose separation. e03e06e07e08e18
The PDF lacks a complete LAPA optimizer/schedule, quantizer/CNN configuration, code replacement schedule and seed-level evaluation recipe. Do not reuse Appendix C's VPT optimizer as a LAPA specification. e22
Proposed checks: vary quantizer capacity while holding the temporal-window experiment fixed; independently recompute partial and strict outcomes from the real-world trial tables, then repeat with matched poses and multiple training seeds. e18e10e23
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Does quantizer capacity explain the long-window failure?
Reader-proposed experiment, not performed: reproduce the Bridgev2-to-SIMPLER window comparison at H=3 and H=10, then repeat it with a larger quantizer. Hold video samples, codebook vocabulary and token length, policy backbone, fine-tuning demonstrations, evaluation poses and optimization-step count constant; report the extra compute for the larger quantizer. Use multiple seeds and measure held-out reconstruction error alongside task completion and grasping. The authors' capacity explanation predicts that added capacity should preferentially recover performance at H=10. If reconstruction improves but downstream completion does not, the bottleneck is not explained by reconstruction capacity alone; temporal ambiguity or unsuitable latent targets remain alternatives. e03e07e08e18
Check 2: Does the real-world advantage survive an outcome audit and matched retraining?
Reader-proposed check, not performed: recompute Tables 13–16 from their per-rollout entries using the three Appendix B partial-credit rubrics, then independently count only full completions. Preserve the supplied object and instruction groups and flag any mismatch with the printed totals rather than adjusting a score to force agreement. Next compare latent pretraining with ground-truth-action pretraining using the same LWM backbone, fine-tuning recipe, demonstrations and paired initial poses over multiple training seeds. Report task-specific completion, reaching, partial scores and uncertainty. The transfer claim is weakened if the aggregate advantage disappears under these controls or is confined to intermediate progress; sink-placement failures are the discriminating case. e06e07e10e12e13e23
8.3 Reading coverage
Visual audit: Visually inspected the title/version/author page, complete method and experiment pages, all Figures 1–17 and Tables 1–16, including appendix equations, protocols, baseline settings, compute discussion and per-rollout outcomes. Every final crop was viewed after extraction: Figures 2, 8 and 15; Tables 2, 12 and 16. Figure 8's current-frame bypass, NSVQ path and omitted stop-gradient notation were cross-checked with Appendix A; Eq. (3)'s closing delimiter is repaired only in the report transcription. Figure 5(c)'s inconsistent task attribution is disclosed in the base report. Acknowledgment/reference pages 11–14 were read as text but not visually inspected. Separate supplements, code and experiments remain outside this reading.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract and Sections 1–6 (PDF pp. 1–10)
- Acknowledgments and references (PDF pp. 11–14)
- Appendix A: latent quantization details (PDF pp. 15–16)
- Appendices B–D: evaluation, baselines and result analysis (PDF pp. 16–18)
- Appendices E–F: latent analysis and ablations (PDF pp. 18–19, 24–25)
- Appendix G.1–G.3 and all detailed result tables (PDF pp. 19–27)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- Version/identity: reviewed arXiv:2410.11758v2, 15 May 2025, marked ICLR 2025. The title matches the catalog apart from capitalization. The PDF credits Yuchen Lin where the catalog says Bill Yuchen Lin; the remaining author list substantially agrees. No earlier revision was supplied or compared (e01).
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout. This acquisition limitation was addressed by inspecting all scientific figure/table pages and six original crops.
- Separate supplemental material availability has not been fully verified. No separate supplement was supplied.
- No code, external links or datasets were inspected; no training or robot experiments were reproduced. References were read as part of the supplied text, without opening cited works.
- Visual inspection covers PDF pp. 1–10 and 15–27. Acknowledgment/reference pages 11–14 were read in text only.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title block, author credits, affiliations and arXiv margin
The title is LATENT ACTION PRETRAINING FROM VIDEOS; the artifact identifies arXiv:2410.11758v2, 15 May 2025, and publication at ICLR 2025. Sixteen authors are credited, including Yuchen Lin. Affiliations are KAIST, University of Washington, Microsoft Research, NVIDIA, and Allen Institute for AI.
Go to primary source ↓e02PDF pp. 1–3, Abstract, Introduction, Figures 1–2 and Section 3 opening
LAPA addresses missing robot action labels and the human–robot distribution gap through sequential latent action quantization, latent prediction pretraining and supervised robot action fine-tuning. Both pretraining stages use the same dataset.
Go to primary source ↓e03PDF p. 4, Section 3.1; p. 15, Appendix A, Figure 8 and Eqs. (1)–(4); p. 16, paragraph following Table 3
The quantizer encodes current/future frames with spatial and causal temporal transformers, differences continuous embeddings, retrieves discrete codebook entries, uses NSVQ and reconstructs the future frame. Eq. (3) conditions the decoder on stop-gradient current-frame patch embeddings through cross-attention. The stated loss is squared L2 reconstruction; the encoder supplies latent labels.
Go to primary source ↓e04PDF p. 4, Section 3.2
A VLM predicts latent labels from the current image and language instruction using a separate single-layer MLP head of vocabulary size |C|. The vision encoder is frozen and the language model is trained. Raw videos require paired language instructions for this stage.
Go to primary source ↓e05PDF pp. 4–5, Section 3.3 and footnotes 2–3
Latent actions are not directly executable. Fine-tuning replaces the latent action head with a newly initialized robot action head, discretizes each action dimension into equal-population bins, freezes the vision encoder and updates the language model. Retaining a latent-to-action decoding head reportedly performed worse; no quantitative ablation is given there.
Go to primary source ↓e06PDF pp. 5–6, Section 4.2; p. 9, Section 4.6; p. 17, Appendix C
LAPA uses the 7B LWM-Chat-1M backbone. Scratch omits action pretraining; ActionVLA uses true action labels with the same backbone; VPT learns a supervised IDM; UniPi uses generated video and an IDM. OpenVLA uses a different VLM backbone. Appendix C describes OpenVLA LoRA fine-tuning at batch 32 and LAPA/ActionVLA batch 128 with real-world image augmentation.
Go to primary source ↓e07PDF p. 16, Table 3; p. 6, Section 4.3; p. 8, Section 4.5; p. 5, Section 4.1
Table 3 lists Language Table simulation 181k trajectories, real 442k, Bridgev2 60k, Open-X 970k and Something v2 200k. Text instead describes real Language Table as 440k and Something-Something V2 as 220K videos. Fine-tuning uses 1k or 7k Language Table trajectories, 100 SIMPLER trajectories, or 450 real-world trajectories (150 per task).
Go to primary source ↓e08PDF pp. 16–17, Appendix B, Language Table and SIMPLER setup; p. 19, Appendix G.1
Language Table uses 50 evaluation rollouts per task category, totaling 250 for each seen/unseen evaluation table. SIMPLER uses 25 successful teacher rollouts per task for fine-tuning, held out from evaluation object poses, and 24 evaluation rollouts per task. Language directions precede actions only in Language Table experiments.
Go to primary source ↓e09PDF p. 6, Table 1; p. 20, Tables 5–9; p. 21, Table 10
Table 1 reports in-domain seen/unseen LAPA success 62.0±8.7/49.6±9.5%, Scratch 15.6±9.2/15.2±8.3%, and ActionVLA 77.0±3.5/58.8±6.6% (StdErr). Cross-task unseen is 54.8% for LAPA versus 60.8% VPT. In-domain seen averages differ between Table 1 and Table 5: UniPi 22.0/22.4, VPT 44.0/43.6, ActionVLA 77.0/76.8.
Go to primary source ↓e10PDF p. 17, Appendix B, real-world setup and three task scoring rubrics; p. 5, Section 4.1
Franka evaluation comprises three tasks, three generalization settings and six trials per setting, totaling 54 rollouts per model. Identical object positions and matched training image resolution aid comparison. Knocking awards 0.5 for reaching; covering awards 0.33/0.66 for intermediate progress; pick-and-place awards 0.25/0.5/0.75 before complete placement. Unseen objects are unseen during fine-tuning, with possible pretraining exposure.
Go to primary source ↓e11PDF p. 7, Figure 3 and Table 2
Table 2 reports Open-X LAPA/OpenVLA partial-credit averages of 50.1/43.9%; unseen-combination scores 57.8/46.2, unseen-object scores 43.9/42.1 and unseen-instruction scores 48.5/43.4. Human-video LAPA averages 34.0, Bridge LAPA 36.8 and Bridge OpenVLA 30.8. Figure 3 favors OpenVLA on pick-and-place.
Go to primary source ↓e12PDF p. 27, Table 16, both total-success rows
Open-X LAPA/OpenVLA total partial-credit rates are 50.09/43.87%, and strict rates 35.19/27.78%. Human-video LAPA has 34.02% partial-credit and 20.37% strict success; Bridge OpenVLA has 30.76/12.96%. No uncertainty is tabulated.
Go to primary source ↓e13PDF p. 27, Table 15, success and reaching rows; pp. 7–8, Section 4.4
Open-X LAPA/OpenVLA pick-and-place partial-credit scores are 45.83/54.17%, strict completion 27.78/50.00%, and reaching 83.33/66.67%. The authors attribute many LAPA failures to early grasping and suggest limited grasp supervision as an explanation.
Go to primary source ↓e14PDF p. 8, Figure 4 and Section 4.5; p. 22, Table 12, success, grasping and moving blocks
Human-video pretraining gives SIMPLER average success 52.1% for LAPA, 45.8% for VPT, 0.7% for UniPi and 34.4% for Scratch (Figure 4). Table 12 gives 50.0% success for LAPA with 10% pretraining data. Full-data versus 10% LAPA averages are 66.7/56.2% grasping and 72.9/62.5% moving; Spoon2Towel success reverses the average trend, 50.0/66.6%.
Go to primary source ↓e15PDF p. 21, Table 11, average success and grasping rows
With Bridgev2 pretraining and 100 SIMPLER fine-tuning trajectories, average success is LAPA 57.3%, ActionVLA 63.5%, VPT 51.0%, UniPi 1.3%, Scratch 34.4%, and OpenVLA 36.4%. Grasping averages are LAPA 71.9% and ActionVLA 80.2%.
Go to primary source ↓e16PDF pp. 8–9, Section 4.6 and footnote 5
The paper reports 8 H100 GPUs for 34 hours, batch 128, totaling 272 H100-hours for LAPA Open-X pretraining, versus 21,500 A100-hours and batch 2048 for OpenVLA. Its 30–40× efficiency estimate assumes a 2–3× H100/A100 speed ratio. It reports one epoch sufficient for LAPA and contrasts latent space 8^4 with action space 256^7.
Go to primary source ↓e17PDF p. 9, Figure 5 and Section 5.1; p. 19, Appendix F; p. 25, Figure 16
Scaling plots vary quantizer size, data, latent sequence length and vocabulary. Main experiments outside Language Table use four tokens from vocabulary eight. Figure 5's caption calls its downstream setting SIMPLER, but the adjacent prose describes panel (c) as a Language Table exception; Appendix F and Figure 16 separately analyze Language Table. No fitted general scaling law is supplied.
Go to primary source ↓e18PDF p. 19, Appendix F; p. 25, Figure 15(a–b)
Robot frame pairs span 0.6 seconds and human pairs 2.4 seconds. Bridgev2 is 5 Hz with default H=3. Figure 15(a) shows similar success at H=3 and H=5, with deterioration at H=10; panel (b) places LAPA above Scratch at all four tested fine-tuning data scales. The authors suggest a 300M-parameter quantizer has difficulty with large visual deltas.
Go to primary source ↓e19PDF pp. 9–10, Section 5.2 and Figure 6; pp. 18–19, Appendix E; p. 24, Figures 12–13; p. 25, Figure 14
Decoder-conditioned examples suggest related movement semantics across embodiments, Language Table latent clusters correlate with 2D actions, and human-video latents capture camera movement as well as hand motion. These are qualitative representation diagnostics, not executed cross-embodiment controls.
Go to primary source ↓e20PDF p. 10, Figure 7 and Section 5.2, closed-loop rollout discussion
A pretrained policy without action fine-tuning predicts latent actions; the quantizer decoder generates successive images for the broccoli-removal instruction. The broccoli disappears in the generated sequence. Multiple-plan selection and task-and-motion planning are proposed extensions, not evaluated robot-control algorithms.
Go to primary source ↓e21PDF p. 10, Section 6; p. 4, footnote 1
Stated limitations include fine-grained grasping, real-time inference latency and untested applications beyond manipulation. Quantization omits past observations owing to computational constraints. Larger latent spaces and a faster hierarchical action head are suggested future work.
Go to primary source ↓e22PDF pp. 4–5, Sections 3.1–3.3; p. 15, Appendix A; p. 17, Appendix C; p. 19, Appendix F
The source specifies NSVQ, early code replacement, a CNN controlling latent length, freezing choices, some batch sizes and frame intervals, but does not supply a complete LAPA optimization schedule, quantizer layer/CNN configuration, replacement schedule or seed-level evaluation protocol. The VPT optimizer details in Appendix C should not be reassigned to LAPA.
Go to primary source ↓e23PDF p. 22, Table 13; p. 26, Table 14; p. 27, Tables 15–16; p. 18, Appendix D; p. 24, Figure 11
Per-rollout real-world partial scores and aggregate strict outcomes are provided. The pairwise comparison reports 31.5% LAPA wins, 16.7% OpenVLA wins and 51.9% ties, or 65.4% LAPA wins when ties are excluded; these summaries are not a formal significance test.
Go to primary source ↓e24PDF p. 18, Appendix D and Table 4
Task-frequency analysis uses lexical matching in Bridgev2, Open-X and Something-Something V2. The authors hypothesize that differences in knocking, covering and pick-and-place frequency help explain transfer, while acknowledging the need for semantic rather than lexical task analysis.
Go to primary source ↓8.5 Primary sources
LATENT ACTION PRETRAINING FROM VIDEOS ↗
PDF · 13,544 extracted words
Source fingerprint
efab4f379e1a30ef7be36c87885c2c509d880ee9578f4cfdcc20d9321c93dce9