LatBot: Distilling Universal Latent Actions for Vision-Language-Action Models
1. Paper overview
In one sentence: LatBot transfers a video teacher's scene and motion representations into a current-view policy, trading substantial annotated pretraining for stronger downstream manipulation. e03e04e05e06e07e12e14e16
| At a glance | What to know |
|---|---|
| Research problem | |
| Core mechanism | |
| A key reported result | SIMPLER WidowX Visual Matching: 87.5%; task rates 95.8%, 87.5%, 83.3%, 83.3%. Average task success rate (%). Four simulated tasks: spoon, carrot, block stacking and eggplant placement. MemoryVLA 71.9%; π0.5 55.2%; UniVLA 47.9%. The reported average exceeds π0.5 by 32.3 percentage points. Eggplant remains below the 100% of MemoryVLA and SpatialVLA; simulation success does not establish physical WidowX deployment. e12 |
| Reading caution | Reader analysis The study reports no dedicated limitations section or confidence intervals. Ten-trial Franka estimates are coarse. Better 50-shot than full-data results on some tasks motivate an author hypothesis about redundant demonstrations, but no controlled redundancy experiment verifies it. e14e15 |
Core contributions
Figure 1. LatBot adds physical-action decoding to the visual reconstruction bottleneck. Original paper, p. 2 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each panel from bottom to top. The dashed input box contains observed frames; the side path carries the initial frame directly to the decoder. In LatBot's rightmost panel, text enters the encoder, and the latent bottleneck contains yellow scene tokens and blue motion tokens. The two outgoing arrows name distinct predictions: the future visual frame V_{t+k} and inter-frame actions A_{t:t+k}. Sections 3.1–3.2 explain that the token groups summarize scene evolution and active motion, and exchange information within the unified decoder. This figure depicts latent-action learning from a demonstrated sequence; it is not the complete deployed policy. e03e04e08
What it supports. The design constrains a shared representation through both visual change and physical action. This gives the authors a route for importing action information from robot and human demonstrations into transferable latents. The initial-frame skip connection also shows that the decoder receives context beyond the bottleneck.
Where the evidence stops. Separate token labels express the intended decomposition, not a demonstrated guarantee of independent causal factors. Internal bidirectional fusion is described in the text but is not resolved into layer-level blocks in this graphic.
2. Motivation
2.1 The problem and the proposed response
Visual reconstruction can reward appearance changes that do not describe executable motion. The authors seek transferable latents that retain task-relevant robot dynamics while separating environmental variation, using physical action annotations alongside video and language. e02e03e04
2.2 What this reading follows
A video can reveal how a manipulation unfolded, while a robot choosing its next action has only its current observations. LatBot uses that asymmetry as supervision. A teacher compresses instructed video sequences into scene and motion representations and learns to decode both a future image and physical actions. A student then learns to anticipate those representations from current observations, with an additional subtask-language objective. Finally, an action expert turns its features into commands. The visual evidence below follows this transfer and tests its payoff: strong simulated results and promising few-shot Franka performance, accompanied by small real-world evaluation samples and an attention diagnostic that selects its most concentrated maps. e03e04e05e06e07e12e14e16
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Not assigned |
| Architecture | Not assigned |
| Prediction paradigm | Not assigned |
| Quadrant | Not assigned |
This table preserves the labels recorded at reading time. The current major category is VLA. View the current classification.
3.1 Evidence-based assessment
Insufficient evidence to decide
The catalog is entirely unassigned, so there is no positive quadrant claim to confirm. Architecturally this is a separate teacher-to-policy distillation pipeline. The teacher infers latents from observed transitions and jointly decodes images/actions during pretraining; the deployed VLA uses an action expert. That supports latent inverse-dynamics-style representation learning, but does not establish a single inference-time world/action model merely because a unified decoder is trained. e03e04e05e06e10
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Use demonstrated futures to teach a motion-sensitive bottleneck
The teacher observes an instructed manipulation sequence and compresses it into scene and motion token groups. The decoder then asks those groups to support two tasks: reconstructing a future frame and predicting the intervening physical actions. This matters because image changes alone can represent background variation, while action labels describe movements in an embodiment's control-related space. The appendix makes the supervision concrete: robot and human data share a 44-dimensional action representation containing arm/hand changes and fingertip deltas, alongside a 46-dimensional state representation. Human videos therefore enter with pose annotations rather than as wholly unlabeled footage. The scene/motion split and bidirectional decoder jointly implement the authors' intended structure, but neither the token names nor the reconstruction objective mathematically guarantees disentanglement. e03e04e07e08e09
Figure 2. A frozen temporal teacher trains features that a current-observation action policy can use. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start at the left teacher, trained on robot and human-hand videos. Its dashed arrow points toward the student's scene/motion targets and explicitly says gradient detach; Section 6.3 confirms that teacher parameters stay frozen. Above the central VLM are camera views and a task instruction. Equation 3 formalizes the student using the first frame, whereas Equation 2 gives the teacher a temporal sequence. Below the VLM, alignment supervises latent features and reasoning preservation supervises a subtask description. Follow the large rightward arrow to the action expert and its output chunks; input noise is drawn above that expert. e05e06e09e10
What it supports. The transfer separates learning from an observed future and choosing actions from present information. Equations 4–6 train the student to match teacher features while retaining subtask-language prediction. Subsequent downstream fine-tuning includes the action expert, making the latent representation useful for actual robot commands.
Where the evidence stops. The diagram summarizes multiple training stages. Its rightward path does not specify a control rate or action horizon, and neither the displayed subtask nor the preservation loss proves that broad language reasoning survives distillation.
5.2 Distill what the teacher knows into what the policy can observe
The teacher's access to multiple frames would be inappropriate as the information requirement for selecting an action before that future occurs. LatBot therefore changes the information available to the student. Equation 2 gives the teacher the sequence; Equation 3 gives the student the first frame and instruction. A detached teacher target is matched with feature error and KL divergence, whose printed direction is student distribution to teacher distribution. A second objective predicts subtask tokens, weighted by 0.5, to support instruction understanding. During distillation the teacher is frozen and the student is updated. Afterwards, downstream fine-tuning includes the vision encoder, VLM and action expert. The crucial distinction is between predicting a useful latent representation and issuing an executable action: the paper explicitly requires the latter adaptation step. e05e06e10
5.3 Separate evidence for task success from evidence for the explanation
Reader analysis: the result tables establish a stronger empirical case for manipulation performance than for every proposed explanation of that performance. WidowX reaches 87.5% average success, LIBERO reaches 98.0%, and all five ten-shot Franka tasks have nonzero success. These are different protocols with different statistical limits, so they should remain separate comparisons. Table 5 supports combining DLA and UAD, but does not establish that the resulting latents are universally embodiment-invariant. Figure 4 adds qualitative attention examples, selected for minimum entropy rather than measured across a held-out localization benchmark. Even the physical results include substantial failures and nonmonotonic effects of demonstration count. A useful reproduction should therefore test both task completion and a controlled prediction about the claimed mechanism. e12e13e14e15e16
5.4 Training and inference
During training
Pretraining uses one million episodes. The appendix specifies OXE, DROID, AgiBoT and EgoDex, expanding the main-text list. A 44-dimensional action space combines two 7-dimensional arm/hand blocks and 30 fingertip deltas; the corresponding state has 46 dimensions with quaternion orientations. EgoDex's original 48-dimensional pose vector is converted into these representations. e07e08
Pretraining lasts 14 days and distillation 7 days on 16 A100 40GB GPUs. The appendix freezes the LAM vision encoder during pretraining, then freezes the teacher and updates all student parameters during distillation. It specifies global batches 512 and 256 respectively; the main text gives 512 without this stage distinction. e07e10
During inference
The distilled VLM conditions on current observations and language; the action expert produces robot action chunks. Figure 2 depicts multiple camera views, whereas Equation 3 formalizes first-frame conditioning. No future-video teacher input is needed in that student formulation, and no inference-time search over decoded futures is described. e05e06
The supplied description leaves action-chunk length, execution frequency and feedback/replanning cadence unspecified, so this report does not infer a particular closed-loop controller from the π0.5 backbone name. e05e06e09e10
5.5 Implementation flow
- Summarize a demonstrated transition
The VLM encoder uses [CP_SCE] and [CP_MOT] to summarize scene evolution and active motion. Defaults are InternVL3.5-2B, 16-frame sequences, and 64 representations in each group. These are learned summaries of observed transitions, not simulator states. e03e07e09
- Ground the representation through joint decoding
A SANA-1.6B-initialized decoder reconstructs the future frame and predicts inter-frame actions. Layer-wise bidirectional scene/motion interactions let scene dynamics inform actions and motion features inform image reconstruction. Figure 1 also supplies the initial frame directly to the decoder. e04e09
- Transfer video knowledge to a current-frame student
The pretrained LAM extracts teacher latents from multiple frames. The VLM inside π0.5 predicts student latents from the first frame and the instruction. Feature and distribution alignment transfer that privileged temporal information; the teacher remains frozen. e05e07e10
- Adapt to executable actions
After distillation, downstream training jointly updates the vision encoder, VLM and action expert. End-effector translation/rotation use MSE and gripper state uses binary cross-entropy. Latent prediction alone is explicitly insufficient for execution. e06e10
6. Experiments & results
LatBot learns scene and motion latents from language-conditioned robot and human manipulation videos, supervises them through future-image and physical-action decoding, then distills them into a VLA student. An action expert converts the student's features into executable commands. The strongest evidence concerns downstream manipulation success; universal physical understanding and preserved reasoning remain broader interpretations of these results (e03–e06, e12–e16).
6.1 Read the original evidence
Table 2. The largest WidowX gain includes a substantial improvement on block stacking. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Each column is a different simulated manipulation task under WidowX Visual Matching; the final column averages the four task rates. Follow the shaded Ours row across before comparing averages. LatBot reports 95.8% for the spoon, 87.5% for the carrot, and 83.3% for both stacking and eggplant. The π0.5 row reports 79.2%, 58.3%, 16.7% and 66.7%, making stacking a particularly large difference in the displayed comparison. MemoryVLA is the strongest listed average comparator at 71.9%. The bold 100% entries in the eggplant column belong to other methods, so aggregate leadership is not dominance on every task. e12
What it supports. The reported 87.5% average exceeds π0.5's 55.2% by 32.3 percentage points and MemoryVLA's 71.9% by 15.6 points. The table supports stronger simulated task completion under this protocol, with an especially visible stacking improvement; it does not measure video quality or physical WidowX execution.
Where the evidence stops. The table supplies no uncertainty estimates, and Section 4.2 does not give SIMPLER trial counts. A common pretraining-data and compute budget is not documented for all baselines, limiting causal attribution of the gain to distillation alone.
Table 4. Few-shot successes span all five physical tasks, with substantial remaining failures. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the nested header carefully: 10, 50 and full denote training demonstrations, not evaluation trials. Full means all 100 demonstrations for each task. The task sequence is cup selection, block insertion, oven closing, brush dipping and pot placement, as defined by Figure 3 and Section 4.4 on the same page. Compare methods within the same subcolumn. At ten demonstrations, LatBot scores 60%, 20%, 80%, 20% and 60%. The rightmost 63.3% averages all five tasks across all three demonstration budgets. Section 4.4 states that each task/setting is evaluated over ten trials and that training resources are matched between methods at a given budget. e14
What it supports. LatBot achieves nonzero success in every ten-shot task, whereas π0.5 has zero success on cup selection, insertion and dipping at that budget. The table also contains nonmonotonic learning curves: for cup selection and insertion, 50 demonstrations outperform the full dataset in this run.
Where the evidence stops. Ten trials make these rates coarse and do not establish reliable deployment. The authors' redundancy explanation for the nonmonotonic results remains a hypothesis; the source provides neither repeated-subset uncertainty nor a controlled data-quality study.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| SIMPLER Google robot manipulation Four tasks, reported separately under Visual Matching and Variant Aggregation. | 78.0% Visual Matching; 70.1% Variant Aggregation. Average task success rate (%) | MemoryVLA: 77.2% and 67.7%; π0: 52.7% and 46.0%. Highest listed aggregate in each protocol. LatBot's drawer-and-apple task is 33.3% in both, below MemoryVLA's 47.2% and 58.3%. No uncertainty is supplied. e11 |
| SIMPLER WidowX Visual Matching Four simulated tasks: spoon, carrot, block stacking and eggplant placement. | 87.5%; task rates 95.8%, 87.5%, 83.3%, 83.3%. Average task success rate (%) | MemoryVLA 71.9%; π0.5 55.2%; UniVLA 47.9%. The reported average exceeds π0.5 by 32.3 percentage points. Eggplant remains below the 100% of MemoryVLA and SpatialVLA; simulation success does not establish physical WidowX deployment. e12 |
| LIBERO four-suite manipulation Mixed-suite fine-tuning for 60,000 steps, batch 64; each suite has 10 tasks and 50 demonstrations per task. Evaluation: 500 rollouts per suite. | 98.0% average; Goal 98.6%, Object 98.8%, Spatial 99.0%, Long 95.4%. Success rate (%) | π0.5: 96.9% average and 92.4% Long; MemoryVLA: 96.5% average. Long improves by 3.0 percentage points over π0.5. This is mixed-dataset adaptation, not evidence of sequential lifelong learning or zero-shot transfer. e13 |
| Franka five-task few-shot manipulation Franka Research 3; 100 demonstrations per task, training subsets of 10, 50 or all 100; 10 evaluation trials per task/setting. | With 10 demonstrations: cup 60%, block insertion 20%, oven closing 80%, brush dipping 20%, pot placement 60%. Per-task success rate (%) | π0.5 at the same budget: 0%, 0%, 60%, 0%, 20%. Every task has some successes, not perfect reliability. Table 4's 63.3% combines all tasks and demonstration budgets; it is not a ten-shot average. e14 |
| SIMPLER component ablation Table 5's four WidowX tasks; UniVLA-style ablation baseline and DLA/UAD variants. | Baseline 51.0%; DLA only 59.4%; UAD only 61.5%; both 87.5%. Average task success rate (%) | Combined design is 36.5 percentage points above this baseline. Supports complementary contributions in the reported setup. The UniVLA-style ablation row differs from Table 2's UniVLA result; the paper does not establish identical baseline configurations. e15e12 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 5. Combining the decoder and representation changes yields the strongest ablation result. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use the two checkmark columns as a small factorial design. The UniVLA-style row has neither component. Ours-v1 adds DLA, the decoupled scene/motion representation; Ours-v2 adds UAD, the unified decoder; Ours-v3 uses both. Their average success rates are 51.0%, 59.4%, 61.5% and 87.5%, respectively. Reading the task columns matters: either component alone lowers stacking relative to this baseline, from 45.8% to 37.5% or 41.7%, while the combined system reaches 83.3%. Thus the reported aggregate improvement should not be paraphrased as every component independently helping every task. This ablation's UniVLA-style row is distinct from Table 2's named UniVLA comparison. e15e12e05e08
What it supports. The joint configuration is 36.5 percentage points above its ablation baseline and is stronger than either single-component variant. This supports the usefulness of combining representation structure with unified decoding in the reported setup, while the task-level pattern suggests that their effects depend on one another.
Where the evidence stops. Exact parameter and compute matching for these variants is not detailed. The table does not independently test the reasoning loss, human-data contribution, or latent invariance; success gains alone cannot identify which physical information the tokens encode.
Figure 4. Selected attention examples suggest improved grounding after distillation. Original paper, p. 13 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read down each task column: the RGB row identifies the target, the middle row shows π0.5, and the bottom row shows LatBot. Red rectangles in the top row mark task targets; the lower rectangles mark highlighted LatBot responses. The caption identifies attention from the final text token to visual features. Section 7 on pages 13–14 supplies an essential selection rule: threshold each head/layer map at its mean, find 8-connected components, compute entropy from their relative areas, and display the map with the lowest entropy. These are selected examples of concentrated attention, rather than maps averaged over the model. e14e16
What it supports. The displayed LatBot maps place strong responses on task-relevant regions such as the cup, oven handle and pan, consistent with the authors' qualitative grounding claim. This offers a diagnostic connection between latent-action distillation and visual attention, separate from the numerical evidence of robot task success.
Where the evidence stops. The original has no quantitative color scale; the task-two target box meets the image boundary. Minimum-entropy selection favors concentrated examples and supplies neither an aggregate localization score nor evidence that these attention patterns caused successful control.
7. Analysis & limitations
7.1 What the evidence leaves open
The study reports no dedicated limitations section or confidence intervals. Ten-trial Franka estimates are coarse. Better 50-shot than full-data results on some tasks motivate an author hypothesis about redundant demonstrations, but no controlled redundancy experiment verifies it. e14e15
Figure 4 selects the lowest spatial-entropy attention map across all heads and layers after thresholding and connected-component analysis. These examples support a qualitative grounding observation, not a population-level localization score or proof that attention caused successful control. e16
Neither preserved reasoning nor universal disentanglement is directly isolated: Table 5 varies DLA/UAD, but does not ablate the reasoning loss, human-data contribution, or invariance to controlled background changes. Joint reconstruction/action supervision does not by itself establish physical accuracy. e04e05e08e15
7.2 Questions for discussion
- Would controlled background perturbations selectively affect scene tokens while preserving motion predictions and executed success?
- Does the reasoning-preservation objective improve held-out instruction understanding when latent alignment, data and compute are held constant?
8. Reproducibility audit
8.1 Requirements and known gaps
A faithful run requires the Table 6 mixture, InternVL3.5-2B/SANA-1.6B initialization, π0.5 student, action/state conversion and quantile normalization. Appendix optimization uses AdamW β1=0.9, β2=0.95, learning rate 10^-4, 2,000-step warm-up and cosine decay to 2.5×10^-6; distillation reuses that schedule. e08e09e10
Resolve the batch-size discrepancy before reproduction. Missing details include KL distribution construction/temperature, cross-model latent projection, complete LAM loss weighting, subtask target construction, and handling of absent arm/fingertip channels. Exact benchmark versions, seeds and SIMPLER trial counts are also unreported in the supplied description. e04e05e07e08e09e10e11e12
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Test whether correctly aligned physical supervision drives transfer
Reader-proposed check, not performed: keep the video sequences, instructions, initialization, token count, decoder capacity and optimization budget fixed. Compare normal image-plus-action pretraining with a visual-only loss control and with action targets temporally permuted within each embodiment, preserving target dimensions and marginal ranges. Distill and fine-tune all three with the same data and settings, then evaluate each WidowX task over repeated seeds and matched rollout counts. Report action-prediction error, image reconstruction and task success separately. The physical-grounding account predicts that correct action alignment should improve transfer beyond both controls; similar success after permutation would weaken that explanation even if image reconstruction remains strong. Missing-channel handling and the distillation batch discrepancy must be resolved consistently first. e04e07e08e10e12e15
Check 2: Check whether the 50-shot advantage survives resampling
Reader-proposed check, not performed: repeat the Franka cup and insertion comparisons using several nested 10/50/100-demonstration subsets and multiple fine-tuning seeds. Within each budget, match methods' updates and resources as in Section 4.4, and evaluate identical randomized initial-condition sets with more than the reported ten trials. Report raw successes and binomial intervals instead of a single rounded rate. Separately construct a 100-demonstration set with redundant trajectories removed or replaced under a prespecified rule. If the original nonmonotonic result mainly reflects subset or rollout noise, its ordering should vary across repeats; a stable reversal improved by redundancy control would better support the authors' data-quality hypothesis. e14e10
8.3 Reading coverage
Visual audit: The title/byline/version and affiliations were visually checked on PDF p. 1. Figure 1 and its labels on p. 2, Equation 1 on p. 3, Figure 2 and Equations 2–5 on p. 4, Equation 6 and Tables 1–2 on p. 5, Table 3 and implementation/evaluation text on p. 6, Figure 3/Table 4 on p. 7, and Table 5 on p. 8 were inspected. Embedded supplement pages 12–14 were inspected for Table 6, action/state definitions, model and optimizer settings, Figure 4, and the attention-selection procedure/Equation 7. All six final original crops were actually viewed; Figure 2's crop was widened for complete borders. Figure 1's arrows agree with its caption and Section 3.2; Figure 2's gradient-detach marker agrees with Section 6.3. The separately supplied label transcription was treated only as derived reading support and checked against original pages 2 and 13. Reference pages 9–11 were read as text; external material was not inspected.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 12, 13, 14. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract and Section 1: Introduction (PDF pp. 1–2)
- Section 2: Related Work (PDF pp. 2–3)
- Sections 3.1–3.3: latent representation, unified decoding, distillation and action-expert fine-tuning (PDF pp. 3–5)
- Sections 4.1–4.5: implementation, SIMPLER, LIBERO, Franka and components analysis (PDF pp. 5–8)
- Section 5: Conclusion (PDF p. 8)
- Acknowledgement and references (PDF pp. 9–11)
- Embedded Supplementary Material, Sections 6.1–6.3 and 7 (PDF pp. 12–14; supplement pp. 1–3)
- All eight supplied text chunks, including the separately marked derived figure-label transcription
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Poppler layout text is faithfully retained without manual repairs. Mathematical symbols, table ordering, figure labels and ligatures require inspection against the original PDF; successful extraction is not scientific reading.
- Separate supplemental material availability has not been fully verified.
- The prior full-text HTML source at https://arxiv.org/html/2511.23034 and its extracted text are preserved unchanged in the private recovery history. The complete original PDF is also preserved and remains available for visual reading; no replacement characters have been edited or guessed.
- The original PDF is complete and readable. Its unchanged raw extraction retains 38 replacement characters in diagram labels. A separately marked, independently image-reviewed TeX transcription covers every affected label in the derived reading text. These annotations are reading support, not author prose or restoration of the original font encoding. Original PDF pages 2, 13 remain required visual evidence and must be inspected and included in the report visual audit.
- Identity/version: the inspected title page matches the catalog title and all four authors, and prints arXiv:2511.23034v1, 28 Nov 2025. No other revision or edition was compared.
- The embedded three-page supplement was read in full. Separate external supplements, linked project materials, code and demonstration videos were not inspected; no experiments were reproduced.
- All scientific figures and tables were visually inspected. Reference-only PDF pages 9–11 were read as text, without page-image inspection.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title, complete byline, affiliations and arXiv margin stamp
Title and authors match the catalog: Zuolei Li, Xingyu Gao, Xiaofan Wang, Jianlong Fu; version 2511.23034v1, 28 Nov 2025. Affiliations are Institute of Microelectronics, Chinese Academy of Sciences; University of Chinese Academy of Sciences; Microsoft Research.
Go to primary source ↓e02PDF pp. 1–3, Abstract, Section 1 and Section 3.1 motivation
The authors argue that appearance-based reconstruction, insufficient temporal context and entangled environmental changes weaken action transfer.
Go to primary source ↓e03PDF p. 3, Section 3.1, Equation (1) and token-definition paragraph
Language-conditioned multi-frame encoding produces scene Zsce and motion Zmot using the added [CP_SCE] and [CP_MOT] tokens.
Go to primary source ↓e04PDF p. 2, Figure 1(c) and caption; PDF pp. 3–4, Section 3.2
The unified decoder conditions on scene/motion latents, receives the initial frame in Figure 1, and outputs Vt+k and At:t+k. The text specifies SANA initialization and layer-wise bidirectional fusion, without a complete mathematical pretraining loss or detailed fusion block.
Go to primary source ↓e05PDF p. 4, Figure 2, Section 3.3 and Equations (2)–(5)
The video teacher supplies detached latent targets; the student sees the first frame and instruction. Alignment combines squared error with KL(p(student)||p(teacher)); next-token subtask supervision is intended to preserve reasoning. Figure 2 separately depicts current views and an action expert with input noise.
Go to primary source ↓e06PDF p. 5, Equation (6) and Action Expert Finetuning paragraphs
Distillation weights reasoning preservation by 0.5. A subsequent action expert makes latents executable, with MSE for translation/rotation and BCE for gripper state.
Go to primary source ↓e07PDF pp. 5–6, Section 4.1 Implementation Details
The main text lists one million OXE/AgiBoT/EgoDex episodes; 16 A100 40GB GPUs for 14 and 7 days; FSDP, per-GPU batch 16, accumulation 2, global batch 512; 16 frames, 64 scene plus 64 action representations, π0.5 backbone.
Go to primary source ↓e08PDF p. 12 (supplement p. 1), Section 6.1 and Table 6
The appendix includes DROID as well as OXE, AgiBoT and EgoDex. It defines EgoDex's 48-dimensional pose annotations, a 44-dimensional unified action space, a 46-dimensional state space, and Euler/quaternion conversion. Table 6 gives dataset mixture ratios, including EgoDex 11.1% and AgiBoT-α 6.3%.
Go to primary source ↓e09PDF pp. 12–13 (supplement pp. 1–2), Section 6.2
Default LAM encoder is InternVL3.5-2B; decoder is initialized from SANA-1.6B. The instruction template requests scene evolution and action dynamics. The student uses the VLM from π0.5.
Go to primary source ↓e10PDF p. 13 (supplement p. 2), Section 6.3
Pretraining freezes the vision encoder; distillation freezes the teacher and updates the student. Appendix batches are 512 for pretraining and 256 for distillation, with 16 GPUs, accumulation 2 and per-GPU batches 16/8. AdamW and learning-rate schedule are specified. Downstream fine-tuning updates vision encoder, VLM and action expert and uses quantile normalization.
Go to primary source ↓e11PDF p. 5, Table 1, Ours/MemoryVLA/π0 rows, Avg and drawer-apple columns; PDF p. 6, Section 4.2
Google simulation averages are 78.0/70.1 for Ours, 77.2/67.7 for MemoryVLA and 52.7/46.0 for π0 under Visual Matching/Variant Aggregation. MemoryVLA wins drawer-apple 47.2/58.3 versus 33.3/33.3.
Go to primary source ↓e12PDF p. 5, Table 2, complete WidowX comparison; PDF p. 6, Section 4.2
Visual Matching Ours task rates are 95.8/87.5/83.3/83.3, average 87.5. MemoryVLA averages 71.9, π0.5 55.2 and UniVLA 47.9. SpatialVLA and MemoryVLA each score 100 on eggplant.
Go to primary source ↓e13PDF p. 6, Table 3, Ours/π0.5/MemoryVLA rows, and Section 4.3
LIBERO Ours rates are Goal 98.6, Object 98.8, Spatial 99.0, Long 95.4, average 98.0. π0.5 averages 96.9 with Long 92.4. Text specifies mixed-dataset 60k-step fine-tuning, batch 64, and 500 rollouts per suite.
Go to primary source ↓e14PDF p. 7, Figure 3, Table 4 and Section 4.4; PDF p. 8, continuation of real-world discussion
Five Franka tasks use primary/top/wrist views, 100 demonstrations each and 10/50/full subsets, with 10 trials per task/setting. Ten-shot Ours rates are 60/20/80/20/60 versus π0.5 0/0/60/0/20. Ours row's combined Avg is 63.3. Nonmonotonic cup/block results are discussed through an untested redundancy explanation.
Go to primary source ↓e15PDF p. 8, Table 5, all rows and Section 4.5; Section 5
The UniVLA-style, DLA-only, UAD-only and combined variants average 51.0, 59.4, 61.5 and 87.5. On stacking, the single-component variants score 37.5/41.7 versus baseline 45.8, while both score 83.3. The conclusion proposes future scaling to more complex and diverse tasks.
Go to primary source ↓e16PDF p. 13 (supplement p. 2), Figure 4 and Section 7; PDF p. 14 (supplement p. 3), Equation (7) and selection procedure
Attention uses the final input text token's query over visual features. Per-head maps are thresholded at their mean, analyzed with 8-connected components and ranked by component-area entropy; the minimum-entropy map across heads/layers is displayed. Figure 4 provides qualitative examples, without a numerical localization benchmark or color scale.
Go to primary source ↓8.5 Primary sources
LatBot: Distilling Universal Latent Actions for Vision-Language-Action Models ↗
PDF · 9,358 extracted words
Source fingerprint
521b27062c87a67339af820ff00623469709cdc2958e4933bdabfce5b7b9f99f