Towards Predictive, Aligned, and Scalable Robot Learning
1. Paper overview
In one sentence: Lumo-2 uses progressive multimodal alignment to make latent dynamics useful for fast action generation, while its experiments leave the separate effects of supervision, history and online prediction unresolved. e03e04e06e08e14e15e18
| At a glance | What to know |
|---|---|
| Research problem | Source description An accurate action autoencoder need not produce tokens that a vision–language model can predict effectively. Lumo-2 addresses this representation mismatch and perceptual aliasing: the same image can occur before or after pouring, requiring different actions. Its proposed solution combines predictive latents, semantic alignment and short-term action history. e03e04 |
| Core mechanism | |
| A key reported result | Collect Eggs from the Conveyor Belt: 100.00 Reported success rate (%). Stage 3-based VLA versus VLWA fine-tuning ablation; Table 3 provides no separate split or trial count. VLA: 92.00; VLWA gains 8.00 percentage points. Supports the combined active-dynamics fine-tuning pathway on this temporal task; no uncertainty is reported. e14 |
| Reading caution | Reader analysis The future-information probe reaches 90% task accuracy versus 43% from an initial frame and 94% from five frames. Because its latents are computed from observed frame pairs, this supports information retention, not the accuracy of futures predicted online. e12 |
Core contributions
Figure 3. Stage 1 teaches the visual transition representation to carry information useful for reconstructing robot actions. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow the upper blue pathway first. Current and future observations become features through the frozen vision encoder; the Vision IDM extracts the visual latent, and the Vision FDM predicts future features using the current feature shortcut. Then follow the lower green pathway from an action chunk through its encoder to its reconstructed output. The large blue arrow sends visual context into latent fusion at the action decoder. The dashed green return arrow denotes gradient-induced physical grounding, not a second observation stream or an executed action. Equations (4)–(5) confirm this direction: dynamics guide action reconstruction, and the reconstruction objective regularizes dynamics. e03e04e05
What it supports. The mechanism couples two otherwise incomplete representations. Visual transitions provide environmental context, while action reconstruction pressures the visual latent to retain motion-relevant information. This supports the paper's alignment design; it does not by itself demonstrate that nuisance variation is eliminated or that the learned latent is a calibrated physical simulator.
Where the evidence stops. This is a training diagram with observed future frames and demonstrated actions. At deployment, the transformer must predict useful dynamics from current context. Reading the upper branch as an online source of ground-truth futures would misrepresent the policy.
2. Motivation
2.1 The problem and the proposed response
An accurate action autoencoder need not produce tokens that a vision–language model can predict effectively. Lumo-2 addresses this representation mismatch and perceptual aliasing: the same image can occur before or after pouring, requiring different actions. Its proposed solution combines predictive latents, semantic alignment and short-term action history. e03e04
2.2 What this reading follows
A robot needs more than a compact encoding of motor commands: it needs commands that can be inferred from what it sees, what it is asked to do, and which phase of the task it has reached. Lumo-2 tackles this through a staged construction of the action representation. Visual transitions first ground action tokens, semantic tasks then connect them to vision and language, and a shared transformer learns to predict dynamics and actions. The six visuals below follow that construction into decoding efficiency and real-robot evaluation. Read the results with their units and protocols intact: representation probes, reconstruction errors and partial task progress answer different questions. e03e04e06e08e14e15e18
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | WAMs |
| Architecture | One Model |
| Prediction paradigm | IDM |
| Quadrant | Q2 · One Model × IDM |
3.1 Evidence-based assessment
Supports the recorded classification
One Model is explicitly supported: both dynamics and action distributions share one multimodal transformer. The IDM label is defensible for future-conditioned action inference, but needs qualification: Stage 1's Vision IDM extracts a transition latent rather than directly producing robot actions, while deployment jointly models dynamics and action tokens with BAR. This is more specific than merely observing joint training or an auxiliary future loss. e03e04e08e15
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Learn a transition representation before asking a policy to predict it
Stage 1 can inspect both ends of a demonstrated transition. Its Vision IDM extracts a compact description of what changes; the forward branch checks whether that description supports future-feature reconstruction. Visual predictability alone is insufficient for the intended use, because lighting or background variation can change features without prescribing robot motion. Injecting the dynamics representation into the action decoder adds a second requirement: it must help reconstruct demonstrated actions. The reader's interpretation is that this creates a useful bridge between environment-level change and embodiment-specific motion detail. Stage 3 subsequently has the harder job of predicting that bridge from current observations, language and history. This distinction explains why the future-derived task-inference probe is informative about representation content but cannot substitute for measuring online dynamics prediction. e03e04e05e10e12
Figure 4. Stage 2 adds semantic constraints while retaining a reconstruction path for precise actions. Original paper, p. 10 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start on the right: the original action representation A enters a semantic module, producing the new semantic representation shown as A superscript s. Its codebook feeds vocabulary-based prediction and action decoding, while the action projector connects it to the shared embedding space. On the left, visual features and encoded world dynamics enter through their respective interfaces. The lower panels specify what makes alignment more than reconstruction: describe behavior, predict instruction-conditioned actions, predict dynamics from actions, and discriminate compatible from perturbed or mismatched actions. The optional dynamics input under action prediction also matters: these training tasks do not all require the same modality combination. e06e19
What it supports. Action tokens are trained to be both decodable control signals and informative partners for vision and language. In the separate semantic probe, Stage 2 action features plus DINO achieve 95.00% accuracy versus 92.85% with Stage 1 features. That association supports semantic enrichment, while leaving the contribution of each Stage 2 objective unresolved.
Where the evidence stops. The VLM projector carries a frozen snowflake here, but pp. 10–11 describe feature projectors as trainable. The decoder is explicitly updated on p. 10 yet omitted from p. 11's abbreviated list. The precise freeze configuration requires clarification.
5.2 Separate action fidelity from action predictability
A tokenizer can reconstruct a trajectory accurately while presenting a difficult target to a policy. Stage 2 therefore constrains action representations through language descriptions, guided generation, cross-modal prediction and compatibility losses, while preserving reconstruction. The relevant question becomes whether these semantics help the policy choose the right trajectory from context. The paper offers complementary evidence: improved action-category classification, mixed reconstruction changes, and lower decoded policy errors over training when using Stage 2 tokens. These are not interchangeable outcomes. In particular, the Stage 2 policy can have higher final token-prediction loss yet lower decoded errors, because its codebook differs. As a reader interpretation, token loss should therefore be assessed alongside physical errors and executed behavior. The reconstruction table's gripper and torso-position regressions make that joint evaluation especially necessary. e06e18e19e20
5.3 Use temporal context and block generation for different reasons
The historical-action queue and BAR solve distinct problems. The queue supplies information about what the robot recently attempted, reducing ambiguity when visually similar scenes belong to different phases. BAR shortens the sequential decoding path by predicting groups of semantic action codes together. Its block-causal mask preserves temporal ordering while permitting within-block attention, producing the measured latency reduction. Neither mechanism alone proves successful anticipatory control. The task suite therefore matters, but its protocol must accompany its charts: static tasks can use VLA fine-tuning, temporal or physical prediction tasks use VLWA, and some long tasks are evaluated as separately initialized subtasks. A reader should interpret Table 3 as evidence for the combined dynamics-enabled pathway and Figure 20 as a broader system comparison, rather than treating both as isolated tests of prediction. e08e09e14e15e16
5.4 Training and inference
During training
The corpus includes roughly 53 million VLM samples plus internet/egocentric video and heterogeneous robot demonstrations. Stage 1 sends action-free human video through its vision branch and robot demonstrations through both branches. e05e07
Stages 1 and 2 run 30,000 and 12,000 steps on 64 H100 GPUs, with AdamW base learning rate 1e-4. Stage 3 runs 120,000 steps on 160 H100s at initial learning rate 1e-5, packs up to 8,192 tokens and drops historical action inputs with probability 0.5. Robot formats mix VLA, VLW and VLWA; VLM examples retain standard autoregression. e05e06e08e10
During inference
The deployable route predicts future-relevant latents from current context; ground-truth future frames belong to representation learning and probes. BAR reduces sequential decoding and the VQ decoder yields continuous actions. This is learned action generation, with no pixel-video rollout or explicit search over candidate plans described. e03e04e08e21
Historical actions feed subsequent predictions, but the paper does not specify a complete execution scheduler, exact queue length or how much of each chunk executes before observation refresh. Action history alone also does not directly measure whether a commanded action succeeded. e08e15
5.5 Implementation flow
- Ground visual transitions in actions
Stage 1 freezes DINOv2 and learns a vision inverse-dynamics representation from current/future features. A forward model reconstructs future features with MSE. In parallel, eight action groups covering torso, arms and grippers are compressed from 32 to four temporal positions, quantized and reconstructed with L1 loss. Attention fuses visual context into action decoding, allowing action loss to shape the dynamics representation. e04e05
- Give action tokens semantic meaning
Stage 2 freezes the original action encoder/codebook and VLM core, adding a semantic module, semantic codebook and projectors. Reconstruction, behavior description, instruction-guided action prediction, cross-modal prediction, contrast and discrimination align the modalities. The action decoder is explicitly updated on p. 10. Figure 4's frozen VLM-projector marker conflicts with the prose's trainable-projector description; the exact freeze configuration remains unresolved. e06
- Predict and decode in context
Stage 3 co-trains a single transformer on VLM, video and robot data. The policy projects latent dynamics, generates action tokens and detokenizes them into control trajectories. A queue of semantic historical actions supplies phase information; BAR predicts semantic token blocks with intra-block attention. Post-training is task-adaptive: static tasks may use action-only VLA, while temporal/physical prediction tasks use VLWA. e03e08e10e15
6. Experiments & results
Lumo-2 trains a Qwen3.5-4B-based robot policy to predict compact visual dynamics before generating semantic action tokens. Progressive dynamics–action and vision–language alignment structures its action representation, while historical actions disambiguate execution phases. Real-robot results favor the system, but partial-progress metrics and bundled training changes limit causal conclusions.
6.1 Read the original evidence
Figure 8(b). BAR shortens the sequential decoding path and reduces measured inference latency. Original paper, p. 16 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the action-generation row before the total. Standard AR performs 32 passes at 6.00 ms each; BAR performs four passes at 11.24 ms each. Each BAR pass is slower, but far fewer passes reduce the stage total from 192.00 to 44.96 ms. The scheduler/runtime row also shrinks, whereas image encoding, observation processing and detokenization remain similar. The lower bars summarize the end-to-end comparison. The inspected companion panel, Figure 8(a), shows the underlying block-causal attention pattern: tokens may attend within a block, and dynamics/action blocks remain temporally ordered. The hardware and precision details continue onto p. 17. e08e09
What it supports. The reported end-to-end speedup is 2.71×, from 253.66 ms to 93.53 ms on one RTX 5090 with vLLM, bfloat16 and FlashAttention-2. The breakdown supports reducing sequential generation as the main efficiency mechanism, rather than attributing the gain to faster visual encoding.
Where the evidence stops. These are inference timings, not wall-clock durations of robot tasks or evidence that AR and BAR have identical control quality. The comparison does not report latency distributions, repeated-run uncertainty or a complete robot execution schedule.
Figure 20. Lumo-2 leads across the six reported categories, measured as task progress. Original paper, p. 31 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use the color legend to compare the three policies within each category. The categories are defined by the primary challenge of each task, even when a task also demands other abilities. Notice the vertical axis: it says Task Progress, although the paper caption calls the values success rates. Section 4.3.2 explains why this matters. Single-step trials are binary, but multi-step trials earn fractional credit for completed meta-subtasks. Some starred tasks in Table 5 instead start each evaluated subtask from an in-domain configuration. Category averages therefore combine different evaluation structures; they are not a uniform measure of uninterrupted whole-task completion. e13e15e16
What it supports. The consistent ordering across all six categories supports broad real-robot gains over the two evaluated baselines. The largest visual separation is in long-horizon execution. Exact category values are not printed on the bars, so this reading retains their qualitative ordering instead of inventing precise percentages.
Where the evidence stops. Each task uses ten layouts repeated twice per model, and the chart provides no uncertainty intervals. Task-adaptive fine-tuning, history choices and different baseline training histories prevent attributing the entire system-level advantage specifically to latent world modeling.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Collect Eggs from the Conveyor Belt Stage 3-based VLA versus VLWA fine-tuning ablation; Table 3 provides no separate split or trial count. | 100.00 Reported success rate (%) | VLA: 92.00; VLWA gains 8.00 percentage points. Supports the combined active-dynamics fine-tuning pathway on this temporal task; no uncertainty is reported. e14 |
| Place Cubes on the Rotating Rack Same Table 3 fine-tuning ablation on a predictive temporal task. | 81.67 Reported success rate (%) | VLA: 74.17; gain 7.50 percentage points. The comparison changes dynamics supervision/projection together, so it does not isolate inference-time latent use. e14 |
| Real-world manipulation across six categories 22 Astribot S1 tasks; ten layouts repeated twice per model; starred tasks use separate subtask starts. | Lumo-2 leads in every displayed category; exact bar values are unlabeled. Normalized task progress; binary only for single-step tasks | Higher bars than pi0.5 and Fast-WAM, with especially large long-horizon separation. Evidence concerns executed robot behavior, but category scores mix partial completion and different task demands. e15e16 |
| Autoregressive decoding latency Single RTX 5090, vLLM, bfloat16 and FlashAttention-2. | 93.53 End-to-end inference latency (ms; lower is better) | Standard AR: 253.66; reported speedup 2.71×. Action generation falls from 192.00 to 44.96 ms; this is inference timing, not complete physical task duration. e09 |
| Embodied vision–language understanding Table 2, common lmms-eval visual and greedy-decoding protocol. | Stage 3: VSIBench 48.56; MindCube 54.93. Reported benchmark score | Qwen3.5-4B: 21.73/39.95; Lumo-2 VLM: 52.34/66.08. Co-training retains useful capabilities while reducing some scores relative to VLM-only training. e11 |
| Action semantic classification 18 categories, 7,200 examples, 4:1 train/test split; initial-frame DINO plus action representations. | 95.00 Top-1 accuracy (%) | DINO alone: 84.10; Stage 1 action plus DINO: 92.85. Stage 2 improves semantic discrimination. The stated classifier is a nonlinear two-layer MLP despite the source's 'linear probing' label. e19 |
| Action tokenizer reconstruction About 20,000 held-out SFT samples; Stage 2 versus a 30,000-step Action Only tokenizer. | Torso rotation: 0.34059°; torso XYZ MAE: 0.00043 m. Decoded physical error (lower is better) | Action Only: 0.66152° and 0.00038 m; both grippers also worsen in Stage 2. The table supports a mixed tradeoff and contradicts the prose's claim that only grippers worsen. e18 |
| Generalization and human-data transfer Pick-and-place evaluations separate novel instructions/objects; human co-fine-tuning matches steps and hyperparameters to robot-only training. | Figures 12 and 22 show higher unseen-condition bars; no exact values transcribed. Instruction-following and pick-and-place success rates | Lumo-2 beats pick-and-place baselines; adding VisionPro or action-free egocentric data improves unseen-object success over robot-only training. Baseline training histories differ. Human-transfer seen-object performance slightly declines; unlabeled bars and missing trial counts limit precision. e13e17 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 3. Adding active dynamics projection improves both reported temporal-task scores. Original paper, p. 23 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each row horizontally because the comparison is within a task. VLA fine-tuning supplies action supervision; VLWA adds world-dynamics projection alongside action learning, starting from Stage 3 models. The entries are percentages according to the table caption. Conveyor collection changes from 92.00 to 100.00, while rotating-rack placement changes from 74.17 to 81.67. The task descriptions distinguish reacting to moving eggs from anticipating a placement opportunity on a rotating rack. The broader evaluation protocol awards normalized subtask progress, so the table's success-rate label should be retained without assuming that every percentage represents complete multi-step trial success. e14e15
What it supports. The observed gains are 8.00 and 7.50 percentage points. This is more directly relevant to the proposed dynamics mechanism than comparing complete systems with different pretraining histories. It supports the combined VLWA fine-tuning design on these two tasks, with no uncertainty estimates reported.
Where the evidence stops. The intervention changes dynamics supervision and active projection together. It cannot separate a training-time representation benefit from dependence on predicted dynamics during execution. No independent ablation trial count or whole-task scoring clarification accompanies Table 3.
Table 6. The reconstruction comparison shows gains and regressions across different control channels. Original paper, p. 35 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Keep the units separate while reading across columns: rotations use geodesic error in degrees, positions use MAE in meters, and gripper errors have no unit specified here. Lower values are better. Compare each Stage 2 entry only with its Action Only counterpart above it. Torso rotation improves from 0.66152 to 0.34059 degrees, and both arm rotation/position errors also decrease. However, torso XYZ rises from 0.00038 to 0.00043 m; left and right gripper errors rise as well. The bold entries faithfully mark these exceptions. This table evaluates decoded demonstrations on held-out data, rather than actions generated while controlling a robot. e18
What it supports. Semantic alignment is compatible with better reconstruction on several important channels, but its benefits are not uniform. The torso-position regression is small in absolute meters yet still contradicts the adjacent prose's claim that grippers are the only exception. The table supports a selective tradeoff, not universal reconstruction improvement.
Where the evidence stops. The authors consider greater gripper error tolerable, but this table does not quantify its downstream cost on contact-sensitive tasks. It also compares Action Only training with a multi-stage checkpoint, so it does not isolate one alignment loss.
7. Analysis & limitations
7.1 What the evidence leaves open
The future-information probe reaches 90% task accuracy versus 43% from an initial frame and 94% from five frames. Because its latents are computed from observed frame pairs, this supports information retention, not the accuracy of futures predicted online. e12
Robot category gains cannot separately identify contributions from pretraining, semantic alignment, history or latent prediction. Table 3 bundles supervision and projection changes. The scaling comparison varies tokenizer and decoder workflow at one training setup; lower decoded errors do not establish universal scaling behavior. e14e15e16e20
7.2 Questions for discussion
- Does predicted dynamics content improve control after matching decoding latency and training supervision?
- Can semantic alignment preserve its gains when evaluated on whole-task completion and unseen embodiments?
8. Reproducibility audit
8.1 Requirements and known gaps
A full reproduction requires the curated corpora, robot calibration/action schema, tokenizer checkpoints and specified multi-stage compute. The paper supplies substantial optimization details, but leaves mixture weights, exact history length, loss weights and parts of the Stage 2 freeze specification unresolved. e05e06e07e08e10
Proposed minimal checks: hold training constant while intervening on predicted dynamics at inference; separately compare Stage 1/2 tokenizers under matched decoding and report control success alongside physical errors. These are proposed experiments, not reproduced results. e14e18e20
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Does control depend on the content of predicted dynamics?
Reader-proposed check: reproduce the two Table 3 tasks from the same Stage 3 initialization with matched fine-tuning data, steps, history and decoding budget. For the VLWA policy, compare normal predicted dynamics against same-length shuffled dynamics tokens and a masked-dynamics control at inference. Preserve token count and measure latency so a speed change cannot explain performance. Report both normalized progress and whole-task success, with uncertainty over scene layouts and training seeds. A reproducible drop only when informative dynamics are removed would support inference-time use; unchanged performance would weaken that explanation even if VLWA training still helps. e08e09e14e15
Check 2: Do semantic tokens improve control after matching reconstruction and decoding?
Reader-proposed check: compare Stage 1 and Stage 2 tokenizer policies on the paper's pick-and-place setup with identical policy training, action groups, decoding schedule and data split. Include an Action Only tokenizer as a reconstruction reference, and document each checkpoint's full training budget. Measure channel-specific physical errors, semantic classification and executed task success; separately test precise gripper tasks such as matching pen caps. If lower average decoded error or higher semantic accuracy fails to improve control, the representation-to-performance argument is incomplete. Report gripper and torso-position regressions explicitly instead of hiding them inside an aggregate. e15e18e19e20
8.3 Reading coverage
Visual audit: Inspected the title/version and contribution credits, model equations and training pages, Figures 1–5, 7–13 and 19–24, and Tables 1–7. All six final crops were visually checked; the latency crop was extended to retain its complete panel label. Inspected Figure 8(a)'s attention structure although only panel (b) is cropped. Checked the Stage 1 gradient/context arrows against Equations (4)–(5), and retained the Stage 2 projector-freezing discrepancy. Every cited numerical, method, training and reproduction-detail page is included above. Figure 6 and rollout Figures 14–18 were read as text/captions only; their images are outside this visual pass. References were read as supplied text, with no external references opened. No appendix is present; separate supplements remain unverified.
PDF pages inspected for this edition: 1, 5, 6, 7, 8, 9, 10, 11, 12, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 31, 32, 33, 34, 35, 36, 37, 39, 40. Appendix coverage: not present.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract and 1. Introduction
- 2. The Lumo-2 Model, Sections 2.1–2.5
- 3. Training Recipe, Sections 3.1–3.3.4
- 4. Experiments, Sections 4.1–4.5
- 5. Related Work
- 6. Conclusion
- 7. Contributions
- References, pp. 41–49
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Identity notes: the supplied PDF is arXiv:2607.11270v1, dated 13 July 2026, with the exact catalog title. Its title-page byline is Astribot Team; Section 7 names eight contributors matching eight catalog authors. The catalog also lists Weiqi Jin, who is not named in this paper's contributions. The PDF gives no unified individual author order. This is a disclosed catalog credit discrepancy; no revision difference beyond the observed v1 is established.
- All 12 supplied text chunks were read individually, covering the complete 49-page PDF, including references. No appendix is present. Visual inspection covers the pages declared in the illustrated edition; example/rollout figures on pp. 13 and 26–30 were read through their text and captions but their images were not inspected.
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout. This supplied extraction limitation was addressed for the declared inspected pages and six final crops.
- Separate supplemental material availability has not been fully verified.
- Code, project-page content, datasets and external references were not inspected; experiments were not reproduced. No institutional affiliation beyond the team branding is asserted.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title block and arXiv margin
The observed title is Towards Predictive, Aligned, and Scalable Robot Learning. The byline is Astribot Team, with the author list directed to Contributions. The margin identifies arXiv:2607.11270v1 [cs.RO], 13 July 2026.
Go to primary source ↓e02PDF p. 40, Section 7, all contribution categories
Eight distinct contributors are named: Baifu Huang, Binyan Sun, Haotian Yang, Jianan Wang, Kuncheng Luo, Peijun Tang, Shangjin Xie, and Shilin Fang. The section lists names alphabetically within roles, rather than giving a single ordered byline.
Go to primary source ↓e03PDF pp. 5–7, Sections 2.3–2.5, Equations (1)–(3), Figure 2
Observations include multi-view vision and proprioception; action inference incorporates language and temporal context. Equation (3) factorizes latent dynamics and action generation within a single multimodal transformer. Its action factor retains the observation, although the following sentence says actions depend solely on the latent dynamics. The backbone is Qwen3.5-4B.
Go to primary source ↓e04PDF p. 8, Figure 3 and caption, Section 3.1, Equations (4)–(5)
A frozen vision encoder feeds an inter-frame Vision IDM and feature-predicting Vision FDM. Visual and action latents are fused for action reconstruction. The blue global-context arrow feeds action decoding; the dashed green physical-grounding arrow represents gradient-induced regularization. Equation (4) decodes actions from action latent A and dynamics latent phi.
Go to primary source ↓e05PDF pp. 8–9, Section 3.1, Training Details
Frozen DINOv2 processes five 224×224 frames sampled every eight frames at 30 FPS. Head features query wrist features. Eight action groups are temporally compressed from 32 to four frames and quantized; losses are feature MSE and action L1. Human video uses the vision branch; robot data uses both branches. Stage 1 uses AdamW, 30,000 steps, 64 H100 GPUs, batch 24 per GPU, base learning rate 1e-4 and 50× codebook learning rate, dead-code reinitialization, cosine decay and 5% warmup.
Go to primary source ↓e06PDF pp. 10–11, Figure 4, Sections 3.2.1–3.2.2 and Training Details
Stage 2 adds a semantic module and semantic action codebook while freezing the original action encoder/codebook and VLM core. The text lists reconstruction, behavior comprehension, guided action generation, cross-modal prediction, contrast and discrimination objectives. It specifies 32 visual context tokens, 1,024 semantic action tokens, 12,000 steps on 64 H100 GPUs, and base learning rate 1e-4. Figure 4 marks the VLM projector with a snowflake, whereas the prose lists feature projectors as trainable; the decoder is explicitly updated on p. 10 but is not separately listed in p. 11's abbreviated trainable-module summary.
Go to primary source ↓e07PDF p. 12, Figure 5 and caption; p. 14, Table 1 and caption; p. 15, Section 3.3.2 and Figure 7
The curated VLM corpus contains roughly 53 million samples organized around general understanding, localization, cognition and planning. Table 1 names public raw datasets and notes filtering, cleaning and additional construction. Dynamics training additionally uses internet video, egocentric video and robot data.
Go to primary source ↓e08PDF pp. 15–17, Sections 3.3.3–3.3.4, Figure 8(a) and caption
A fixed-length queue of past actions is encoded by the Stage 2 semantic encoder and concatenated with observation tokens. BAR permits intra-block attention while retaining block-level causality. Figure 8 describes alternating dynamics/action token blocks with sizes four and eight. VQ detokenization produces continuous action trajectories; the history-input dropout is 0.5.
Go to primary source ↓e09PDF pp. 16–17, Figure 8(b), latency breakdown and accompanying paragraph
On one NVIDIA RTX 5090 using vLLM, bfloat16 and FlashAttention-2, total latency is 253.66 ms for standard AR and 93.53 ms for BAR, a reported 2.71× speedup. Action generation changes from 32 steps at 6.00 ms each to four steps at 11.24 ms each, totaling 192.00 versus 44.96 ms. Scheduler/runtime costs are 29.82 versus 16.75 ms.
Go to primary source ↓e10PDF p. 17, Stage 3 Training Details
Stage 3 initializes from Qwen3.5-4B and Stage 2 vocabulary embeddings, trains 120,000 steps on 160 H100 GPUs, and mixes VLA, VLW and VLWA robot formats with assigned but unlisted weights. VLM data retain ordinary autoregression and egocentric video uses VLW. Sequence packing allows 8,192 tokens; AdamW starts at 1e-5 with weight decay 0.1 and cosine scheduling.
Go to primary source ↓e11PDF p. 18, Section 4.1 evaluation protocol and Table 2, VSIBench and MindCube rows
Evaluation uses lmms-eval and common visual/greedy-decoding settings. VSIBench scores are 52.34 for Lumo-2 VLM, 48.56 for Stage 3 and 21.73 for Qwen3.5-4B. MindCube scores are 66.08, 54.93 and 39.95 respectively. Thus co-training does not preserve every benchmark score.
Go to primary source ↓e12PDF pp. 19–21, Section 4.2.1, Figures 9–11 and task-inference paragraph
PCA and feature-distance visualizations illustrate task-relevant changes, while cross-embodiment retrieval selects clip pairs with cosine similarity above 0.99. In the task-inference probe, initial-frame DINO features yield 43% accuracy, five-frame features 94%, and an initial frame plus dynamics representations computed from consecutive frame pairs 90%. The probe uses observed future information.
Go to primary source ↓e13PDF pp. 21–22, Section 4.2.2, Figure 12 and caption
Generalizable pick and place compares Stage 3 Lumo-2 without further fine-tuning to pi0.5 fine-tuned for ten epochs and Fast-WAM for three. Basic evaluation uses 60 seen objects; unseen-object evaluation uses 105 new objects. IFR measures identifying/approaching the requested object or location; SR measures successful completion. The plotted ratio is SR/IFR, although the caption once reverses the notation.
Go to primary source ↓e14PDF pp. 22–23, Section 4.2.3 and Table 3, both task rows
Stage 3-based fine-tuning compares VLA action supervision against VLWA with active latent dynamics projection. Table 3 reports success rates of 100.00 versus 92.00 for Collect Eggs from the Conveyor Belt, and 81.67 versus 74.17 for Place Cubes on the Rotating Rack. The table gives no uncertainty estimates or separate trial-count specification.
Go to primary source ↓e15PDF pp. 23–25, Section 4.3, Section 4.3.2, Tables 4–5 and Figure 13
Astribot S1 evaluation covers 22 tasks in six subcategories. Each task has ten scene layouts, each tested twice per model. Multi-step scores average completed meta-subtasks; starred tasks are evaluated from in-domain subtask starts. Static geometry tasks use VLA fine-tuning; temporal/physical prediction tasks use VLWA. History may use action tokens or projected end-effector traces. Fast-WAM receives dedicated image preprocessing.
Go to primary source ↓e16PDF p. 31, Figures 19–20, axes, legends and captions
The task-level and category-level bar charts place Lumo-2 above pi0.5 and Fast-WAM across all displayed tasks/categories. Axes say Task Progress even though captions say success rates. The bars have no exact numeric labels or uncertainty intervals. The long-horizon category shows a particularly large qualitative advantage.
Go to primary source ↓e17PDF pp. 32–34, Section 4.4, Figures 21–22 and evaluation protocol
Human/robot co-fine-tuning uses matched steps and hyperparameters. VisionPro hand poses are retargeted to robot actions for VLWA; action-free multi-view egocentric video uses VLW. Pick-and-place evaluation distinguishes objects seen and unseen in native robot training. Both human sources improve plotted unseen-object success, with slightly lower seen-object bars; trial counts and uncertainty are not specified.
Go to primary source ↓e18PDF pp. 34–35, Section 4.5 reconstruction protocol, Table 6 and preceding paragraph
An Action Only tokenizer trained for 30,000 steps is compared with Stage 2 on about 20,000 held-out SFT samples. Table 6 gives torso rotation error 0.66152° versus 0.34059°, torso XYZ MAE 0.00038 versus 0.00043 m, and left gripper MAE 0.40307 versus 0.58081. Both arm position/rotation errors improve; both gripper errors worsen. The prose incorrectly states that grippers are the only exception.
Go to primary source ↓e19PDF pp. 35–36, Figure 23, action semantic prediction protocol and Table 7
The semantic probe uses 18 action categories, 400 examples each, and a 4:1 train/test split. Top-1 accuracies are 84.10% for DINO alone, 92.85% with Stage 1 action features and 95.00% with Stage 2 semantic action features. Despite being called linear probing, the specified classifier is a two-layer MLP with ReLU, hidden dimension 1,024 and dropout 0.1.
Go to primary source ↓e20PDF pp. 36–37, Scaling with Semantic Action Representation and Figure 24
Stage 1- and Stage 2-tokenizer policy variants are trained for 100,000 steps on eight H100 GPUs, batch 40 per device, using the same pick-and-place data and hyperparameters but with minor decoder-workflow differences. Validation uses 100 complete trajectories, about 15,000 frames. Stage 2 has lower decoded errors except grippers despite higher final BAR loss. This is a training-step comparison, not a model-size scaling-law fit.
Go to primary source ↓e21PDF p. 39, Section 6
The conclusion attributes efficient future-aware control to latent dynamics and progressive alignment without pixel-level future rollouts at inference.
Go to primary source ↓8.5 Primary sources
Towards Predictive, Aligned, and Scalable Robot Learning ↗
PDF · 19,374 extracted words
Source fingerprint
a77b3d5bf5050365ef253eb60c20fb6f59240bdf3cf15024dd60cd34a37e39ea