ViPRA: Video Prediction for Robot Actions
1. Paper overview
In one sentence: Joint future-image and motion-latent pretraining supplies useful robot-control priors, but deployment relies on action-only continuous decoding and gains depend on the benchmark. e-methode-latente-pretraine-adapte-simplere-reale-timing
| At a glance | What to know |
|---|---|
| Research problem | Source description Passive videos show interactions without the robot commands needed for imitation learning. ViPRA asks whether predicting visual change and motion latents can supply transferable control priors, leaving a smaller labeled demonstration set to align those priors with a robot's motor space. e-identitye-method |
| Core mechanism | Source description A non-causal inverse model learns discrete transition tokens through causal visual reconstruction, perceptual supervision and bidirectional optical-flow consistency. e-latente-latent-architecture |
| A key reported result | Physical single-arm manipulation: ViPRA-FM 54.1 Average full-task success (%). Cover-Object, Pick-Place and Stack-Cups; fixed front camera, 180 demonstrations per task and seven-of-fourteen action execution. Figure 3: π₀ 41.8 and Scratch-FM 23.8; Section 5.4 instead prints π₀ 40.1. The figure supports a 12.3-point advantage over π₀. The conflicting prose baseline is not silently reconciled. Trial counts and uncertainty are not given for this aggregate. e-datae-reale-real-protocole-hardware |
| Reading caution | Source description The authors identify limited data diversity, dexterous contact control and unknown scaling behavior as open problems. Test-time planning and reinforcement-learning alignment are future directions, not evaluated capabilities. e-limitations |
Core contributions
- Source description
A non-causal inverse model learns discrete transition tokens through causal visual reconstruction, perceptual supervision and bidirectional optical-flow consistency. e-latente-latent-architecture
- Source description
A shared video-language transformer predicts an end-of-horizon image and the intervening latent-action sequence during pretraining; continuous finetuning then learns embodiment-specific action chunks. e-pretraine-adapt
Figure 2. The shared backbone changes its prediction interface between pretraining and motor adaptation. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each stage from its inputs upward. On the left, the full video clip enters the inverse tokenizer Iβ; discrete tokens and observation history feed the forward decoder Fα. In the middle, past frames and instruction c condition a shared video-language model, while future frames provide training targets through frozen tokenizers. Green modules embed and predict latent actions. On the right, robot image history and noisy action tokens feed that backbone and the flow decoder Hη. The snowflakes and flames agree with the frozen tokenizers and trainable components specified in the algorithms. These columns represent successive stages, not three simultaneously executed controllers. e-methode-latent-architecturee-pretraine-pretrain-confige-adapte-flow-convention
What it supports. The architecture supports a shared-model interpretation: visual and latent-action prediction train the same backbone, and continuous adaptation reuses that backbone with an action encoder and flow head. Algorithm 3 makes the deployment distinction explicit: it conditions on history, language and noisy actions, without a generated future frame or explicit discrete-latent input.
Where the evidence stops. The caption's mapping from latents to actions is conceptual; Algorithm 3 specifies the actual inputs. The flow branch's printed interpolation and integration direction also disagree, so the diagram alone cannot define a reproducible sampler.
2. Motivation
2.1 The problem and the proposed response
Passive videos show interactions without the robot commands needed for imitation learning. ViPRA asks whether predicting visual change and motion latents can supply transferable control priors, leaving a smaller labeled demonstration set to align those priors with a robot's motor space. e-identitye-method
2.2 What this reading follows
A video can reveal that a hand lifts an object without revealing the motor command that caused the lift. ViPRA uses that distinction to learn from actionless human and robot footage. It first compresses visual transitions into discrete motion tokens, then trains a shared video-language model to anticipate a future image and the latent transitions leading to it. Robot demonstrations subsequently adapt the backbone to continuous action chunks. The figures below trace that handoff and test its value against execution results. They also expose important boundaries: generating a plausible transition is different from completing a manipulation, and faster chunk execution is different from faster visual feedback. e-methode-latente-pretraine-adapte-simplere-reale-timing
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | WAMs |
| Architecture | One Model |
| Prediction paradigm | Joint prediction |
| Quadrant | Q1 · One Model × Joint prediction |
3.1 Evidence-based assessment
Supports the recorded classification
The recorded One Model × Joint prediction classification is supported for pretraining: a single shared transformer produces future visual tokens and latent actions through modality heads. Separate training tokenizers do not turn this into a deployed video-generator-plus-inverse-controller pipeline. Qualify the label at inference: ViPRA-FM emits continuous actions without joint future-video prediction. e-methode-pretraine-adapte-design-ablation
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 1. Learn a transition representation before learning a controller
Begin with a short clip whose frames are observations, not action labels. ViPRA's inverse encoder sees the complete clip, so a token can use surrounding motion to distinguish superficially similar transitions. DINOv2 supplies spatial features, temporal attention supplies context, and NSVQ compresses the result into discrete codes. The forward decoder then receives observation history and those codes and must reconstruct the next frame causally. Pixel and perceptual losses reward reconstruction; RAFT consistency also asks reconstructed motion to agree with observed motion in both temporal directions. The latter is introduced only after a warm-up. This construction teaches a representation of visual change, while leaving the mapping to a particular robot's motors for the later demonstration-supervised stage. e-methode-latente-latent-architecturee-latent-confige-adapt
5.2 2. Use the future image to structure pretraining, then remove it from control
The multimodal stage has two recent frames and a language instruction as context. Its autoregressive sequence predicts the image at the end of the horizon and then the latent actions spanning that horizon. Teacher forcing supplies the observed future-image tokens when training the latent predictions. The image objective and latent objective therefore shape one transformer, rather than supervising unrelated predictors. During continuous adaptation, Algorithm 3 changes the interface: image history, language and noisy actions feed the backbone, and a flow head predicts the action refinement field. No generated future frame is listed as an input. Reader interpretation: the learned predictive structure transfers through the parameters. This distinction explains why the catalog's joint-prediction label describes pretraining more accurately than the deployed control loop. e-pretraine-pretrain-confige-adapte-design-ablation
5.3 3. Separate smooth action chunks from frequent perception
ViPRA-FM predicts a chunk of fourteen actions, using ten stated Euler refinement steps and cached image/language attention states. In the single-arm protocol, seven commands are executed before the policy replans. Smooth within-chunk commands can therefore reduce repeated model computation, but they also create an interval without a fresh policy decision. The source reports 510 milliseconds per chunk and separately claims robot-side operation up to 22 Hz; its scheduler description does not establish 22 new image-conditioned decisions per second. A second implementation boundary appears in the equations: the printed interpolation puts the target at zero and noise at one, while the sampler describes the reverse direction. Reader interpretation: both the timing contract and flow convention must be resolved before an exact control reproduction. e-adapte-flow-conventione-real-protocole-timing
5.4 Training and inference
During training
Pretraining uses 198k Something-Something v2 human videos and actionless robot subsets of 87k Fractal, 25.4k BridgeV2 and 85.6k Kuka videos. Adaptation uses 100 collected multi-task SIMPLER trajectories; the physical single-arm study states 180 GELLO demonstrations per task. e-data
Latent learning runs 240,000 steps, batch 128, with flow supervision after 60,000 steps. Multimodal pretraining runs 50,000 steps, batch 512, with H=14; SIMPLER flow adaptation uses 12,000 steps. The VQ tokenizer remains frozen, while the video backbone is trainable. e-latent-confige-pretrain-confige-adapt
During inference
The stated sampler uses ten forward Euler steps. Language/image KV states are cached across these steps. The physical single-arm protocol predicts 14 actions and executes seven before replanning from updated observations; it reports a capped effective rate of 3.5 Hz. e-adapte-real-protocole-timing
Future-image prediction is removed for action finetuning and control. Thus video modeling supplies learned priors rather than an online visual-planning loop. The inverse tokenizer labels training clips; it is not a runtime inverse model translating generated frames into commands. e-pretraine-adapte-design-ablation
5.5 Implementation flow
- Extract motion tokens
The DINOv2-initialized inverse encoder Iβ sees the entire clip, including future frames, and quantizes contextual features through NSVQ into a shared eight-entry codebook. The causal forward decoder Fα reconstructs each next frame from observation and latent histories. These training-time inferred latents are not measured robot commands. e-methode-latente-latent-architecture
- Learn visual and latent transitions together
Frozen VQ-VAE tokenization maps observations to visual tokens. LWM-Chat-1M, latent embedding Eϕ and token decoder Hψ are trained with teacher forcing. From history and language, the sequence predicts future image tokens before the latent tokens, which condition on that future image during training. e-pretraine-pretrain-config
- Adapt the shared backbone to motors
A noisy-action encoder Eγ embeds the continuous action chunk and interpolation time; the backbone and flow head Hη predict its refinement field. Algorithm 3 updates all three components. Its conditioning contains history, language and noisy actions, with no explicit generated image or discrete-latent input. e-adapt
6. Experiments & results
ViPRA learns discrete motion representations from actionless human and robot videos, jointly pretrains an LWM video-language backbone on future images and latent actions, then adapts that backbone to continuous action chunks. Its control policy does not generate future video at deployment. Gains on SIMPLER and physical manipulation coexist with weaker LIBERO-10 results and unresolved flow-matching notation (e-method, e-pretrain, e-adapt, e-simpler, e-real, e-libero, e-flow-convention).
6.1 Read the original evidence
Table 1. SIMPLER distinguishes completing the task from merely obtaining a grasp. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the upper Success Rates block, then compare its average row across the discrete and continuous groups. ViPRA-AR and ViPRA-FM belong to different action-output families, so their comparison also changes the decoder. The lower Grasp Rates block asks whether the correct object was grasped, without requiring successful transport or placement. For StackG2Y, OpenVLA's grasp rate is much higher than its completion rate, making post-grasp behavior relevant to interpretation. Dashes in UniVLA's task rows are missing reported entries, not zero scores. Appendix E describes four Bridge tasks and adaptation using one hundred collected multi-task trajectories. e-simplere-simpler-protocole-baselines
What it supports. ViPRA-AR reaches 69.8% average success and ViPRA-FM 62.5%. Relative to LAPA's 53.1%, the AR gain is 16.7 percentage points. The FM result exceeds Scratch-FM's 41.7% by 20.8 points. These are aggregate gains on the stated simulation task suite.
Where the evidence stops. The table supplies no uncertainty intervals or per-task UniVLA values. Baselines have different pretraining sources and architectures, and some results are imported from prior work; this is not a compute- and data-matched mechanism test.
Figure 3. Physical completion improves, while grasp-to-completion gaps remain substantial. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use the shared legend to track Scratch-FM in yellow, π₀ in blue and ViPRA-FM in green. The left panel measures partial success, defined as grasping the correct object; the middle panel requires the whole task. Follow one task across both panels before reading the average. For Cover-Object, ViPRA's high cloth-grasp rate does not imply equally reliable draping. The photograph shows the broader two-arm platform, while these plotted evaluations concern three single-arm tasks. The described policy uses the front camera, predicts fourteen actions and executes seven before replanning, with a stated 3.5 Hz evaluation cap. e-reale-real-protocole-hardwaree-smoothness
What it supports. The plotted average full success is 54.1% for ViPRA-FM, 41.8% for π₀ and 23.8% for Scratch-FM. ViPRA's Cover-Object grasp/full-success pair is 98.1%/59.2%, showing that a strong initial interaction still leaves a sizable downstream placement or covering challenge.
Where the evidence stops. Section 5.4 prints π₀'s average as 40.1%, conflicting with the figure's 41.8%; the plot values are retained here. The source gives no aggregate trial counts or error bars, and physical comparisons exclude discrete policies.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| SIMPLER four-task manipulation Four Bridge tasks on WidowX after 100 multi-task finetuning trajectories; success and grasp are separate metrics. | ViPRA-AR 69.8; ViPRA-FM 62.5 Average full-task success (%) | LAPA 53.1; OpenVLA 38.6; Scratch-FM 41.7; π₀ 27.1; UniVLA 42.7. AR exceeds LAPA by 16.7 percentage points; FM exceeds Scratch-FM by 20.8 points. This simulation ranking does not establish physical superiority of AR. e-simplere-simpler-protocol |
| Physical single-arm manipulation Cover-Object, Pick-Place and Stack-Cups; fixed front camera, 180 demonstrations per task and seven-of-fourteen action execution. | ViPRA-FM 54.1 Average full-task success (%) | Figure 3: π₀ 41.8 and Scratch-FM 23.8; Section 5.4 instead prints π₀ 40.1. The figure supports a 12.3-point advantage over π₀. The conflicting prose baseline is not silently reconciled. Trial counts and uncertainty are not given for this aggregate. e-datae-reale-real-protocole-hardware |
| LIBERO-10 long-horizon manipulation Ten tasks, ten runs per task under five seeds: 500 episodes. Appendix E calls the aggregate average reward; Table 8 labels it success rate. | 0.79 Reported success rate | OpenVLA 0.54; π₀-FAST 0.60; π₀ 0.85; UVA 0.90; UniVLA 0.92. Transfer is positive but not leading. Sensing differs across methods; the authors associate precision-grasp failures with accumulated delta-end-effector drift. e-libero |
| Future-state and chunking ablation Table 2, average SIMPLER success over four Bridge tasks, H=14. | Full AR/FM: 69.8/62.5; without pretraining state prediction: 59.4/53.2; without chunking: 59.2/44.8. Full-task success (%) | Moving state prediction into finetuning gives 53.1/31.3. Joint pretraining helps. The +SP3 rows also remove pretraining state prediction, so they do not isolate adding a finetuning loss while holding pretraining fixed. e-design-ablation |
| Video composition ablation Human-only, robot-only and combined latent learning/pretraining, evaluated on LIBERO-10. | Combined: 0.79; 0.84 Success rate; action-probe MSE | Robot-only: 0.72; 0.91. Human-only: 0.69; 0.99. Combining sources improves these reported measures; the table does not establish a data-volume-matched comparison. e-data-ablation |
| Optical-flow supervision ablation Latent learning with versus without optical-flow loss; frozen-backbone probe predicts BridgeV2 actions. | With flow: 0.84; 5.01; 1.59 Action MSE; perplexity; entropy | Without flow: 0.92; 5.63; 1.74. The probe supports more action-informative representations. Lower token uncertainty alone does not prove physical correctness or better executed control. e-flow-ablation |
| Bimanual Place-in-Bowl and Mix-with-Whisk Two Franka arms, single front camera, shared policy, 14-action chunks at stated 7 Hz. | Place-in-Bowl: 10/18; Mix-with-Whisk: 8/12 Completed trials | No quantitative bimanual baseline is supplied. Demonstrates coordinated execution in these setups; the authors attribute bowl-task failures mainly to grasping limitations of the custom gripper. e-bimanual |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 2. The pretraining and finetuning columns are essential to interpreting each ablation. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Translate FS as future-state prediction, L as latent-action prediction, A as robot actions and H as the fourteen-step chunk horizon. Read the Pretrain and Finetune columns together before comparing the Succ. column. Within each decoder family, −SP2 removes future-state prediction during pretraining, and −AC changes both latent and robot action sequences to one-step targets. The +SP3 rows require special care: their pretraining column is latent-only, while future-state prediction appears in finetuning. Thus +SP3 does not start from the same joint-pretrained configuration as the full model. This matters when interpreting the paper's explanation in terms of prediction-error accumulation. e-design-ablatione-pretraine-adapt
What it supports. Removing pretraining state prediction lowers AR/FM success from 69.8/62.5 to 59.4/53.2. Removing chunking gives 59.2/44.8. Together these comparisons support the utility of the full design on SIMPLER. They do not independently isolate every causal contribution to the large +SP3 degradation.
Where the evidence stops. The −AC rows change chunking in two stages, and +SP3 changes both the pretraining and finetuning objectives. These are design comparisons, not fully factorial controls; no uncertainty is supplied for the reported averages.
Table 4. An action probe complements code-distribution statistics in testing optical-flow supervision. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Compare the default and no-flow rows column by column. Perplexity and entropy describe the latent representation's reported token uncertainty; Action MSE measures an action probe. The surrounding text specifies a frozen backbone and BridgeV2 action prediction, making the last column more directly relevant to control information than token statistics alone. The ablated flow loss belongs to latent-token learning: it compares RAFT estimates between reconstructed and observed frame pairs. It is distinct from the flow-matching decoder used for continuous robot actions. Keep those two uses of the word flow separate when interpreting what this small table tests. e-latente-flow-ablation
What it supports. With optical-flow supervision, the reported action MSE is 0.84 rather than 0.92, alongside perplexity 5.01 rather than 5.63 and entropy 1.59 rather than 1.74. The action probe supports the claim that this reconstruction constraint helps preserve motion information useful for predicting robot actions.
Where the evidence stops. The paper does not fully specify the probe's split, normalization or uncertainty. Lower perplexity and entropy alone do not establish better representations; this table also supplies no direct robot-success comparison for removing optical flow.
Figure 6. Qualitative command traces show local discontinuities that aggregate success rates conceal. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow the blue and green curves within each joint panel rather than comparing absolute values across joints. Both policies track similar broad motion trends, but green LAPA commands contain isolated drops and spikes, particularly in the Joint 2 and Joint 6 panels. The legend identifies blue as ViPRA, whose continuous decoder generally changes more smoothly in these examples. The horizontal axis is time steps, not a calibrated timing measurement. Section 5.5 and Appendix F clarify that these plots replay finetuning trajectories through the inference pipeline, so the observations come from demonstration data rather than an independent held-out physical test. e-smoothness
What it supports. The traces support a qualitative distinction between these continuous and discretized command outputs. They help motivate the paper's deployment choice, alongside the authors' separate report that discontinuous discrete commands triggered Franka safety stops. They do not quantify how much smoother the policy is across tasks or unseen conditions.
Where the evidence stops. No jerk statistic, uncertainty estimate or smoothness threshold is plotted. Training-trajectory replay and differing decoder designs limit causal attribution, while smooth commands by themselves do not guarantee accurate contact forces, safe execution or successful task completion.
7. Analysis & limitations
7.1 What the evidence leaves open
The authors identify limited data diversity, dexterous contact control and unknown scaling behavior as open problems. Test-time planning and reinforcement-learning alignment are future directions, not evaluated capabilities. e-limitations
Smoothness figures replay finetuning trajectories through the inference pipeline; they are qualitative diagnostics, not held-out jerk statistics. Baselines vary in pretraining data, architecture and action space, limiting causal attribution of the headline gains. e-baselinese-smoothness
The reported 510 ms per 14-action chunk and approximately 1.95 chunk predictions/s do not mean 22 fresh visual decisions/s. Appendix G's 3.5/7 Hz rollout terminology does not fully specify the scheduler or reconcile robot-side execution with replanning. e-timing
7.2 Questions for discussion
- Would the pretraining advantage survive matching video volume, compute, sensing and action parameterization across baselines?
- How much of observed retry behavior depends on temporal pretraining versus observation history and action-chunk execution?
8. Reproducibility audit
8.1 Requirements and known gaps
Required components include LWM-Chat-1M, its VQ tokenizer, DINOv2, RAFT, the four video sources and downstream demonstrations. Tables 5–6 report eight NVIDIA H100 GPUs for 168 hours of latent learning and 144 hours of pretraining. Section 7 claims released code/models but describes the collected SIMPLER trajectories as a future release. e-releasee-datae-latent-confige-pretrain-config
Before reproducing control, resolve the contradictory flow endpoints and field scaling in Section 4.3, Equations (7)–(10) and Algorithm 3. Also specify action normalization, dataset splits, software versions, physical-trial counts and timing hardware: these are not sufficiently pinned down by the supplied configuration tables and evaluation descriptions. e-flow-conventione-adapte-latent-confige-pretrain-confige-simpler-protocole-reale-hardwaree-timing
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Factor pretraining and finetuning image prediction independently
Reader-proposed, not run: use the same actionless clips, LWM initialization, finetuning demonstrations, seeds and H=14 action chunks in a two-by-two experiment toggling future-image supervision during pretraining and during finetuning independently. Evaluate identical SIMPLER initial states, reporting full success, grasp success and seed variability. The decisive control is to add finetuning image prediction while retaining the full pretrained model; Table 2's +SP3 instead also removes pretraining state prediction. If the disadvantage persists in this controlled pair, it supports the claimed cost of coupling generated images to action decoding. If it disappears, the original ablation does not isolate that explanation. e-pretraine-adapte-design-ablatione-simpler-protocol
Check 2: Check the printed flow path against its sampling endpoints
Reader-proposed, not run: before robot deployment, use fixed Gaussian noise and a fixed target action chunk to test the printed interpolation at both endpoints, its derivative, the (1-s) field factor in the loss, and ten-step Euler sampling. Compare the literal printed equations with a consistently defined noise-to-target convention established through author clarification. Hold examples and step count fixed. Require finite fields and reconstruction of the intended target endpoint in the constant-target test, then compare endpoint error on held-out demonstrations. Failure of the literal convention would locate a reproducibility defect; success of an alternative would not establish that it matches the authors' implementation. e-flow-conventione-adapt
8.3 Reading coverage
Visual audit: Visually inspected the title/author/version block; Figure 2 and its caption against Sections 4.1–4.3 and Algorithms 1–3; Figures 3–11; Tables 1–9; and the supporting method, training, evaluation and appendix pages listed here. All six final original crops were inspected, including the corrected smoothness crop. Figure 3's π₀ average conflicts with Section 5.4, and the printed flow path conflicts with its stated integration direction; both are disclosed. The +SP3 table columns were checked directly and show changed pretraining as well as finetuning. Figure 1 was read through its text/caption but was not visually inspected. Separate supplements, external videos, code and earlier revisions were outside the supplied visual material.
PDF pages inspected for this edition: 1, 4, 5, 6, 7, 8, 9, 10, 11, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract and Sections 1–7
- Acknowledgments and References
- Appendix A: Background
- Appendix B: Latent Action Learning
- Appendix C: Multimodal Video Pretraining with Latent Actions
- Appendix D: Flow Matching Decoder for Continuous Control
- Appendix E.1–E.4: SIMPLER and LIBERO Long
- Appendix F: Action Output Analysis
- Appendix G.1–G.5: Real World Experiments
- Appendix H.1–H.2: Bimanual Manipulation
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Identity/version: the title and all five authors match the catalog. The inspected artifact is arXiv:2511.07732v2, dated 30 March 2026, and marked as published at ICLR 2026. The catalog BibTeX instead names NeurIPS Workshop, 2025; that earlier edition was not supplied or compared.
- The supplied text extraction does not reconstruct figure images; original PDF pages and six final crops were inspected to recover visual and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- All 11 supplied text chunks were read, covering the complete 36-page PDF. Visual inspection covered the claim-supporting pages declared in the edition; reference-list pages were read as text.
- Code, external project material and rollout videos were not inspected; no experiments were reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e-identityPDF p. 1, title block, arXiv margin and Abstract
Title and five authors match the supplied identity; affiliations are Carnegie Mellon University, Skild AI and University of California, Irvine. Margin states arXiv:2511.07732v2, 30 March 2026; header names ICLR 2026.
Go to primary source ↓e-methodPDF p. 4, Figure 2 and Section 4.1
Three-stage architecture; inverse encoder uses full observation clips and an eight-entry shared codebook.
Go to primary source ↓e-latentPDF p. 5, Section 4.1, Equations (1)–(2)
Causal reconstruction uses L1, LPIPS and two-direction RAFT flow consistency activated after warm-up.
Go to primary source ↓e-latent-architecturePDF p. 23, Appendix B and Algorithm 1; p. 24, Figure 7
DINOv2-initialized spatial encoding, bidirectional inverse temporal attention, NSVQ and causal forward temporal attention define latent learning.
Go to primary source ↓e-pretrainPDF p. 5, Section 4.2 and Equation (3); p. 25, Algorithm 2
LWM jointly trains visual and latent token predictions with teacher forcing; latent prediction conditions on end-of-horizon visual tokens.
Go to primary source ↓e-latent-configPDF p. 24, Table 5, Training Configuration and Latent Action Quantization
240,000 steps, batch 128, 60,000-step flow warm-up, LPIPS/flow weights 0.5/0.1, eight H100 GPUs for 168 hours; codebook size eight.
Go to primary source ↓e-pretrain-configPDF p. 26, Table 6
Frozen VQ-VAE, trainable LWM and latent modules; 50,000 steps, batch 512, H=14, AdamW at 4e-5, bfloat16; eight H100 GPUs for 144 hours.
Go to primary source ↓e-adaptPDF p. 6, Section 4.3; p. 27, Algorithm 3 and Table 7
Continuous adaptation updates backbone, noisy-action encoder and flow decoder; Beta(1.5,1) interpolation, H=14 and ten Euler steps; SIMPLER uses 12,000 steps.
Go to primary source ↓e-flow-conventionPDF p. 6, Section 4.3 interpolation and inference paragraph; pp. 21–22, Equations (7)–(10); p. 27, Algorithm 3 lines 2–7
Printed interpolation weights noise by s and target by 1-s, but the sampling description calls s=0 noise and s=1 target. The objective scales predicted field by 1-s while Euler integration uses the field directly.
Go to primary source ↓e-dataPDF p. 6, Section 5.1, Training Dataset, Simulation Benchmarks and Real World Manipulation
Lists 198k human videos; 87k Fractal, 25.4k BridgeV2 and 85.6k Kuka videos; 100 collected SIMPLER trajectories and 180 physical demonstrations per task.
Go to primary source ↓e-simplerPDF p. 7, Table 1, Success Rates and Grasp Rates, AVG and task rows
Average success: ViPRA-AR 69.8, ViPRA-FM 62.5, LAPA 53.1, OpenVLA 38.6, Scratch-FM 41.7, π₀ 27.1, UniVLA 42.7. UniVLA per-task entries are absent.
Go to primary source ↓e-simpler-protocolPDF p. 28, Appendix E.1–E.2
Four Bridge tasks, WidowX, 100 collected multitask adaptation trajectories; separates complete task success from successful grasp.
Go to primary source ↓e-realPDF p. 7, Figure 3, Full and Partial panels; p. 8, Section 5.4
Figure average full success is 54.1 for ViPRA-FM, 41.8 for π₀ and 23.8 for Scratch-FM; prose instead gives π₀ 40.1. Figure shows Cover-Object partial/full ViPRA values 98.1/59.2 and no error bars.
Go to primary source ↓e-real-protocolPDF p. 7, Section 5.1 continuation below Table 1
Front camera, full versus grasp success definitions, H=14, seven executed actions before replanning and stated 3.5 Hz evaluation cap.
Go to primary source ↓e-baselinesPDF p. 7, Section 5.2
Baseline families differ in backbone, action representation and pretraining sources; UniPI/VPT results are taken from prior work in a stated comparable setting.
Go to primary source ↓e-design-ablationPDF p. 9, Table 2, Pretrain/Finetune/Succ. columns; pp. 8–9, Section 5.5
Full AR/FM 69.8/62.5; −SP2 59.4/53.2; −AC 59.2/44.8; +SP3 53.1/31.3. The +SP3 rows train latent-only before adding future-state prediction at finetuning.
Go to primary source ↓e-data-ablationPDF p. 9, Table 3; p. 10, Human and robot data composition ablation
Combined, robot-only and human-only success/MSE: 0.79/0.84, 0.72/0.91, 0.69/0.99; task is LIBERO-10.
Go to primary source ↓e-flow-ablationPDF p. 9, Table 4; p. 10, Latent action analysis
With/no flow: perplexity 5.01/5.63, entropy 1.59/1.74, action MSE 0.84/0.92. Text identifies a frozen-backbone BridgeV2 action probe.
Go to primary source ↓e-smoothnessPDF p. 9, Figure 6; p. 10, Section 5.5 continuation; pp. 30–31, Appendix F and Figure 8
Blue ViPRA-FM traces are smoother than green LAPA traces in joint space. The diagnostic replays finetuning training trajectories through the inference pipeline; physical safety-stop observations are separate author reports.
Go to primary source ↓e-liberoPDF p. 29, Table 8 and Appendix E.3–E.4
LIBERO-10 uses 500 episodes; ViPRA-FM 0.79 trails π₀ 0.85, UVA 0.90 and UniVLA 0.92. Text discusses image-only sensing and delta-EEF drift.
Go to primary source ↓e-hardwarePDF p. 32, Appendix G.1–G.3; p. 33, Figure 9 and G.4
Two Franka Panda arms, one front-mounted ZED camera used monocularly, GELLO collection at 15 Hz; object/cloth/cup changes and qualitative retries are described.
Go to primary source ↓e-timingPDF pp. 33–34, Appendix G.5 and Table 9
Image/language KV reuse across Euler steps; 14-action chunk in 510 ms, about 1.95 Hz. Text separately reports 7/14 and 14/14 rollout rates of 3.5 and 7 Hz and robot-side rates up to 22 Hz; table identifies ViPRA as 7B.
Go to primary source ↓e-bimanualPDF p. 35, Appendix H.1 and Figure 10; p. 36, H.2 and Figure 11
Shared policy coordinates both Franka arms from one camera; stated 14-action chunks at 7 Hz. Place-in-Bowl completes 10/18 and Mix-with-Whisk 8/12 trials.
Go to primary source ↓e-limitationsPDF p. 10, Section 6
Limits include dexterity, dataset diversity and uncharacterized scaling; tactile/force sensing, broader embodiments, RL alignment and planning are proposed directions.
Go to primary source ↓e-releasePDF p. 11, Section 7
Names open components and claims released code, models and benchmark scripts, while describing the collected SIMPLER dataset release in future tense.
Go to primary source ↓8.5 Primary sources
ViPRA: Video Prediction for Robot Actions ↗
PDF · 17,239 extracted words
Source fingerprint
769b722d869fabb831d11f7b3398bd4b8a922b1715eb78ad909de99be829ee7c