mimic-video: Video-Action Models for Generalizable Robot Control Beyond VLAs
1. Paper overview
In one sentence: A separate inverse-dynamics decoder turns video-model hidden states into robot actions, gaining decoder-data efficiency while leaving video adaptation costs and task-dependent inference choices unresolved. e-architecturee-traininge-samplinge-efficiencye-oraclee-real-setup
| At a glance | What to know |
|---|---|
| Research problem | Author claim The authors argue that static image–text pretraining leaves expensive robot demonstrations to teach dynamics as well as control. They use video pretraining for visual-dynamics priors and learn a translator from video representations to motor commands. This motivates a representation choice; it does not establish that video pretraining learns physical causality. e-motivation |
| Core mechanism | Source description A two-model construction combines a language-conditioned video flow model with an inverse-dynamics action decoder trained on intermediate hidden states. e-architecturee-training |
| A key reported result | LIBERO action-decoder data efficiency: 77% success at 2% data (one episode per task); author-reported approximately 10x decoder-data efficiency Average success and action-data fraction. Decoder data subsets of 2%, 10%, 50% and 100% across the three evaluated suites. The authors describe 10%-data mimic-video as reaching the full-data matched VLA level. This does not establish a tenfold reduction in video pretraining, robot-video adaptation or total compute. e-efficiencye-training |
| Reading caution | Source description The authors acknowledge a fixed single view, narrow real-world task coverage and no unified cross-embodiment model. e-limits |
Core contributions
- Source description
A two-model construction combines a language-conditioned video flow model with an inverse-dynamics action decoder trained on intermediate hidden states. e-architecturee-training
- Source description
Independent video and action noise schedules permit marginal action sampling without completing video generation. e-samplinge-training
Figure 3. Video features condition a separate action generator with its own denoising schedule. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the language branch on the left and the observed video entering the blue block. Section IV identifies the language encoder as T5 and the video backbone as Cosmos-Predict2. The crucial connection is specified in the text: the green action decoder cross-attends to hidden states from an intermediate video layer. It also receives proprioception, which this schematic does not draw explicitly. The two Repeat loops belong to separate flow processes, with video time τᵥ and action time τₐ. Follow Algorithm 1 for execution: video integration may stop early, while the action process continues to a clean action chunk. e-architecturee-samplinge-training
What it supports. The architecture separates learning visual predictions from translating predictive features into motor commands. At the default τᵥ = 1, the video integration loop is skipped, but a video-network feature pass still occurs and the action decoder still denoises. Thus the compute saving concerns video reconstruction, not elimination of the video backbone.
Where the evidence stops. The visible future frames are a conceptual illustration. Algorithm 1, rather than their appearance, establishes what inference computes. The diagram alone does not show the frozen-backbone training boundary or specify a real-time latency.
2. Motivation
2.1 The problem and the proposed response
The authors argue that static image–text pretraining leaves expensive robot demonstrations to teach dynamics as well as control. They use video pretraining for visual-dynamics priors and learn a translator from video representations to motor commands. This motivates a representation choice; it does not establish that video pretraining learns physical causality. e-motivation
2.2 What this reading follows
A video model can help a robot without producing a watchable movie at every decision. mimic-video adapts a pretrained video generator to robotics, freezes it, and learns a separate action decoder that reads its hidden states. The surprising inference setting supplies pure noise in the future-video slots: observed frames and language still shape the features, while full action denoising produces motor commands. Read the architecture alongside the benchmark tables and noise sweep. Together they support useful predictive representations and strong action-decoder efficiency, but they also expose differences between oracle futures, simulated success, physical execution and total training cost. e-architecturee-traininge-samplinge-efficiencye-oraclee-real-setup
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | WAMs |
| Architecture | Dual-system |
| Prediction paradigm | IDM |
| Quadrant | Q4 · Dual-system × IDM |
3.1 Evidence-based assessment
Supports the recorded classification
Dual-system × IDM is supported by separate video/action DiTs, disjoint training and action decoding from video hidden states. Joint-distribution language does not imply one shared predictor. With default pure-noise future input, visual planning means predictive features rather than a completed trajectory. e-architecturee-traininge-sampling
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Separate what the video model learns from what the robot executes
The training sequence explains why this is an inverse-dynamics construction. First, robotics videos adapt the pretrained video backbone through LoRA. Next, action demonstrations teach a separate flow decoder while the adapted video backbone stays frozen. During this second stage, the model sees observed context plus a corrupted version of the demonstration's actual future video; the action target is corrupted independently. The decoder can therefore learn to translate predictive visual representations into commands without updating the video generator through an action loss. At inference, the future is unavailable and must be represented by noise or partially generated latents. Algorithm 1 then completes action denoising. Figure 4's executed trajectories and decoded movies are separate outputs: the movies shown for explanation were not decoded during autonomous execution. e-traininge-architecturee-samplinge-real-setup
5.2 Understand why cleaner video need not produce better action features
The oracle case study and noise sweep ask different questions. Figure 2 supplies privileged future video and obtains near-perfect success, showing that its representations can support action decoding. Figure 7 asks how an autonomous policy behaves when future conditioning comes from the video model itself. Figure 8 returns to noisy ground-truth futures and measures held-out reconstruction MSE, with an optimum around video time 0.4. The authors propose two explanations: generated video may differ from the training distribution, and internal features may become less useful near the clean endpoint. Reader interpretation: neither mechanism is isolated by these plots, and better pixels need not mean better hidden-state conditioning. At time one, clean observations and language remain available even though future slots are pure noise; a useful feature pass does not require a visible future movie. e-oraclee-noisee-oracle-msee-hypothesese-architecturee-sampling
5.3 Read sample efficiency with its training budget attached
The useful comparison is not simply video model versus VLA as broad categories. The paper builds a π₀.₅-style baseline using PaliGemma, FAST backbone training and an action decoder matching mimic-video's design. That controls an important downstream architectural factor, but the pretrained backbones and learning objectives still differ. Figure 5 reduces the action-decoder dataset; it does not demonstrate a matching reduction in robot-video adaptation or all pretraining. The real-world experiment makes that distinction concrete: its small task-specific action sets coexist with a 200-hour video-adaptation corpus. Reader interpretation: the evidence supports reusing video representations to reduce action supervision, while a total-data-efficiency claim would need all stages counted. Likewise, Figure 6 counts decoder updates with a common batch size; the abstract's twofold convergence summary is not a measured wall-clock result. e-baselinee-efficiencye-real-setupe-traininge-convergence
5.4 Training and inference
During training
First adapt the video model with LoRA on robotics videos; then freeze it and train the action decoder from scratch. Algorithm 2 independently corrupts ground-truth future video and actions and optimizes action-flow regression. These are disjoint parameter stages, not joint end-to-end action training. e-training
Table IV specifies video/action learning rates 1.778e-4/1e-4 with constant/linear schedules, AdamW, 1000 warmup steps, weight decay 0.1 and gradient clipping 10. LIBERO uses batch size 128, 7k–8k video-adaptation steps and 50k decoder steps. e-hyper
During inference
Initialize future-video and action latents with Gaussian noise. Integrate video to the selected time, extract layer-k features, then integrate actions fully to time zero. At video time one, video integration disappears: one backbone feature pass still conditions iterative action generation. No measured control frequency or latency is provided. e-sampling
Bimanual executions use this time-one setting. Figure 4 separately decodes full videos for illustration; those frames are not generated during autonomous execution. The precise chunk-execution and replanning schedule is unspecified. e-real-setupe-sampling
5.5 Implementation flow
- Encode observed and future video
The 2B Cosmos-Predict2 DiT consumes clean context latents from a pretrained 3D tokenizer and noisy future latents. Self-attention spans the video sequence; T5-encoded instructions enter through cross-attention. Video prediction is not conditioned on candidate robot actions. e-architecturee-motivation
- Read internal video features
Extract hidden states h at layer k rather than decoded frames. Appendix C favors k = 19 and five context frames over one, but gives no numerical layer/horizon ablation table. e-architecturee-layer
- Translate features into actions
A separate DiT embeds proprioception and noisy actions with MLPs, adds learned temporal positions, cross-attends to video features and self-attends over actions. AdaLN receives both flow times. Random proprioceptive-token masking during training discourages overreliance on that input. e-architecture
6. Experiments & results
mimic-video turns a robot-adapted video generator into features for a separate action policy. Its default setting combines observed frames with entirely noisy future latents, avoiding video reconstruction while retaining full action denoising. Experiments support strong decoder-data efficiency and manipulation performance, with task-dependent noise tuning and limited deployment evidence (e-architecture, e-sampling, e-efficiency, e-limits).
6.1 Read the original evidence
Table I. Default and task-tuned performance answer different evaluation questions. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the last column first, then return to the four task columns to see where the average comes from. The setting is SIMPLER simulation with a Widow-X policy trained on BridgeDataV2. In the paper's terminology, scratch means target-only robot action training; every row still benefits from image or video pretraining. The matched π₀.₅-style row is therefore the most direct decoder comparison. Keep the final row separate: it changes the inference-time video-noise setting for each task. The table includes the input header, but the training-regime definitions are supplied in the original caption and summarized here. e-benchmarkse-baselinee-simpler
What it supports. Default mimic-video averages 46.9%, versus 35.4% for the matched VLA: an 11.5-percentage-point difference. Per-task tuning raises the reported average to 56.3%. Gains are uneven: the default reaches 100.0% on Eggplant but only 12.5% on Stack Blocks, below the matched VLA's 20.8% there.
Where the evidence stops. The published baseline rows use different robot-action pretraining regimes. The source gives neither uncertainty nor the validation protocol used to select task-specific noise levels, so the tuned row is not an untuned generalization estimate.
Table II. mimic-video improves the matched comparison while remaining below OpenVLA-OFT. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read across the Spatial, Object and Goal columns before comparing the averages. These are three separate LIBERO suites with ten tasks each; the supplied experiment does not include LIBERO-Long. The nominal source dataset provides fifty demonstrations per task, with failed action-replay episodes removed during preprocessing. Compare mimic-video with the π₀.₅-style row to assess the paper's matched decoder experiment, then inspect OpenVLA-OFT to locate the strongest published comparator in this table. The scratch and finetuned labels refer to robot-action training history, not whether the visual backbone started from random weights. e-benchmarkse-preprocesse-baselinee-libero
What it supports. mimic-video reports 94.2%, 96.8% and 90.6% across the three suites, averaging 93.9%. Its average is 8.0 percentage points above the matched VLA's 85.9%. OpenVLA-OFT reports 96.9%, so this visual supports a strong result under the target-only action regime, not an unconditional best score.
Where the evidence stops. The comparison mixes action-data histories and provides no uncertainty estimates. The matched baseline is the authors' π₀.₅-style construction; these numbers should not be presented as a direct evaluation against the original fully pretrained π₀.₅.
Table III. Physical execution results favor mimic-video in the two reported columns. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Begin with the column names exactly as printed: Packing and Package handover. They differ from the prose's Tape Stowing and Package Sorting names, and the report retains that difference rather than silently mapping them. The first two rows use a DiT-Block policy with workspace-only input and then added wrist cameras. mimic-video uses the workspace view despite the physical setup also carrying wrist cameras. The setup comprises two Panda arms with dexterous mimic hands. Its video model was adapted on a 200-hour corpus, whereas the sorting and stowing action decoders use 512 and 480 task episodes, respectively. e-real-setupe-real-results
What it supports. The reported scores rise from 11.0/30.0 for workspace-only DiT-Block and 42.6/74.1 with wrist cameras to 72.0/93.0 for mimic-video. This is evidence about robot execution in a focused bimanual setting. Figure 4 separately labels fully decoded videos as visualization, not an input generated during autonomous execution.
Where the evidence stops. Table III prints no unit label, rollout counts or uncertainty; values are retained as scores without adding a percent sign. The method comparison changes more than camera access, and the table-to-prose task mapping remains unstated.
Figure 5. The strongest efficiency claim concerns action-decoder demonstrations. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the horizontal axis as a fraction of data used to train the action decoder, not the fraction of all pretraining data. At each fraction, blue is mimic-video and orange is the matched VLA. The vertical axis is success as a fraction, whereas the result tables print percentages. The prose supplies the exact low-data anchor: 2% corresponds to one episode per task and gives 77% average success across the three evaluated LIBERO suites. For the tenfold claim, compare the blue 10% bar with the orange 100% bar; the authors describe them as reaching the same performance level. e-efficiencye-traininge-benchmarks
What it supports. The large low-data separation supports the claim that video-model features make action decoding data efficient in this setup. The exact textual result is 77% success with one demonstration per task. The roughly tenfold comparison concerns similar achieved performance, not a claim that every bar differs by a fixed multiplicative factor.
Where the evidence stops. Robot-video adaptation and Internet-scale backbone pretraining are outside the varied decoder-data fraction. The plot has no uncertainty bars; exact equality between the 10% and 100% comparison should not be inferred from bar heights.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| SIMPLER-Bridge four-task manipulation BridgeDataV2-trained Widow-X policy in SIMPLER; target-only action training with pretrained visual backbones. | 46.9 default; 56.3 with per-task video-time tuning Average success rate (%) | Matched pi0.5-style VLA: 35.4; published FLOWER: 45.0 under a different action-data regime. Default performance gains 11.5 percentage points over the matched VLA. Tuning adds 9.4 points, with extra computation and an unspecified tuning-validation protocol. e-benchmarkse-baselinee-simpler |
| LIBERO Spatial, Object and Goal Three simulated Panda suites, each with ten tasks and fifty demonstrations per task before preprocessing. | Spatial 94.2; Object 96.8; Goal 90.6; average 93.9 Success rate (%) | Matched VLA average 85.9; OpenVLA-OFT 96.9 with external action pretraining. An 8.0-point gain over the matched VLA, but not the highest published score in Table II. LIBERO-Long is not evaluated. e-benchmarkse-liberoe-preprocess |
| Real-world Packing and Package handover Bimanual Panda arms with mimic hands; workspace-view mimic-video. Video adaptation uses 200 hours; sorting/stowing decoders use 512/480 episodes. | Packing 72.0; Package handover 93.0 Reported benchmark score; Table III does not explicitly label units | DiT-Block workspace-only: 11.0/30.0; with wrist cameras: 42.6/74.1. Retains Table III labels. Prose instead names Tape Stowing and Package Sorting; the exact column mapping is not explicitly documented. These are robot-execution scores. e-real-setupe-real-results |
| LIBERO action-decoder data efficiency Decoder data subsets of 2%, 10%, 50% and 100% across the three evaluated suites. | 77% success at 2% data (one episode per task); author-reported approximately 10x decoder-data efficiency Average success and action-data fraction | The authors describe 10%-data mimic-video as reaching the full-data matched VLA level. This does not establish a tenfold reduction in video pretraining, robot-video adaptation or total compute. e-efficiencye-training |
| Decoder convergence Both decoders use batch size 128 and individually optimized learning rates. | Author-claimed 2x faster convergence; Figure 6 shows earlier high success Success versus training steps | Matched VLA improves more slowly and plateaus lower. The ratio is an author summary, not measured wall-clock speedup or a precisely specified time-to-threshold statistic. e-convergence |
| Effect of video denoising on action quality SIMPLER predicted-video rollouts and a separate held-out BridgeDataV2 ground-truth-video diagnostic. | High-noise predicted features perform well; oracle-feature MSE is lowest around video time 0.4. Rollout success; separately, action reconstruction MSE | Near-clean conditioning is worse in the diagnostic; task-specific rollout peaks differ. Figure 7’s intermediate-peak caption conflicts with the prose’s universal-sounding time-one claim. Reconstruction MSE and autonomous success are distinct protocols. e-noisee-oracle-msee-simpler |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 7. Useful control features occur at high noise, with task-dependent peaks. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each panel separately: the vertical ranges differ substantially, so equally tall curves do not imply equal success. The horizontal labels are video noise levels; the caption describes the axis as logit-scaled. Equation (1) fixes the direction: zero is clean video and one is pure noise. These are rollout results with model-predicted future conditioning, not the ground-truth-video MSE experiment in Figure 8. Carrot, cubes and spoon show peaks before their rightmost samples, while eggplant improves toward high noise. Avoid extracting exact optimum times from plot spacing; Table I provides the separate task-tuned performance values. e-noisee-flowe-samplinge-simplere-oracle-mse
What it supports. The shared observation is that fully resolved video is unnecessary for good control, and nearly clean predicted conditioning can perform poorly. The reliable practical conclusion is to treat video time as a task-dependent inference parameter. A high-noise feature pass can be useful even when its future input has not been converted into a viewable plan.
Where the evidence stops. The caption's intermediate-noise peaks and Table I's tuning gains qualify the prose on pages 8–9 claiming best performance at τᵥ = 1. Preserve this discrepancy; the source does not establish a universal optimum or report uncertainty.
7. Analysis & limitations
7.1 What the evidence leaves open
The authors acknowledge a fixed single view, narrow real-world task coverage and no unified cross-embodiment model. e-limits
Matching decoders helps comparison, but changing video versus VLM pretraining also changes backbone size, training objective and pretraining corpus. Tables lack uncertainty and rollout counts; performance alone does not establish causal physical understanding. e-baselinee-simplere-liberoe-real-results
Oracle future-video access is privileged. Near-perfect oracle success and intermediate-noise MSE support representation usefulness; distribution-mismatch and hidden-state explanations remain hypotheses. e-oraclee-oracle-msee-hypotheses
7.2 Questions for discussion
- Does held-out task tuning retain the reported gain from selecting video noise?
- How much decoder-data efficiency survives when robot-video adaptation data is reduced simultaneously?
8. Reproducibility audit
8.1 Requirements and known gaps
Reproduction needs the stated backbone/tokenizer, adaptation videos, action demonstrations and simulator/robot interfaces. Preserve 480×640 images, six-dimensional rotations, chunk-relative poses, gripper/hand conventions and the documented BridgeDataV2 and LIBERO filtering. e-architecturee-real-setupe-preprocess
The text omits LoRA rank, decoder dimensions, numerical action horizon, solver step counts, execution frequency, GPU counts/models and software versions. Table IV supplies optimizer settings, not a complete executable configuration. A project link does not establish dataset or code availability. e-architecturee-samplinge-traininge-real-setupe-hyper
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Test video-time selection without evaluating on the tuning trials
Reader-proposed check: hold the trained BridgeDataV2 backbone and decoder fixed and compare τᵥ = 1, a preregistered intermediate-time grid, and a nearly clean endpoint on the four SIMPLER tasks. Choose each task's time using separate validation initializations, then evaluate on untouched initializations with repeated seeds. Hold action solver settings, observations and execution horizon constant; report success, uncertainty, backbone calls and measured latency. Run the noisy-ground-truth-video MSE sweep separately as an oracle diagnostic. If the 56.3% versus 46.9% tuning advantage disappears on held-out rollouts, the claimed tuning benefit weakens; stable intermediate optima would qualify a universal time-one recommendation. e-simplere-noisee-oracle-msee-sampling
Check 2: Separate action-data savings from robot-video adaptation
Reader-proposed check: on the three reported LIBERO suites, repeat the 2%, 10% and 100% action-decoder subsets with multiple independently drawn demonstration subsets. Cross this with a fixed full-video adaptation condition and a reduced-video condition that excludes action-held-out episodes. Compare the matching π₀.₅-style decoder using the same action splits and success evaluation. Keep preprocessing, observation history and decoder capacity fixed; report video and action budgets separately. Measure whether one-episode performance remains near the reported 77% and whether 10%-data mimic-video remains comparable to full-data VLA. A large collapse only after video data is reduced would localize the efficiency benefit to action supervision rather than total robot data. e-efficiencye-baselinee-traininge-preprocesse-layer
8.3 Reading coverage
Visual audit: Visually inspected the title/authors/version and affiliations on page 1; Figures 1–8; Tables I–IV; Eqs. (1)–(3); Algorithms 1–2; method, evaluation, limitations and Appendix A–E pages supporting the retained claims. All six final original crops were individually viewed; the Figure 7 crop was widened to retain complete axis labels. Pages 10–13 contain references and were read as text. All seven supplied text chunks were read completely. No separate supplement, code or external project material was inspected. Figure 7's caption/prose discrepancy and Table III's missing units/task-label mismatch remain explicit.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 14, 15. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Title and abstract
- I. Introduction
- II. Related Work
- III. Case Study: How Does Video Generation Quality Affect Robot Policy Performance?
- IV. Video-Action Models (A–F), Algorithms 1–2
- V. Experiments (A–C)
- VI. Discussion and Future Work
- Acknowledgments and References
- Contributions
- Appendix A–E, including Table IV
Outside the original text pass
- Reviewed edition: arXiv:2512.15692v2, 19 December 2025; title and all six authors match. The catalog submission date is 17 December 2025. The earlier version was not supplied, so revision changes were not compared (e-identity).
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- The extraction limitation was addressed by inspecting original PDF pages 1–9 and 14–15, including all eight figures and four tables. Reference-only pages 10–13 were read as text, not rendered.
- No code, project website or separate supplement was inspected, and no experiments were reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e-identityPDF p. 1, title block and arXiv margin stamp
Title and six authors match the catalog. The artifact is arXiv:2512.15692v2, dated 19 December 2025. Affiliations are mimic robotics, Microsoft Zurich, ETH Zurich, ETH AI Center and UC Berkeley.
Go to primary source ↓e-motivationPDF pp. 1–2, Abstract and Section I; p. 2, Section II(c)
The authors motivate video pretraining as temporal and visual-dynamics priors and explicitly consider video models without action conditioning.
Go to primary source ↓e-oraclePDF p. 3, Section III and Figure 2
Ground-truth future-video conditioning gives near-perfect success with both pretrained and robot-finetuned backbones; finetuning improves predicted-video conditioning. Detailed rollout counts and uncertainty are absent.
Go to primary source ↓e-flowPDF p. 3, Section IV-A, Eqs. (1)–(2); p. 4, Eq. (3)
Flow matching interpolates clean data at zero with Gaussian noise at one, regresses toward noise minus clean data, and samples by integrating from one to zero.
Go to primary source ↓e-architecturePDF p. 4, Figure 3 and Sections IV-B–D
A 2B Cosmos-Predict2 video DiT with a pretrained 3D tokenizer and T5 conditioning takes five context frames and noisy future latents. A separate action DiT cross-attends to intermediate video states and processes proprioception and actions with independent flow-time conditioning.
Go to primary source ↓e-samplingPDF p. 5, Section IV-E and Algorithm 1, lines 2–6
Video integration stops at a chosen time before layer-k features condition full action denoising. At video time one, integration is skipped and one backbone feature pass suffices. Numerical latency and control frequency are not supplied.
Go to primary source ↓e-trainingPDF p. 5, Section IV-F and Algorithm 2
LoRA video adaptation precedes action-decoder training with the backbone frozen. Independently sampled times corrupt ground-truth future video and actions. Video times follow a logit-normal distribution.
Go to primary source ↓e-benchmarksPDF p. 5, Section V(a), Evaluation setups
SIMPLER evaluates BridgeDataV2-trained Widow-X policies. LIBERO uses Goal, Object and Spatial, each with ten tasks and fifty demonstrations per task.
Go to primary source ↓e-baselinePDF p. 6, Section V(b), Comparisons
The matched baseline uses 3B PaliGemma, FAST next-token backbone training and an identical flow action decoder on equivalent target datasets. It is pi0.5-style, not original fully pretrained pi0.5. Published baselines use distinct action-data regimes.
Go to primary source ↓e-real-setupPDF pp. 5–6, Section V(a); p. 6, Figure 4; p. 7, Section V-A(c)
Two Panda arms carry 16-DoF mimic hands. Video adaptation uses 200 hours; sorting/stowing decoders use 512 episodes/1 h 33 min and 480 episodes/2 h 14 min. mimic-video uses the workspace view. Figure 4 distinguishes execution from videos decoded only for visualization.
Go to primary source ↓e-simplerPDF p. 7, Table I, all rows and Average SR column
Average success is 46.9% for default mimic-video, 56.3% with per-task time tuning and 35.4% for matched VLA. Default task scores are 37.5, 37.5, 12.5, 100.0%; tuned scores are 54.2, 41.7, 29.2, 100.0%.
Go to primary source ↓e-liberoPDF p. 7, Table II, mimic-video, pi0.5-style VLA and OpenVLA-OFT rows
mimic-video Spatial/Object/Goal scores are 94.2/96.8/90.6%, average 93.9%. Matched VLA averages 85.9%; OpenVLA-OFT averages 96.9% under a different action-pretraining regime.
Go to primary source ↓e-real-resultsPDF p. 7, Table III and Section V-A(c)
Packing/Package handover scores are 72.0/93.0 for mimic-video, 11.0/30.0 for workspace-only DiT-Block and 42.6/74.1 with wrist cameras. Table III gives no unit label, trial counts or uncertainty.
Go to primary source ↓e-efficiencyPDF p. 7, Section V-B and Figure 5; p. 8, Section V-B continuation
The authors report 10%-data mimic-video reaching the full-data VLA decoder level and one demonstration per task (2%) yielding 77% average success. The experiment varies decoder training data.
Go to primary source ↓e-convergencePDF p. 8, Figure 6 and Section V-B; p. 1, Abstract
Figure 6 plots decoder success against steps at batch size 128 and individually optimized learning rates. The abstract summarizes convergence as 2x faster; wall-clock speedup is not quantified.
Go to primary source ↓e-noisePDF p. 8, Section V-C and Figure 7 with caption; p. 9, opening continuation
Curves favor high noise over nearly clean predicted video. The caption describes intermediate-noise peaks, while prose claims best performance at time one and later calls it the highest average. A universal per-task optimum is not established.
Go to primary source ↓e-oracle-msePDF p. 8, Figure 8 and Section V-C, right column
Held-out BridgeDataV2 action reconstruction from noisy ground-truth video features has lowest MSE at an intermediate time, described as approximately 0.4. Both endpoints are worse.
Go to primary source ↓e-limitsPDF p. 9, Section VI, final paragraph
Authors identify the single fixed workspace view, absence of a unified cross-embodiment model and narrow real-world task set as limitations.
Go to primary source ↓e-preprocessPDF p. 14, Appendix B(a–c)
Images are 480×640; rotations use the top two matrix rows. Actions use chunk-relative end-effector poses and embodiment-specific hand/gripper representations. BridgeDataV2 removes 3046 uninformative labels and initial state/null actions; LIBERO removes demonstrations failing action replay.
Go to primary source ↓e-layerPDF p. 14, Appendix C–D
The authors favor video layer 19 and five observation frames over one; VLA features favor layer 11. No numerical layer/horizon ablation table is provided.
Go to primary source ↓e-hyperPDF p. 15, Table IV
Video/action learning rates are 1.778e-4/1e-4 with constant/linear schedules. AdamW, 1000 warmup steps, weight decay 0.1 and gradient clipping 10 are shared. LIBERO video/action training uses 7k–8k/50k steps with batch size 128.
Go to primary source ↓e-hypothesesPDF pp. 14–15, Appendix E(a–b)
The authors propose predicted-video distribution mismatch and changing hidden-state information across denoising as explanations. These interpret oracle/noise findings rather than independently isolate the mechanisms.
Go to primary source ↓8.5 Primary sources
mimic-video: Video-Action Models for Generalizable Robot Control Beyond VLAs ↗
PDF · 10,380 extracted words
Source fingerprint
a7d2d3fa9367921bde900889e862025afa1e28e93437f287aa018b3b17e39658