VideoVLA: Video Generators Can Be Generalizable Robot Manipulators
1. Paper overview
In one sentence: Jointly predicting video latents and robot actions transfers useful video priors into manipulation, with strong transfer results but substantial execution gaps and inference latency. e-identitye-architecturee-sim-objectse-real-skillse-duale-motione-speed
| At a glance | What to know |
|---|---|
| Research problem | Source description Can a pretrained video generator transfer its visual knowledge into robot control? The paper studies instruction-conditioned manipulation with unfamiliar objects or skills absent from the target robot’s demonstrations. Those skills can still occur in another embodiment’s training data; this is a specific transfer setting, not unrestricted acquisition of unseen skills. e-identitye-datae-real-skills |
| Core mechanism | Source description A single multimodal diffusion transformer jointly predicts continuous actions and future video latents. The video decoder is optional for visualization, while latent video prediction remains part of policy inference. e-formulatione-architecture |
| A key reported result | Real-world cross-embodiment skill transfer: 58.0%. Reported average success rate across six task columns. WidowX-plus-Realman training; familiar Realman objects with skills demonstrated only by WidowX. CogACT 35.1%; π0 28.5%. VideoVLA leads all six columns, but Take Out Block reaches 37.5% and Wipe Table 41.7%. These are physical executions under this restricted transfer protocol. e-real-skillse-trials |
| Reading caution | Reader analysis The motion-similarity diagnostic uses actual execution video, so it is retrospective. Human-rated imagined success is 84.0%/63.4% for novel objects/new skills versus executed 65.2%/48.6%; these match the simulation generalization results. Plausible video is neither proof of successful action nor a demonstrated pre-execution confidence estimator. e-motione-imaginede-sim-objectse-sim-skills |
Core contributions
- Source description
A single multimodal diffusion transformer jointly predicts continuous actions and future video latents. The video decoder is optional for visualization, while latent video prediction remains part of policy inference. e-formulatione-architecture
- Reader analysis
The authors attribute generalization to video pretraining and dual prediction. Backbone, loss, horizon, attention and diffusion-schedule ablations support the design, but do not fully isolate physical understanding from representation and optimization benefits. e-backbonee-horizone-duale-attentione-schedule
Figure 2. Language and the current image condition one transformer that predicts both kinds of future. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Begin on the left: the text encoder yields green language tokens, the video VAE yields yellow frame latents, and the recorded robot trajectory supplies blue seven-dimensional actions. The right panel separates clean conditions from noisy targets. Language and the first-frame latent enter as conditions; future latents and actions enter together as noisy inputs. Follow the upward arrows through the diffusion transformer to the paired denoised outputs. The leftward arrow from video latents reaches the pink optional decoder. That branch produces viewable frames; actions already emerge from the shared transformer. Section 3.3 explains why the first latent contains only the observed frame. e-formulatione-architecturee-attention
What it supports. The architecture makes video prediction part of action inference, rather than merely a training target discarded at deployment. It also avoids requiring an inverse-dynamics module to recover actions from rendered frames. The optional decoder does not make the latent video branch optional: the default denoiser still generates both output modalities.
Where the evidence stops. The diagram labels latent positions with N, while Section 3.3 distinguishes N frames from n latents. Preserve that notation discrepancy when interpreting horizons. Figure 2 does not draw the attention mask; Appendix B establishes default bidirectional video/action attention.
2. Motivation
2.1 The problem and the proposed response
Can a pretrained video generator transfer its visual knowledge into robot control? The paper studies instruction-conditioned manipulation with unfamiliar objects or skills absent from the target robot’s demonstrations. Those skills can still occur in another embodiment’s training data; this is a specific transfer setting, not unrestricted acquisition of unseen skills. e-identitye-datae-real-skills
2.2 What this reading follows
A video generator can imagine an instruction being completed, but a robot still needs executable motions. VideoVLA adds continuous actions to a pretrained video diffusion transformer and learns to denoise the two futures together. Read the architecture first to see why generated images are optional while video latents remain part of control. Then compare unfamiliar-object pickup and skills transferred from another embodiment. The ablation asks whether video supervision actually matters; the final diagnostic asks how imagined motion relates to executed motion. Across these views, keep the evaluation boundary explicit: a plausible imagined future, an aggregate success score and completion of a physical task are different observations. e-identitye-architecturee-sim-objectse-real-skillse-duale-motione-speed
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Not assigned |
| Architecture | Not assigned |
| Prediction paradigm | Not assigned |
| Quadrant | Not assigned |
This table preserves the labels recorded at reading time. The current major category is WAMs. View the current classification.
3.1 Evidence-based assessment
Insufficient evidence to decide
The catalog snapshot is entirely unassigned, so there is no substantive label to confirm. Architecture evidence supports a One Model assessment with joint future-video/action prediction: one transformer exchanges information bidirectionally and denoises both modalities during inference. Actions are generated directly, rather than extracted by a separate inverse-dynamics model. This is a reader assessment; the recorded snapshot is preserved. e-formulatione-architecturee-attentione-schedule
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Turn video prediction into a policy without first rendering a plan
The key interface is the action vector. Section 3.1 assigns three components to wrist rotation, three to wrist translation, and one to gripper state. These continuous vectors enter the same transformer as the video latents after projection to a shared embedding dimension. Training supplies complete clips and action trajectories, but the causal VAE keeps the current-observation latent independent of later frames. During inference, language and that observation latent remain conditions while action and video targets are jointly denoised. The robot executes the first three of six predicted actions, obtains another observation and repeats. Decoding the imagined frames is useful for inspection, but the default policy can send the directly predicted actions without rendering those frames. This distinction explains how one generator supplies both a visual forecast and a control output. e-formulatione-architecturee-training
5.2 Read the ablations as separate tests of prior, supervision and interaction
The ablations probe different parts of the design. Table 7 compares pretrained CogVideoX-5B with the same named backbone trained from scratch: the three-task average is 80.4% versus 12.6%. Table 8 changes the visual horizon, increasing the reported average from 75.2% with 13 frames to 80.4% with 49. Table 9 removes supervision or the video branch. Appendix B then changes information flow: preventing video tokens from attending to actions reduces the average from 80.4% to 75.5%, while asynchronous training with synchronous inference reaches 73.8%. A reader interpretation is that initialization, supervised forecasting and reciprocal denoising each contribute. However, these are separate interventions rather than a complete factorial study; they do not establish that one isolated physical-reasoning mechanism explains every gain or its interactions with compute. e-backbonee-horizone-duale-attentione-schedule
5.3 Separate transfer, imagined success and completed physical tasks
The transfer experiments hold out a skill from the target embodiment, while allowing that skill in another embodiment’s demonstrations. Novel-object pickup tests a different axis: the skill is familiar and the objects change. These conditions should not be collapsed into a single notion of open-world competence. Likewise, the real-world in-domain aggregate combines multiple tasks and, for Place, averages pickup and placement stages; a successful pickup can therefore coexist with weak final placement. The imagination diagnostic adds another distinction. Human judges assess instruction following and visible plausibility, whereas actual execution must satisfy the task. The motion-similarity analysis goes further by comparing predicted and executed trajectories, but only after execution. My interpretation is that VideoVLA demonstrates useful transfer and coherent forecasting while leaving prospective failure prediction and consistently successful physical completion unresolved. e-datae-sim-objectse-real-skillse-real-domaine-motione-imagined
5.4 Training and inference
During training
Robot pretraining uses an OXE subset of 22.5 million frames. Real-world adaptation uses 5,824 teleoperated samples of pick, stack and place collected with a Realman seven-DoF arm and gripper. Simulation evaluation uses OXE-trained models; real-world skill transfer combines WidowX data and the collected Realman data. e-datae-real-skills
The reported recipe uses 100K pretraining and 15K fine-tuning iterations, batch size 256, 32 AMD MI300X GPUs, and AdamW with learning rate 1e-5 and weight decay 1e-4. The backbone is CogVideoX-5B. e-training
During inference
Predict six actions, execute the first three, obtain a fresh observation and repeat. The paper describes 13 video latents/49 frames for simulation and four latents/13 frames for real deployment. Actions and frames need not share a frequency; decoding images is unnecessary for action execution. e-formulatione-architecturee-training
Main-text implementation details specify 50 DDIM steps. Appendix D instead specifies 10 steps for the real-world timing configuration: approximately 1.1 seconds on one H100 GPU, yielding around 3 Hz effective control. The report preserves both settings rather than assigning one to every result. e-traininge-speed
5.5 Implementation flow
- Encode the conditions
T5 maps the instruction to 226 tokens. The CogVideoX causal video VAE encodes a training clip; its first latent represents only the current observation. At inference, that image alone supplies the first latent. This prevents future frames from entering the observation condition. e-formulation
- Denoise both futures in one transformer
Flatten spatial video latents in raster order, concatenate language, current-image latent, noisy future latents and noisy actions, and project modalities to a common embedding dimension. The CogVideoX-initialized transformer uses self-attention, diffusion-timestep adaptive LayerNorm and DDPM denoising supervision on video and actions. e-architecture
- Allow reciprocal information flow
Default attention is bidirectional between action and video tokens. The appendix’s causal-mask variant permits actions to attend to video but prevents video from attending to actions. The default uses a shared diffusion timestep and synchronous denoising for both modalities. e-attentione-schedule
6. Experiments & results
VideoVLA adapts CogVideoX-5B into a robot policy that jointly denoises future video latents and executable action chunks, conditioned on language and the current image. Its clearest gains concern novel objects and skills transferred between embodiments. Video supervision matters strongly in the reported ablations, but imagined success exceeds executed success and deployment remains slow.
6.1 Read the original evidence
Table 2. Novel-object pickup reveals a larger separation than the in-domain aggregate. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each object column as a separate Google-robot pickup condition in SIMPLER, then use the far-right average to compare methods across all ten objects. The objects come from YCB and GSO and are absent from the Google robot’s training data. The paper reports 25 trials per novel object in Appendix A. In the Ours row, bold entries identify the strongest scores, while underlined entries mark second place. Compare against SpatialVLA as well as CogACT: SpatialVLA is the stronger average baseline here. Inspect cleaner bottle and toy airplane to see where the overall ordering reverses. e-sim-objectse-trials
What it supports. VideoVLA reports 65.2% average success versus SpatialVLA’s 50.8%, a 14.4 percentage-point difference. It leads on eight of ten objects. The two exceptions matter: cleaner bottle is 44.0% versus 56.0%, and toy airplane is 28.0% versus 32.0%, so generalization is uneven across object conditions.
Where the evidence stops. “Novel” is defined relative to the target robot’s training data, not an audited absence from every upstream video-pretraining source. This is simulated pickup with a fixed skill. The table provides no confidence intervals or repeated-seed uncertainty.
Table 6. The target robot transfers skills that were demonstrated by a different embodiment. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the training boundary: this experiment combines WidowX demonstrations with the collected Realman dataset. The tested skills occur in WidowX data but are absent from Realman demonstrations; the manipulated objects are familiar to Realman. Read the six columns as physical tasks, not six unfamiliar objects. Two columns instantiate moving with different objects, while the remaining columns test grabbing, toppling, taking out and wiping. Compare Ours with CogACT, the strongest listed average baseline. Finally inspect the weakest Ours columns before reading the average, because a gain in transfer does not imply dependable completion of every skill. e-real-skillse-trials
What it supports. The reported average rises from CogACT’s 35.1% to VideoVLA’s 58.0%, a 22.9 percentage-point difference. VideoVLA leads every task column. Yet performance spans 81.3% for moving a block down to 37.5% for taking out a block, with wiping at 41.7%; transfer remains far from uniformly reliable.
Where the evidence stops. These are embodiment-held-out skills, not skills absent from all training. Appendix A states 16 trials per new real-world skill, but 41.7% cannot be a single integer-success fraction of 16. The paper leaves the additional aggregation or count explanation unresolved.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| SIMPLER in-domain manipulation OXE-trained models; WidowX Visual Matching (VM) and Google VM/Variant Aggregation (VA); four task groups per setting. | 63.0% overall; WidowX VM 53.1%, Google VM 73.1%, Google VA 62.8%. Reported average success rate | CogACT: 62.6% overall. WidowX VM ties π0 at 53.1%; CogACT leads Google VM at 75.2%. Competitive in-domain performance, not a win in every setting. The comparison removes π0 proprioception; no uncertainty supports a significance claim for the small overall margin. e-sim-domaine-trials |
| SIMPLER novel-object pickup Google robot; ten YCB/GSO objects absent from its training data; Appendix A reports 25 trials per object. | 65.2%. Average success rate | SpatialVLA 50.8%; CogACT 42.4%. A 14.4 percentage-point gain over the strongest listed baseline; VideoVLA leads on eight objects, but trails SpatialVLA on cleaner bottle and toy airplane. e-sim-objectse-trials |
| SIMPLER cross-embodiment skill transfer Google executes skills represented in WidowX data but absent from Google training demonstrations. | 48.6%. Reported average success rate | CogACT 20.4%. A 28.2 percentage-point transfer gain. Table 3 groups directional sliding in one column; the appendix’s stated 20 trials per skill does not explain every reported fraction. e-sim-skillse-trials |
| Real-world in-domain manipulation Realman; OXE pretraining and collected-data fine-tuning; pick, stack and place. | 64.6%. Reported Task (All) average success rate | CogACT 58.4%; π0 50.7%. This aggregate mixes task groups and stage scores. For Place, VideoVLA reports 87.5% pickup and only 25.0% placement, whose stage average is 56.3%; 64.6% is not end-to-end placement success. e-real-domain |
| Real-world novel-object manipulation Realman pickup-and-place instruction; 12 novel objects represented by 13 table conditions, including two orientations of the black bottle. | 50.6%. Reported average success rate | CogACT 26.9%. A substantial average gain with varied difficulty; pen success remains 16.7%. Appendix A states 12 trials per novel object. e-real-objectse-trials |
| Real-world cross-embodiment skill transfer WidowX-plus-Realman training; familiar Realman objects with skills demonstrated only by WidowX. | 58.0%. Reported average success rate across six task columns | CogACT 35.1%; π0 28.5%. VideoVLA leads all six columns, but Take Out Block reaches 37.5% and Wipe Table 41.7%. These are physical executions under this restricted transfer protocol. e-real-skillse-trials |
| Dual-prediction ablation Google SIMPLER VM, three in-domain task groups; generalization follows Tables 2–3. | Default: 80.4% / 65.2% / 48.6%. Average success rate: in-domain / novel objects / new skills | No video loss: 27.0% / 12.7% / 4.4%; action only: 25.5% / 11.3% / 2.1%. Keeping video supervision is decisive in this setup. The three-task 80.4% average is different from Table 1’s four-task Google VM average. e-duale-sim-domain |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 9. Removing video supervision sharply degrades both familiar-task execution and transfer. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the rows as three distinct learning setups. Default predicts video and actions and applies denoising losses to both. No video loss retains joint modeling but supervises only actions. Action only removes future-video prediction and trains action denoising alone. The left block evaluates three Google-robot Visual Matching task groups; its average excludes the fourth Google task group in Table 1. The right block reuses the novel-object and new-skill protocols from Tables 2 and 3. Comparing Default with No video loss is especially informative because merely retaining video variables without their denoising target does not preserve the reported performance. e-duale-sim-domaine-sim-objectse-sim-skills
What it supports. In-domain success falls from 80.4% to 27.0% without video loss and to 25.5% with action-only prediction. The same ordering holds for novel objects, 65.2% versus 12.7% and 11.3%, and new skills, 48.6% versus 4.4% and 2.1%. The gains therefore extend beyond the familiar-task average in this experiment.
Where the evidence stops. The table supports the value of video supervision in this implementation. It does not fully separate learned dynamics from regularization or optimization effects. Removing the video branch also changes computation; training-cost matching, loss normalization and repeated-seed variability are not established here.
Figure 3. Successful executions tend to resemble their imagined motions more closely. Original paper, p. 10 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the horizontal axis as motion similarity between an imagined video and the corresponding actual execution. The vertical groups indicate successful and failed executions, not predicted success probability. Orange/red marks belong to successful trials; blue marks belong to failures. The two panels aggregate SIMPLER tasks separately for Google and WidowX robots. Section 4.4 builds the similarity by finding SIFT keypoints, retaining SAM-segmented foreground points, tracking them with SAM-PT, matching trajectories with the Hungarian algorithm and averaging normalized cosine similarities. Follow the relative horizontal positions of the success and failure distributions; the figure does not supply a numerical correlation coefficient. e-motione-imaginede-sim-objectse-sim-skills
What it supports. For both embodiments, the displayed successful executions occupy higher-similarity regions than failures. This supports an association between imagined/actual motion agreement and task outcome. The companion human-evaluation table also shows the gap: imagined success reaches 84.0% for novel objects and 63.4% for new skills, while execution reaches 65.2% and 48.6%.
Where the evidence stops. Similarity requires the execution video, so this is a retrospective diagnostic. It neither demonstrates causal improvement from a better imagination nor provides a deployable pre-action reliability threshold. Human judgments of semantic and physical plausibility are also distinct from environment-measured task completion.
7. Analysis & limitations
7.1 What the evidence leaves open
The motion-similarity diagnostic uses actual execution video, so it is retrospective. Human-rated imagined success is 84.0%/63.4% for novel objects/new skills versus executed 65.2%/48.6%; these match the simulation generalization results. Plausible video is neither proof of successful action nor a demonstrated pre-execution confidence estimator. e-motione-imaginede-sim-objectse-sim-skills
Latency limits feedback speed. Figure 2 labels latents with N, whereas the formulation distinguishes N frames and n latents; implementation prose also calls the reported latent counts “future” despite the separate first-frame condition. Exact horizon indexing and per-table use of 10 versus 50 denoising steps need clarification. e-formulatione-architecturee-traininge-speed
Tables report no confidence intervals or seed variability. Appendix A lists 16 trials per real-world new skill, yet Wipe Table’s 41.7% is not an integer-success fraction of 16. Raw counts or additional aggregation details are needed; the reported percentages should not be reverse-engineered into trial outcomes. e-real-skillse-trials
7.2 Questions for discussion
- Would video supervision retain its benefit under matched-compute training and repeated seeds? (e-dual)
- Can an imagination-only signal predict failure before execution, without using actual execution trajectories? (e-motion)
- Which denoising-step and video-horizon settings best trade task completion against feedback latency? (e-training, e-speed)
8. Reproducibility audit
8.1 Requirements and known gaps
Reproduction requires the exact OXE mixture, aligned video/action sampling, CogVideoX/T5/VAE components, collected Realman demonstrations, and SIMPLER setups. The supplied paper does not fully specify mixture weights, action normalization, camera preprocessing, frozen-module policy, loss weighting, software versions or random seeds. e-formulatione-architecturee-datae-training
Proposed checks: repeat the dual-loss comparison with matched initialization, data and compute; separately measure success and latency at 10 and 50 DDIM steps under the same deployment horizon. Record seeds, trial-level outcomes and complete-stage success rather than relying only on rounded averages. e-duale-traininge-speede-real-domain
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Does video supervision help under matched training conditions?
Reader-proposed, not performed: reproduce Default and No video loss using identical CogVideoX initialization, OXE examples, video/action horizons, attention, optimizer updates and sampling steps. Keep the video tokens in both variants to control sequence length and architecture; report wall-clock training cost and modality-loss normalization. Repeat multiple seeds, then evaluate the same three Google VM task groups plus the novel-object protocol with trial-level outcomes and confidence intervals. A persistent advantage for Default would support a benefit from video supervision beyond merely retaining video variables. If the gap disappears under these controls, the strong mechanistic interpretation of Table 9 would weaken. e-architecturee-traininge-duale-trials
Check 2: Resolve the denoising-step tradeoff at the deployment horizon
Reader-proposed, not performed: use one checkpoint and an H100 to compare 10 and 50 DDIM steps with the reported real-world horizon of four video latents/13 frames, six predicted actions and three executed actions. First document whether the first-observation latent is counted in that horizon. Hold camera input, task starts, control interface and video-decoder use fixed. Measure complete-task success, separate Place pickup/placement success, and median and tail inference latency across repeated trials. The test should show whether faster sampling preserves completion while improving feedback frequency; a large success loss or latency above the reported timing would reject that operating-point assumption. e-formulatione-traininge-speede-real-domaine-trials
8.3 Reading coverage
Visual audit: The title/byline/version page, method and training pages, every quantitative table, the motion diagnostic, and all qualitative figure pages were visually inspected. Pages 15–16 support trial counts, attention direction, diffusion schedules and deployment timing. All five final crops were inspected for readability and context. Figure 2’s N-versus-n notation discrepancy is disclosed; its decoder arrows agree with the caption and method. The full text, including introduction, related work and references, was read; their remaining pages were not included in this visual pass. Static frame panels do not constitute viewing continuous demonstrations. Separate supplements and code remain uninspected.
PDF pages inspected for this edition: 1, 4, 5, 6, 7, 8, 9, 10, 15, 16, 17, 18. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract (p. 1)
- 1 Introduction (pp. 2–3)
- 2 Related Works (pp. 3–4)
- 3 Methodology, Sections 3.1–3.4 (pp. 4–5)
- 4 Experiment, Sections 4.1–4.4 (pp. 5–10)
- 5 Conclusion (p. 10)
- Acknowledgments and References (pp. 11–14)
- Appendix A Evaluation Details (p. 15)
- Appendix B More Analysis (pp. 15–16)
- Appendix C More Visualizations (pp. 16–18)
- Appendix D Limitations and Broader Impacts (p. 16)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- The title page identifies arXiv:2512.06963v1 [cs.RO], 7 December 2025, and NeurIPS 2025. The title and all eight authors match the catalog; no other revision or edition was supplied or compared.
- Acquisition omission: text extraction does not reconstruct figure images. The retained PDF was therefore visually inspected for figures, tables and mathematical layout.
- Separate supplemental material availability has not been fully verified. No separate supplement was supplied.
- Code, external project material and continuous demonstration videos were not inspected; no experiments were reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e-identityPDF p. 1, title block, arXiv margin, venue footer and Abstract
The title and eight-author byline match the supplied catalog. The artifact is arXiv:2512.06963v1, 7 Dec 2025, labeled NeurIPS 2025; affiliations are IAIR, Xi’an Jiaotong University, Microsoft Research Asia and Fudan University.
Go to primary source ↓e-formulationPDF p. 4, Sections 3.1–3.4, unnumbered action/video definitions and encoder descriptions
Defines instruction/current-image conditioning, seven-dimensional actions, separate action/frame rates, 226 T5 tokens and the causal VAE’s first-observation latent; distinguishes N frames and n latents.
Go to primary source ↓e-architecturePDF p. 5, Figure 2 and caption; Section 3.4 continuation
One DiT receives conditions and noisy video/action inputs, predicts both, and uses an optional visualization decoder. Prose specifies raster flattening, shared embedding projection, DDPM losses, adaptive LayerNorm and CogVideoX initialization. Figure latent labels use N.
Go to primary source ↓e-dataPDF p. 5, Section 4, Dataset and Evaluation paragraphs
The OXE subset has 22.5 million frames; collected Realman data has 5,824 samples of pick/stack/place. Distinguishes simulation-only OXE training from real-world fine-tuning and target-embodiment skill/object generalization.
Go to primary source ↓e-trainingPDF p. 6, Implementation Details
Specifies CogVideoX-5B, 100K/15K iterations, 32 MI300X GPUs, batch 256, AdamW settings, 50-step DDIM, simulation 13 latents/49 frames, real-world four latents/13 frames, and six predicted/three executed actions.
Go to primary source ↓e-sim-domainPDF p. 6, Table 1, Ours/CogACT/π0 rows and caption; Section 4.1 In-Domain Evaluation
Table 1 reports Ours 53.1/73.1/62.8 across WidowX VM/Google VM/Google VA and 63.0 overall; CogACT overall is 62.6 and Google VM 75.2. π0 lacks proprioception in this comparison. VM matches appearance; VA varies it.
Go to primary source ↓e-sim-objectsPDF p. 6, Table 2, all rows and object columns; PDF p. 7, Section 4.1 novel-object continuation
Google pickup uses ten objects absent from its training data. Ours averages 65.2 versus SpatialVLA 50.8 and CogACT 42.4; SpatialVLA is stronger on cleaner bottle and toy airplane.
Go to primary source ↓e-sim-skillsPDF p. 7, Table 3 and Section 4.1 Generalization to New Skills
WidowX-to-Google transfer has reported averages 48.6 for Ours and 20.4 for CogACT. The prose says eight skills while the table uses seven task columns with directional sliding grouped.
Go to primary source ↓e-real-domainPDF p. 7, Table 4 and caption; Section 4.2 In-Domain Evaluation
Realman aggregate scores are 64.6 for Ours and 58.4 for CogACT. Place reports pickup and placement separately: 87.5 and 25.0 for Ours, averaging 56.3.
Go to primary source ↓e-real-objectsPDF p. 8, Table 5 and caption; Section 4.2 novel-object continuation from p. 7
Realman novel-object task averages 50.6 for Ours and 26.9 for CogACT; pen is 16.7. The prose states 12 objects, while the table has 13 conditions, including upright/horizontal black bottle and upright yellow bottle.
Go to primary source ↓e-real-skillsPDF p. 8, Table 6 and Section 4.2 Generalization to New Skills
Training combines WidowX and Realman data. Skills are new to Realman while objects are familiar. Ours averages 58.0 versus CogACT 35.1 and π0 28.5; Take Out Block is 37.5 and Wipe Table 41.7.
Go to primary source ↓e-backbonePDF p. 8, Table 7 and Section 4.3 Backbone; p. 9 continuation
Google VM three-task averages: pretrained CogVideoX-5B 80.4, scratch CogVideoX-5B 12.6, pretrained OpenSora-1.1 50.2.
Go to primary source ↓e-horizonPDF p. 8, Table 8; PDF p. 9, Section 4.3 Time Horizon
Reported averages increase from 75.2 to 77.4 to 80.4 with 13, 25 and 49 frames; prose maps these to four, seven and thirteen latents.
Go to primary source ↓e-dualPDF p. 9, Table 9 and Section 4.3 Dual-Prediction Strategy
Default, no-video-loss and action-only variants score 80.4/27.0/25.5 in-domain, 65.2/12.7/11.3 on novel objects, and 48.6/4.4/2.1 on new skills. No-video-loss retains joint modeling but supervises actions only.
Go to primary source ↓e-motionPDF p. 9, Section 4.4 Motion Similarity; PDF p. 10, Figure 3 and caption
SIFT keypoints filtered by SAM foreground masks are tracked with SAM-PT, paired using Hungarian matching and compared by mean normalized cosine trajectory similarity. Figure 3 groups actual successes/failures across SIMPLER tasks for Google and WidowX.
Go to primary source ↓e-imaginedPDF p. 9, Table 10; PDF p. 10, Section 4.4 Task Performance Comparison continuation
Humans assess instruction following and absence of salient geometry/physics violations. Imagined novel-object/new-skill success is 84.0/63.4 versus execution 65.2/48.6; no judge count or agreement statistic is provided.
Go to primary source ↓e-trialsPDF p. 15, Appendix A, Tables 11–12
Reports Google VM counts 300/240/216/108; Google VA 825/600/378/189; WidowX VM 24 each; simulation novel objects 25 each and new skills 20 each. Real-world pick/stack/place counts are 24/48/24, novel objects 12 each, new skills 16 each.
Go to primary source ↓e-attentionPDF p. 15, Appendix B Causal masking vs. bidirectional attention; PDF p. 16, Table 13
Causal masking lets action tokens attend video tokens but disallows the reverse. Google VM averages are 80.4 for default and 75.5 for causal masking.
Go to primary source ↓e-schedulePDF p. 16, Appendix B Asynchronous noising and inference; Table 14
Default is synchronous training/inference with shared modality timesteps. Async training with sync inference scores 73.8; async training with staged video-then-action inference scores 71.0; default scores 80.4.
Go to primary source ↓e-speedPDF p. 16, Appendix D Limitations and Broader Impacts
Ten-step DDIM deployment predicts four video latents and six actions, executes three, and takes approximately 1.1 seconds on one H100 for around 3 Hz effective control. Acceleration methods are future directions.
Go to primary source ↓e-qualitativePDF p. 10, Figure 4; PDF pp. 17–18, Figures 5–6 and captions
Selected real-world and simulation frame sequences juxtapose imagined futures with corresponding executions; these qualitative examples are not independent success-rate measurements.
Go to primary source ↓8.5 Primary sources
VideoVLA: Video Generators Can Be Generalizable Robot Manipulators ↗
PDF · 9,320 extracted words
Source fingerprint
a00c753d5ce1b6211647d72d1768b382c8c3bd91abeb837b0b4f0f98fa5daf6a