VidMan: Exploiting Implicit Dynamics from Video Diffusion Model for Effective Robot Manipulation
1. Paper overview
In one sentence: VidMan transfers video-diffusion features into a shared-backbone action policy, improving reported manipulation results while avoiding iterative video generation during control. e-rationalee-videoe-actione-calvine-ablationse-oxe
| At a glance | What to know |
|---|---|
| Research problem | Source description Robot demonstrations are scarce and heterogeneous. The authors seek to transfer temporal knowledge from robot videos into precise actions while avoiding the latency and pixel-prediction errors of explicitly generating a future video before applying a separate inverse-dynamics model. e-rationalee-action |
| Core mechanism | |
| A key reported result | CALVIN zero-shot five-instruction chains: 3.42 tasks; 46.7% Average completed chain length; five-task success rate. OXE pretraining; language-labeled CALVIN A/B/C adaptation; test on D with static/wrist cameras and proprioception. GR-1: 3.06 and 40.1%; 3D Diffuser Actor: 3.35 and 41.2%. The authors report 11.7% relative average-length improvement over GR-1. Different pretraining, modalities and All/Lang data access prevent isolating the adapter’s contribution. No uncertainty is given here. e-calvin |
| Reading caution |
Core contributions
- Reader analysis
The authors attribute improved data use to implicit inverse dynamics. The ablations support specific training and architectural choices, but do not identify physical state variables or prove an inverse-dynamics factorization. e-idme-ablations
Figure 2. One video backbone supports sequential video learning and action adaptation. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow panel (a) upward: a sampled robot trajectory becomes history and future video tokens. In panel (b), the green dashed box combines two channel groups: clean historical tokens with zero future slots, and a noised trajectory. Language and diffusion step k also enter Open-Sora. The upward first-stage branch predicts noise for the video loss. The second-stage branch sends learned action queries through the layer-wise adapter to action embeddings and the action head. Cross-reading Eq. (4) and Section 4.2 is essential: these are successive training stages, and stage two fixes k at K rather than repeatedly denoising video. Snowflakes appear on both tokenizers. e-videoe-actione-settingse-ablations
What it supports. The useful transfer occurs through pretrained parameters and intermediate features. The action branch can exploit what video training encoded while avoiding explicit future-frame generation. In the preferred second-stage configuration, action supervision updates the backbone and adapters, so the visual prediction loss is not retained as an auxiliary control objective.
Where the evidence stops. The drawing is schematic about the action decoder. Eq. (6) noises V_action rather than explicitly specifying a demonstrated-action target and conditioning interface. That ambiguity remains unresolved; the diagram cannot supply the missing tensor-level definition.
2. Motivation
2.1 The problem and the proposed response
Robot demonstrations are scarce and heterogeneous. The authors seek to transfer temporal knowledge from robot videos into precise actions while avoiding the latency and pixel-prediction errors of explicitly generating a future video before applying a separate inverse-dynamics model. e-rationalee-action
2.2 What this reading follows
A robot policy can benefit from learning how scenes evolve without generating a movie every time it acts. VidMan explores this idea in two stages: learn robot-video denoising on OXE, then adapt intermediate features into action embeddings using gated attention. The resulting policy keeps the video backbone but changes its job. This reading follows that transition through the architecture, examines the adapter’s information flow, and separates simulated task completion from offline action accuracy. The strongest evidence comes from CALVIN comparisons and ablations; source inconsistencies and incomplete decoder details limit what can be reconstructed from the paper alone. e-rationalee-videoe-actione-calvine-ablationse-oxe
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | WAMs |
| Architecture | Dual-system |
| Prediction paradigm | IDM |
| Quadrant | Q4 · Dual-system × IDM |
3.1 Evidence-based assessment
Conflicts with the recorded classification
WAM relevance and the authors’ implicit-IDM designation are supported. The recorded Dual-system architecture overstates separation: Figures 2 and 5 and Eq. (5) show a shared video backbone transformed into an action policy with adapters and a decoder. Two training stages and the System 1/System 2 analogy do not establish separately operating models. Inference is integrated, without an explicit future-video-to-IDM pipeline or joint future/action output. The catalog snapshot remains unchanged. e-rationalee-videoe-actione-adapter
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Why video denoising can train an action policy indirectly
Start with the distinction between a visual observation and a state useful for control. Section 3 motivates inverse dynamics through an action conditioned on a state transition, but VidMan does not recover an explicit physical state sequence with a separate estimator. Instead, stage one teaches the transformer to predict noise added to robot-video latents while conditioning on historical images and language. Its features therefore must help predict how the visual trajectory evolves. Stage two reuses those parameters to construct action embeddings. As a reader interpretation, this is a transfer-learning route to a dynamics-sensitive policy: the training target supplies temporal pressure, while the deployed action module need not expose or generate the intermediate future. Calling this implicit inverse dynamics expresses that design intent, not a measured recovery of physical dynamics. e-idme-videoe-action
Figure 5. Layer-wise gated attention turns backbone features into action embeddings. Original paper, p. 15 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read from the bottom token row upward. Green tokens denote Open-Sora block outputs, pink tokens denote state information, and yellow tokens denote learned action queries. The drawing connects their combined stream to Q, K and V at a self-attention block. Its output passes through a tanh gate and a residual addition, then through the feed-forward block, another tanh gate and another residual addition. Yellow output tokens represent the action-query pathway refined by these operations. Section 4.2 places an adapter after each video-transformer layer, allowing the policy to draw on intermediate representations rather than only a final visual feature. e-adaptere-actione-ablations
What it supports. The adapter provides an explicit architectural connection between video features and action prediction. Table 2(c) reports average CALVIN chain length 1.54 without the layer-wise adapter, 2.98 when the backbone is frozen, and 3.42 when it is trained. This supports the chosen connection within the tested configurations.
Where the evidence stops. Section 4.2 describes proprioceptive tokens as extra keys/values, whereas this schematic uses a common Q/K/V input bus. Exact routing, masks and gate initialization are not resolved by the figure. Its residual arrows should not be read as a separately specified control loop.
5.2 Follow the inference input instead of the training movie
At inference, the clean historical condition still carries the observed scene. What changes is the other channel group: the noisy trajectory input is fixed at the maximum video diffusion step and becomes Gaussian noise. Learned action queries collect features from the adapted backbone in one pass, and the small diffusion head converts their embeddings into actions. There is no generated future-image sequence that must finish before an independent inverse-dynamics model can begin. This explains why two-stage training does not imply two separately running systems. It also identifies a reproduction hazard: the source specifies the maximum action diffusion step but not a complete sampling schedule, and Eq. (6) does not clearly distinguish the demonstrated action being denoised from the conditioning embedding. Those details require clarification before implementation. e-actione-settingse-adapter
5.3 Use the ablations to separate useful features from attractive videos
The main CALVIN comparison establishes task performance under its reported protocol, but several design choices change across baselines. Table 2 moves closer to the mechanism: after video pretraining, retaining the video loss during action adaptation reduces performance, and removing layer-wise adaptation also hurts. Table 3 adds a complementary observation: increasing frame interval improves control up to the preferred setting even as video FID and FVD worsen. A reader inference is that the most useful temporal representation for immediate actions need not optimize the same ranking as video generation. This does not establish that video fidelity is irrelevant. Interval changes both history and future distance, and action-only training changes supervision. A discriminating follow-up should hold those other factors fixed before assigning the gain to an inverse-dynamics bias. e-calvine-ablationse-spacinge-history
5.4 Training and inference
During training
Initialize from Open-Sora’s 16×256×256 checkpoint and extend its input embedder with zero weights for conditioning channels. Stage one uses 25 selected OXE datasets, sampling two history and two future frames at interval 3, with third-person images at 256×256 and maximum diffusion step K=1000. e-settings
Stage two uses 224×224 third-person and wrist images, predicting 12 actions on OXE or 10 on CALVIN. Its action diffusion maximum is L=100. The best configuration updates backbone and adapters using action supervision alone; freezing the backbone is an ablated alternative. e-settingse-ablations
During inference
Set the video timestep to K and supply Gaussian noise alongside the clean historical condition. One backbone pass produces action embeddings without iterative video denoising. The separate diffusion head still decodes actions; its sampling schedule and the number of actions executed before feedback are unspecified. e-actione-settings
Simulation and recorded-action evaluation support distinct conclusions. Generated video and offline action traces do not establish physical deployment success. e-calvine-oxe-protocole-rlbenche-qual
5.5 Implementation flow
- Encode and condition trajectories
Tokenize history and future images into V_h and V_f. Noise the full trajectory embeddings, then channel-concatenate them with clean history padded by zero future embeddings. Language and the diffusion timestep also condition Open-Sora. Figure 2 marks the video and text tokenizers with snowflakes. e-video
- Adapt intermediate dynamics features
After each video-transformer layer, concatenate learnable action tokens with its features. Self-attention and a feed-forward network, each with tanh gating and residual addition, produce action embeddings. Proprioception can supply additional keys and values. Figure 5 depicts a combined token stream without an explicit attention mask. e-actione-adapter
- Decode actions
A small diffusion head converts embeddings to low-level controls. The source describes executable end-effector/gripper signals, but Eq. (6) noises action embeddings rather than explicitly showing a demonstrated-action target and its conditioning; the exact training tensor interface remains ambiguous. e-action
6. Experiments & results
VidMan adapts a video diffusion transformer into a language-conditioned robot policy. Video prediction on OXE supplies pretrained dynamics features; gated adapters and a diffusion action head turn those features into action sequences without generating future images during control. Evidence supports improved simulated manipulation and offline prediction, while the latent inverse-dynamics interpretation remains an architectural inductive bias.
6.1 Read the original evidence
Table 1. VidMan’s strongest main-text result is longer successful CALVIN instruction chains. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. First inspect Training Data: Lang means only language-labeled CALVIN data, while All includes visual data without language annotations. Then read the numbered columns as success percentages for completing that many tasks in a row; Avg. Len. summarizes chain completion. VidMan’s shaded row reaches 3.42 average tasks and 46.7% at five tasks. The relevant GR-1 row has 3.06 and 40.1%. The 3D Diffuser Actor row has a slightly lower average, 3.35, but higher one- and two-task success than VidMan. These experiments train on CALVIN environments A, B and C and test zero-shot on D. e-calvine-settings
What it supports. The paper reports an 11.7% relative improvement in average chain length over GR-1. The table also supports a narrower conclusion than uniform superiority: VidMan has the highest average among these rows and stronger longer-chain completion, while 3D Diffuser Actor performs better on the first two success columns.
Where the evidence stops. Lang/All describes CALVIN access, not total pretraining data or compute. Baselines also differ in sensory inputs and architecture. No uncertainty appears in Table 1, so small differences cannot be assigned statistical significance from this source.
Table 7. Offline improvements appear across four OXE domains, with a rounding discrepancy in the reported UR5 gain. Original paper, p. 16 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Treat each dataset block as its own comparison. The MSE column measures action prediction error; xyz and angle columns report thresholded accuracies, and their average is the final column. VidMan has the lowest MSE and highest displayed average accuracy in all four blocks. For example, Cable Routing changes from Octo-base’s 14.9 average accuracy and 7.5 MSE to VidMan’s 24.8 and 3.3. Bridge changes from 49.8 and 2.2 to 55.4 and 0.8. These are in-distribution evaluations against recorded actions after OXE training, so read improvements as offline prediction evidence rather than completed manipulation tasks. e-oxee-oxe-protocole-spacinge-noise
What it supports. Using the displayed averages, VidMan’s gains over Octo-base are 5.6, 2.6, 9.9 and 8.9 percentage points for Bridge, Taco Play, Cable Routing and Autolab UR5. The last difference comes from 33.2 minus 24.3. The paper’s caption instead reports 9.0%, which the rounded table does not reproduce exactly.
Where the evidence stops. The source does not supply exact held-out trajectory splits, uncertainty or a fully reproducible accuracy implementation. MSE normalization is unspecified. Table 3 and the smaller frozen-backbone Table 5 report different Bridge configurations and values; they should not be merged with these rows.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| CALVIN zero-shot five-instruction chains OXE pretraining; language-labeled CALVIN A/B/C adaptation; test on D with static/wrist cameras and proprioception. | 3.42 tasks; 46.7% Average completed chain length; five-task success rate | GR-1: 3.06 and 40.1%; 3D Diffuser Actor: 3.35 and 41.2%. The authors report 11.7% relative average-length improvement over GR-1. Different pretraining, modalities and All/Lang data access prevent isolating the adapter’s contribution. No uncertainty is given here. e-calvin |
| OXE in-distribution offline action prediction OXE stage-two training; offline Bridge, Taco Play, Cable Routing and Autolab UR5 evaluation. | Bridge 55.4 / 0.8; Taco Play 66.4 / 1.0; Cable Routing 24.8 / 3.3; Autolab UR5 33.2 / 3.4 Average xyz/angle accuracy (%), higher better; action MSE, lower better | Octo-base respectively: 49.8 / 2.2; 63.8 / 1.9; 14.9 / 7.5; 24.3 / 8.6. Rounded accuracy differences are 5.6, 2.6, 9.9 and 8.9 percentage points, calculated from Table 7. The caption calls the last gain 9.0%. These are offline accuracy differences, not relative percentages or task-success gains. e-oxe-protocole-oxe |
| CALVIN training and adapter ablations Table 2 CALVIN finetuning experiments. | Action-only 3.42; OXE pretraining 3.42; unfrozen backbone with adapter 3.42 Average completed chain length | Co-training video/action losses 2.70; no video pretraining 2.89; Ego4d 3.29; no adapter 1.54; frozen backbone 2.98. Variants support sequential specialization, robotics pretraining and layer-wise adaptation. VidMan-GPT also changes prediction horizon and objective, so it is not a single-factor diffusion test. e-ablationse-calvin |
| Frame interval and control utility Two historical plus two future frames; intervals 1–4; Bridge prediction and CALVIN control. | Interval 3: FID 38.4, FVD 376, MSE 0.89, CALVIN 3.42 Bridge FID/FVD/MSE; CALVIN average chain length | Interval 1: 29.5, 327, 2.80, 2.24; interval 4: 51.9, 422, 1.20, 3.03. Better video distribution scores do not track better action performance here. Interval changes both historical coverage and future distance, so the experiment does not isolate their contributions. e-spacing |
| RLBench-100 multi-task manipulation 18 tasks; agents trained with 100 demonstrations; 25 evaluation episodes per task, with five-seed reporting described in the appendix. | 67.4 Average success rate (%) | Act3D 65.1; RVT 62.9; PerAct 42.7. Simulation evidence extends beyond CALVIN. Per-task ± quantities are called variance in the prose; aggregate uncertainty is absent. Inference-speed units and hardware are unspecified, preventing a control-frequency claim. e-rlbench |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 2(a–c). Action-only adaptation, OXE pretraining and an unfrozen adapted backbone give the best reported settings. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each panel independently; every CALVIN number is average completed chain length, and gray marks the preferred setting. Panel (a) changes second-stage supervision: co-train includes video and action losses, whereas action-only removes video supervision after pretraining. Panel (b) compares no video pretraining, Ego4d video pretraining and OXE video pretraining. Panel (c) changes how action queries access the backbone and whether backbone weights update. The no-adapter variant inserts action tokens before Open-Sora blocks instead of using the layer-wise modules. Repeated 3.42 entries identify the shared preferred configuration rather than three independent replications of it. e-ablationse-config
What it supports. Adding video loss in stage two lowers the reported average from 3.42 to 2.70. OXE pretraining exceeds no video pretraining by 0.53 tasks, calculated from 3.42 minus 2.89. Updating the backbone with adapters also improves on the frozen variant’s 2.98. Together these comparisons favor specialization after pretraining.
Where the evidence stops. The prose’s 0.13 Ego4d decrease is relative to OXE; Ego4d actually exceeds no-pretraining in this table. Loss weighting, compute matching and uncertainty are insufficiently documented to conclude that auxiliary video prediction must always harm control.
Table 3. The best action setting does not have the best video scores. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the arrows before comparing rows: lower FID, FVD and MSE are preferred, while higher xyz accuracy, angle accuracy and average chain length are preferred. All settings use two historical and two future frames; the interval controls how far apart the sampled frames are. Moving from interval 1 to 3 raises CALVIN average length from 2.24 to 3.42 and reduces Bridge action MSE from 2.80 to 0.89. Yet FID rises from 29.5 to 38.4 and FVD from 327 to 376. Interval 4 worsens both the listed video scores and the action metrics relative to interval 3. e-spacinge-history
What it supports. This table is direct evidence that better video-distribution scores need not imply a better manipulation policy. A larger temporal span can provide more useful action features even when future images become harder to predict. That explanation is a reader interpretation consistent with the authors’ discussion, not a separately isolated causal result.
Where the evidence stops. Changing interval simultaneously expands history and moves future targets farther away. Appendix Table 6 investigates frame counts, but its neighboring prose reverses the history/future symbols. Neither experiment cleanly identifies which temporal component causes the interval effect.
7. Analysis & limitations
7.1 What the evidence leaves open
Authors identify absent 3D perception, limited complex-instruction understanding and coarse whole-image perception. Qualitative video examples lose details such as occluded objects. e-limitse-qual
Source inconsistencies remain: A.1 contrasts the model with CLIP although A.2 specifies CLIP. Table 6 uses m for history and n for future, while adjacent prose reverses them. The Ego4d discussion’s 0.13 decrease is relative to OXE, not no-pretraining. Figure 4 plots iterations, not elapsed time, and does not substantiate the prose’s faster frozen-backbone convergence claim. e-limitse-confige-historye-videoe-ablations
OXE lacks exact held-out trajectory splits and uncertainty. Its xyz-accuracy description combines 0.5 radians with 50% of motion norm, insufficient to reconstruct a precise implementation. Evidence does not establish real-robot robustness. e-oxe-protocole-oxee-qual
7.2 Questions for discussion
- Would action-only adaptation still win with matched auxiliary-loss scales and training compute? (e-ablations)
- Does temporal coverage help through history, future targets, or both when varied independently? (e-spacing, e-history)
8. Reproducibility audit
8.1 Requirements and known gaps
Appendix A.2 specifies VideoAutoencoderKL, CLIP and reduced STDiT-XL/2: 12 layers, 16 heads, width 1152. Table 4 gives batch sizes 384/1440/400, learning rates 2e-5/1e-4/1e-4 and durations 100k steps/300k steps/20 epochs for stage one/OXE stage two/CALVIN stage two. All configurations use AdamW, dropout 0.1 and weight decay 0.1; Table 4 lists no stage-one LR schedule and inverse-square-root decay for stage two. e-config
Hardware is 16 V100 32GB GPUs for each OXE stage and 8 for CALVIN. The source states 42 hours for stage one and 32 for OXE stage two, which accumulates five batches of 18 per GPU. Preparation excludes missing-image/non-delta-control datasets, reweights diversity, zero-pads missing cameras and maps gripper open/closed to +1/0. e-confige-data
Mixture membership/weights, exact checkpoints, software versions, action normalization, decoder architecture and sampling/feedback settings need clarification. A proposed small check is the frozen-backbone Bridge noise-input ablation: batch 192, 100k iterations, identical data. Pure noise/zeros/unnoised images yield reported MSE 4.8/12.1/11.2. Keep these reduced-training results separate from Table 7. e-actione-confige-datae-noise
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Does pure noise transfer useful pretrained features?
Reader-proposed check, not executed: reproduce the reduced Bridge protocol in Appendix A.4 using a frozen stage-one backbone, batch size 192 and 100k stage-two iterations. Compare pure-noise, pure-zero and unnoised-image inputs while holding clean history, trajectory split, action head, initialization seeds and optimizer settings fixed. Include a same-architecture backbone without stage-one robot-video pretraining as a control. Report MSE and both accuracy components over repeated seeds, after resolving the accuracy definition. Table 5’s pure-noise advantage should recur under the documented reduced setup; disappearance of that advantage, or an equal gain with the control backbone, would weaken the explanation that noise specifically extracts pretrained dynamics knowledge. e-noisee-datae-oxe-protocole-action
Check 2: Is action-only adaptation better because of loss interference?
Reader-proposed check, not executed: start every CALVIN variant from the same OXE stage-one checkpoint and use the same language-labeled A/B/C data, observations, action horizon, trainable parameters and optimization budget. Compare action-only training with action-plus-video training across a declared video-loss weight sweep. Keep a zero-weight control to verify identical pipelines. Evaluate identical instruction chains in D over repeated seeds, recording average length and five-task success with uncertainty. If a nonzero video-loss weight matches or exceeds action-only performance, Table 2(a)’s gap would support sensitivity to objective balance rather than a general requirement to remove video supervision. If the gap persists across matched settings, the sequential-specialization explanation becomes stronger. e-ablationse-calvine-settingse-config
8.3 Reading coverage
Visual audit: The title, all method/equation pages, training and evaluation settings, all original Figures 1–7 and Tables 1–8, appendix hardware/hyperparameters, data preparation and qualitative examples were visually inspected. Six final original crops were inspected individually; the CALVIN and OXE crops were corrected to exclude caption fragments and inspected again. Figure 2 was checked against Eqs. (4)–(6), and Figure 5 against Section 4.2, including gates, residual arrows and the unresolved state-token Q/K/V routing. Figure 4’s iteration axis and curve labels were checked against its convergence claim. Reference pages 10–13 were read as text only; separate supplements and code were outside this reading. All retained method, numerical and reproduction details are supported on the declared pages.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 14, 15, 16, 17, 18, 19. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract and title block (p. 1)
- 1 Introduction (pp. 1–2)
- 2 Related Work and 3 Preliminaries (p. 3)
- 4 Method, 4.1 Dynamics-aware Visionary Stage, 4.2 Dynamics-modulated Action Stage (pp. 3–5)
- 5 Experiment, 5.1 Settings, 5.2 Comparisons, 5.3 Ablations (pp. 5–9)
- 6 Conclusion (p. 9)
- Acknowledgements and References (pp. 10–13)
- A.1 Negative Impacts and Limitations, A.2 Network and Training Details (pp. 14–15)
- A.3 OXE data details and A.4 Additional Experimental Results (pp. 15–17)
- A.5 Visualization (pp. 17–19)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- All six supplied text chunks were read individually. PDF images were inspected on pages 1–9 and 14–19; reference pages 10–13 were read as text.
- Identity notes: the inspected title and all seven authors match the catalog. The PDF explicitly identifies arXiv:2411.09153v1 [cs.CV], 14 November 2024, and NeurIPS 2024. This report concerns that supplied v1 artifact; equivalence to the separately listed proceedings edition or any later revision was not established.
- No code, model checkpoints, external project resources or separate supplements were inspected; no experiments were reproduced. The abstract promises future code/model release, which does not establish present availability.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e-identityPDF p. 1, title block, arXiv margin and conference footer
The exact catalog title and seven authors appear with Shenzhen Campus of Sun Yat-Sen University, Pengcheng Laboratory and Huawei Noah’s Ark Lab affiliations. The artifact states v1, 14 November 2024, NeurIPS 2024; the abstract promises code and models.
Go to primary source ↓e-rationalePDF p. 2, Figure 1 and Introduction, two-stage description
Video prediction pretraining transfers dynamics through shared architecture and parameters into action prediction without explicitly generated future trajectories.
Go to primary source ↓e-idmPDF p. 3, Section 3, Eqs. (1)–(3)
Inverse dynamics is P(a | s_t, s_{t+1}); states are conceptual latent quantities inferred from observations. These preliminaries motivate the method, rather than specify an independently trained state estimator.
Go to primary source ↓e-videoPDF p. 4, Figure 2 and Section 4.1, Eq. (4)
History m and future n frames are encoded. Clean history padded with zero futures conditions noisy trajectory latents through channel concatenation. Text and timestep condition denoising; tokenizer snowflakes appear in the diagram.
Go to primary source ↓e-actionPDF p. 5, Section 4.2, Eqs. (5)–(6)
Adapters combine learned action queries with each VDT layer; proprioception supplies extra keys/values. At k=K the noisy branch is Gaussian and iterative video denoising is disabled. The head decodes controls, but Eq. (6) explicitly noises V_action and omits a distinct demonstrated-action target.
Go to primary source ↓e-settingsPDF p. 6, Section 5.1.1, both training-stage paragraphs
Open-Sora initialization, 25 OXE datasets, two history/two future frames at interval 3, resolutions 256 and 224, K=1000, L=100 and 12/10 action horizons are specified.
Go to primary source ↓e-calvinPDF pp. 6–7, Sections 5.1.2 and 5.2, Table 1, VidMan/GR-1/3D Diffuser Actor rows; p. 7 VidMan-GPT paragraph
CALVIN trains on A/B/C and tests D with five-instruction chains. Table 1 gives 3.42/3.06/3.35 average lengths and 46.7/40.1/41.2 five-task success. VidMan uses language-labeled CALVIN, both cameras and proprioception. The GPT baseline changes horizon and reconstruction objective.
Go to primary source ↓e-oxe-protocolPDF p. 6, Section 5.1.2 Offline Evaluation; p. 7, Section 5.2 Offline Performance
Offline OXE evaluation is in-distribution on four datasets. Average xyz/angle accuracy and action MSE are reported; xyz tolerance is worded as 0.5 radians and 50% of motion norm; rotation tolerance is 0.5 radians. Exact validation split identifiers are absent.
Go to primary source ↓e-ablationsPDF p. 8, Table 2(a–c) and Section 5.3; p. 9, pretraining/adapter paragraphs and Figure 4
Co-train/action-only scores are 2.70/3.42; no-pretrain/Ego4d/OXE 2.89/3.29/3.42; no-adapter/frozen/unfrozen 1.54/2.98/3.42. Co-train adds video loss in stage two. Figure 4 labels frozen orange and unfrozen blue and plots training iterations; blue is generally lower despite the frozen-convergence assertion.
Go to primary source ↓e-spacingPDF p. 9, Table 3, interval 1–4 rows and frame-sampling discussion
Interval 1 gives FID/FVD/MSE 29.5/327/2.80 and CALVIN 2.24; interval 3 gives 38.4/376/0.89 and 3.42; interval 4 gives 51.9/422/1.20 and 3.03. Historical coverage and future distance both change.
Go to primary source ↓e-limitsPDF p. 14, Appendix A.1
Authors describe 2D-only perception, limited instruction understanding and missing fine-grained object inputs. A.1’s CLIP comparison conflicts with A.2’s explicit CLIP text encoder.
Go to primary source ↓e-configPDF p. 14, Appendix A.2, Model details, Training details and Table 4
VideoAutoencoderKL, CLIP, STDiT-XL/2, 12 layers, 16 heads and width 1152 are specified. Table 4 supplies optimization settings; prose gives V100 counts, memory, accumulation and OXE durations. Stage-one schedule is None in the table despite the prose’s general decay description.
Go to primary source ↓e-adapterPDF p. 15, Figure 5 and Appendix A.2 Layer-wise adapter
Open-Sora outputs, state tokens and action queries join the input. Q/K/V arrows arise from the combined stream. Attention and feed-forward blocks use tanh gating and residual addition; action tokens appear at the top. No mask or gate-initialization value is shown.
Go to primary source ↓e-dataPDF p. 15, Appendix A.3
Selection filters image streams and delta end-effector control, describes qualitative diversity reweighting without complete mixture membership/weights, zero-pads missing camera channels and standardizes gripper +1=open, 0=closed.
Go to primary source ↓e-noisePDF p. 15, Appendix A.4 Why use pure noise; p. 16, Table 5
Bridge-only frozen-VDT ablation uses 100k iterations and batch 192. No-noise/pure-noise/pure-zero MSE is 11.2/4.8/12.1; pure-noise xyz/angle accuracy is 32.7/37.6.
Go to primary source ↓e-historyPDF p. 16, Table 6 and historical/future-frame discussion; p. 4, Section 4.1
Table 6 labels m as history and n as future, consistent with Section 4.1 but reversed in adjacent prose. Rows (1,1), (2,1), (2,2) give CALVIN 2.09/2.80/3.42 and Bridge MSE 4.20/1.80/0.89.
Go to primary source ↓e-oxePDF p. 16, Table 7, VidMan/Octo-base rows across four dataset blocks; p. 8, Figure 3 and caption
VidMan average accuracy is 55.4/66.4/24.8/33.2 and MSE 0.8/1.0/3.3/3.4. Octo-base accuracy is 49.8/63.8/14.9/24.3 and MSE 2.2/1.9/7.5/8.6. Captions call gains 5.6%, 2.6%, 9.9%, 9.0%; displayed UR5 averages differ by 8.9 points.
Go to primary source ↓e-rlbenchPDF p. 16, Appendix A.4 Results on RLBench; p. 17, Table 8 and following paragraph
18 tasks, 25 episodes per task and agents trained with 100 demonstrations are specified. Five seeds and variance are mentioned. Aggregate successes are 67.4/65.1/62.9/42.7 for VidMan/Act3D/RVT/PerAct; inference-speed units, hardware and aggregate uncertainty are absent.
Go to primary source ↓e-qualPDF p. 17, Appendix A.5; p. 18, Figure 6; p. 19, Figure 7
Yellow video borders mark ground truth and blue borders predictions; occluded details can be missing. Offline action plots compare x/y/z/yaw/pitch/roll/grasp. Figure 7’s caption omits z from its list although plots include it. These are not physical execution trials.
Go to primary source ↓8.5 Primary sources
VidMan: Exploiting Implicit Dynamics from Video Diffusion Model for Effective Robot Manipulation ↗
PDF · 10,222 extracted words
Source fingerprint
5bcbd6c282c62dd6823e1e674a2942319651ba9aa37f9d3922a1aaa91a9b0373