Vidar: Embodied Video Diffusion Model for Generalist Manipulation
1. Paper overview
In one sentence: Vidar transfers a video prior through a sparse inverse-dynamics adapter, gaining low-demonstration manipulation performance while depending on visible robot state and costly video generation. e01e03e04e05e08e11e12e14e17
| At a glance | What to know |
|---|---|
| Research problem | |
| Core mechanism | |
| A key reported result | Real-world manipulation after 20 minutes of target demonstrations: 68.2% seen; 66.7% unseen tasks; 55.6% unseen backgrounds. Scenario success rate. Vidu 2.0 backbone; six seen-task/background tasks, five unseen tasks, and six unseen-background tasks. UniPi: 36.4%, 6.7%, 22.2%; VPP: 4.5%, 13.3%, 0.0%. These measure executed tasks. Baselines were adapted to Vidu, but UniPi lacks Vidar’s embodied pre-training and VPP uses closed-loop control, preventing an isolated architecture comparison. e06e09e11 |
| Reading caution |
Core contributions
Figure 1. Multi-view video generation and masked action decoding form separate stages of the control pipeline. Original paper, p. 3 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Begin with the bottom row: image plus instruction enters the embodied video model, candidate selection follows, and MIDM produces actions. The upper-left panels describe embodied pre-training and target fine-tuning; the Internet-pretrained starting checkpoint is an earlier stage described in the text. At upper right, language splits into Robot, Camera, and Task fields alongside the combined camera imagery. In the MIDM inset, the input image feeds mask prediction and the multiplication node; masked pixels then feed action regression. This direction agrees with the rounded-mask multiplication defined in Section 2.3. The action signal supervises the adapter during training, while the video observation space itself excludes actions. e02e03e04e05e06e07e20
What it supports. The design moves transferable prediction into the video model and leaves robot-specific decoding to MIDM. Its approximately 750K embodied episodes and 232 target episodes describe different training stages. Small target-data demand therefore does not imply that the overall system learns from only 20 minutes of experience.
Where the evidence stops. Section 2.1 additionally mentions proprioceptive traces and contact-consistent objectives, but the explicit conditioning and loss definitions do not explain those mechanisms. The diagram establishes information flow, not guaranteed physical feasibility.
2. Motivation
2.1 The problem and the proposed response
New robot platforms need control knowledge without large platform-specific demonstration corpora. Vidar uses video as the transferable intermediate representation, while recognizing that visually plausible futures alone cannot guarantee executable contact and motion. e02e03
2.2 What this reading follows
Imagine teaching a new pair of robot arms by showing what successful motion should look like, then learning how those images map to controls. Vidar divides adaptation along that boundary. A video generator receives multi-view imagery and language describing the robot, cameras, and task; a masked inverse dynamics model extracts controls from its predictions. The experiments test both parts: embodied pre-training improves video metrics, while masking and candidate selection improve executed-task success. This edition reads the supplied December 2025 v4. Its central question is when a convincing visual future contains enough information to become a reliable physical action. e01e03e04e05e08e11e12e14e17
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Not assigned |
| Architecture | Not assigned |
| Prediction paradigm | Not assigned |
| Quadrant | Not assigned |
This table preserves the labels recorded at reading time. The current major category is WAMs. View the current classification.
3.1 Evidence-based assessment
Insufficient evidence to decide
The supplied catalog is unassigned, so there is no concrete classification to confirm. Architecture evidence supports separate video generation and inverse-dynamics action decoding, with video generation used during inference. It does not support joint future/action prediction by one model. This assessment preserves the catalog snapshot. e03e04e05e08
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Put the transferable prior in a common visual space
Vidar’s first move is to make heterogeneous demonstrations look like compatible prediction problems. Camera images are resized and aggregated, while language states which robot, camera arrangement, and task the video depicts. Equation (3) keeps robot actions outside this common observation space. The generator can therefore learn from embodied videos without translating their controls into a universal action vocabulary. Training starts from an Internet-video checkpoint, continues on embodied data, and then fine-tunes all video-model parameters on target demonstrations. Figure 1 labels the latter two stages. At deployment, the generated video becomes the interface to a separately trained target adapter. Appendix F supplies the crucial condition for that interface: enough of the robot must be visible to recover the required controls. e02e03e04e06e07e17
Figure 3. Action supervision learns selective image masks in two unseen reflective-background examples. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the four panels as two input/output pairs. In each pair, the first panel contains the full combined observation and the second shows what remains after masking. Retained regions include portions of the arms, while much of the tabletop and reflective background disappears. These are task-trained selections, not ground-truth robot segmentations. Section 2.3 explains why: a prediction error rewards pixels useful for recovering demonstrated actions, while an L1 penalty discourages retaining everything. The rounded mask gates the regressor’s input. The nearby quantitative diagnostic is Table 4 on page 8, which tests action prediction against the paper’s joint and gripper error tolerances. e05e13e17
What it supports. The examples are consistent with the intended background-suppression mechanism. Table 4 adds quantitative support: both models reach 99.9% training accuracy, but MIDM obtains 49.0% test accuracy versus ResNet’s 24.3%. That gap suggests better generalization in this evaluation rather than merely a better fit to training frames.
Where the evidence stops. These selected images do not establish that every retained pixel is causally necessary or that masks are temporally stable. The offline threshold metric differs from executed success, and its physical units and detailed split construction are not specified.
5.2 Learn a mask from the cost of getting the action wrong
The mask is learned through action prediction rather than drawn by a segmentation annotator. For an input image, the U-Net produces soft mask values; rounding creates the gate applied before the ResNet regressor. Huber loss rewards accurate demonstrated actions, and the L1 term penalizes the soft mask’s spatial extent. Straight-through estimation lets training propagate through the rounding operation. This creates competing pressures: retain enough information to infer the controls, but discard pixels that do not help. Figure 3 illustrates the resulting selections in reflective scenes. Figure 5 and Table 12 show why the penalty needs tuning: excessive sparsity removes useful evidence, while a weak penalty leaves much of the background. Table 4 evaluates this mechanism using action tolerances, separately from executed task success. e05e07e13e15
5.3 Separate the quality of a plan from the outcome of execution
Candidate selection addresses a failure upstream of the robot controller. The real-world setup generates three independently seeded videos, samples frames from them, and asks GPT-4o to rank plausibility and instruction agreement. MIDM then decodes the selected video; the robot does not request a new video after the initial generation. My interpretation is that success therefore depends on both a suitable imagined trajectory and accurate transfer of that trajectory into the physical scene. Better VBench scores establish only one part of this chain. The component ablation supplies stronger executed-task evidence, while Figure 8 shows prediction and mechanical alignment failures that can still break it. Appendix C’s simulation protocol allows three inferences, so evidence from that benchmark should retain its separate feedback and horizon conditions. e08e10e12e14e18
5.4 Training and inference
During training
Start from Internet-pretrained checkpoints: Wan2.2 for simulation and Vidu 2.0 for the main real-world study. Embodied training uses 746,533 episodes from Agibot-World, RoboMind, and RDT; simulation adds EgoDex. Real-world target adaptation uses 232 episodes covering 81 tasks, described as 20 minutes. e06e07
Video training uses batch size 128, 10,000 embodied pre-training steps, then full-parameter fine-tuning for 12,000 Wan2.2 or 13,000 Vidu steps. Videos are sampled at 8 fps and augmented with random-start, variable-length clips. MIDM trains only on target demonstrations, with AdamW and sparsity weight 0.003. e04e07
During inference
Main real-world execution is open-loop: generate once, select among three parallel candidates using GPT-4o, and decode locally with MIDM. The evaluator sees 5–7 sampled frames per video. One 60-frame, 7.5-second generation takes about 25 seconds on eight NVIDIA Ampere-series 80GB GPUs. e08
Simulation disables ranking (K=1). Appendix C allows three model inferences of 60 steps each, capped at 180 steps, unlike the single-generation real-world protocol. e08e10
5.5 Implementation flow
- Standardize observations
Resize and aggregate camera views into a consistent tensor; concatenate robot, camera, and task descriptions. Equation (3) explicitly excludes actions from the video model’s observation space. e04e06
- Generate and select a future
A rectified-flow generator maps Gaussian noise toward a conditioned video. Independently seeded candidates may be ranked for physical plausibility and instruction agreement. e04e08
- Decode controls
A U-Net predicts a soft spatial mask. Round it, multiply it into the image, and regress actions with a ResNet. Action loss and sparsity train both components through straight-through estimation. e05e07
6. Experiments & results
Vidar adapts an embodied video generator to a target bimanual robot, then converts predicted imagery into controls using a separately trained masked inverse dynamics model. Low-demonstration real-world results and component ablations support this design; action observability, open-loop execution, and substantial prior training constrain its generality.
6.1 Read the original evidence
Table 1. RoboTwin success depends strongly on test conditions; Pi0* is a separate single-task leaderboard reference. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Compare Vidar with Pi0.5 vertically within a column before comparing columns. Low data means 20 demonstrations per task with adjusted views; standard data means 50 with the original views. Both train on clean scenes, so the randomized columns test a shift beyond that training environment. The evaluation covers 50 tasks under one multi-task policy, with the paper specifying 100 evaluation episodes. Appendix C disables candidate reranking and allows three 60-step inferences. The asterisk has an essential qualification in the original caption: Pi0* comes from independently trained single-task leaderboard policies and is not a matched comparator for these multi-task results. e06e08e10
What it supports. Vidar exceeds Pi0.5 in all four matched columns. Under standard data, clean success is 65.8% versus 44.8%, while randomized success is 17.5% versus 14.2%. The same table that supports an advantage also shows that absolute robustness under randomization remains limited.
Where the evidence stops. Low versus standard changes camera visibility as well as demonstration count, so it cannot isolate the effect of more data. These simulation results also use a different inference protocol from the main real-world experiment.
Table 2. After the 20-minute target adaptation set, Vidar leads the reported real-world scenario comparisons. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read across a method’s row to distinguish three evaluation groups, rather than treating the columns as repeated measurements of one task distribution. The groups contain six seen tasks, five unseen tasks, and six tasks in unseen backgrounds. All use the main Vidu-based setup after adaptation on 232 demonstrations covering 81 tasks. The seen background is a cluttered office; unseen backgrounds include a green-screen studio and a workspace with reflective cupboards. These are robot execution outcomes, unlike the VBench video scores elsewhere on the page. Appendix Table 11 supplies individual task and arm labels, which matter when interpreting the aggregate scenario rates. e06e08e09e11
What it supports. Vidar reports 68.2%, 66.7%, and 55.6% success across the three groups, exceeding both video-based comparators. This supports useful transfer within the tested task and background shifts, including instruction-dependent selection tasks. It does not by itself establish unrestricted transfer to arbitrary robot morphologies.
Where the evidence stops. The comparisons share a Vidu backbone but differ in embodied pre-training and control design: VPP is closed-loop, while Vidar is open-loop. Explicit trial counts and confidence intervals are missing, limiting precision and significance claims.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| RoboTwin 2.0 multi-task manipulation 50 tasks; clean training; 20 demonstrations per task with adjusted views (low), or 50 with original views (standard). The paper specifies 100 evaluation episodes. | Low clean/randomized: 60.0% / 15.7%; standard clean/randomized: 65.8% / 17.5%. Average task success rate | Pi0.5: 25.0% / 9.2% and 44.8% / 14.2%, respectively. Vidar leads the matched multi-task comparator, but randomization sharply reduces absolute success. Data quantity and viewpoint change together across regimes. Pi0* follows a separate single-task leaderboard protocol. e06e10 |
| Real-world manipulation after 20 minutes of target demonstrations Vidu 2.0 backbone; six seen-task/background tasks, five unseen tasks, and six unseen-background tasks. | 68.2% seen; 66.7% unseen tasks; 55.6% unseen backgrounds. Scenario success rate | UniPi: 36.4%, 6.7%, 22.2%; VPP: 4.5%, 13.3%, 0.0%. These measure executed tasks. Baselines were adapted to Vidu, but UniPi lacks Vidar’s embodied pre-training and VPP uses closed-loop control, preventing an isolated architecture comparison. e06e09e11 |
| Video quality after embodied pre-training Original versus embodied-pretrained Vidu 2.0 on the unseen target domain. | 0.855 / 0.909 / 0.667. VBench subject consistency / background consistency / imaging quality | Original Vidu: 0.565 / 0.800 / 0.345. Quality improves, but robotic data and the unified format change together. These metrics measure neither contact accuracy nor executed success. e12 |
| MIDM generalization and component ablations Offline prediction plus the real-world scenarios above. Prediction tolerance is maximum error below 0.06 for joints and 0.6 for grippers, for both arms. | MIDM: 49.0% test accuracy, 0.0308 L1 error; both models attain 99.9% training accuracy. Prediction accuracy; testing L1 error; executed success | ResNet: 24.3%, 0.0430. Removing TTS gives 45.5% / 33.3% / 44.4% task success; replacing MIDM gives 59.1% / 26.7% / 22.2%. Both components help this setup. Offline threshold accuracy differs from robot success; the source does not state physical units for its thresholds. e13e14 |
| Additional real-world video backbones Wan2.2 versus Pi0.5 uses 2,307 adaptation episodes and seven seen plus seven unseen tasks. HunyuanVideo is separately tested on six tasks. | Wan2.2: 69.3% seen, 67.1% unseen. HunyuanVideo: reported 58.3% overall. Reported average task success | Pi0.5: 34.3% seen, 12.9% unseen; no matched HunyuanVideo baseline is tabulated. The Wan experiment is separate from the 20-minute study. HunyuanVideo’s six rates have a reader-calculated unweighted mean of 61.7%, not 58.3%; absent trial counts or weights, its reported average remains unresolved. e16 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 5. Removing either candidate ranking or masked decoding reduces the reported scenario success rates. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use the full Vidar row as the reference and compare each removal within the same scenario column. “w/o TTS” disables test-time candidate selection; “w/o MIDM” replaces the masked model with the ResNet baseline rather than removing action decoding. The columns match the real-world scenarios in Table 2, and Appendix Table 11 gives the corresponding task-level values. Read the unseen-task column carefully: it falls from 66.7% to 33.3% without TTS and to 26.7% without MIDM. This connects a decision before action decoding with a separate decision inside the decoder, showing that both parts contribute to the tested system. e08e11e13e14
What it supports. On unseen backgrounds, replacing MIDM reduces success from 55.6% to 22.2%; removing TTS reduces it to 44.4%. These are decreases of 33.4 and 11.2 percentage points, calculated from the displayed rounded values. Masked decoding is particularly consequential for this background-shift evaluation.
Where the evidence stops. There is no row removing both components, no reported uncertainty, and no compute-matched alternative reranker. The table supports the usefulness of each component here, but cannot quantify their interaction or establish a universal effect size.
Figure 5. The mask penalty trades retaining useful arm information against retaining distracting scene detail. Original paper, p. 16 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow the panels left to right as the sparsity weight decreases: 0.1, 0.01, 0.003, 0.001, then 0.0001. The first mask leaves very little information, whereas the final one retains much more of the scene. The middle settings preserve selected arm regions. Connect this visual change to the loss in Section 2.3: increasing the weight penalizes mask area more strongly, while the action loss still favors pixels needed for regression. Table 12 on page 19 supplies the companion measurements. Training accuracy remains high across this sweep, making testing accuracy and testing L1 error more informative than training fit alone. e05e13e15
What it supports. The displayed tradeoff has a quantitative counterpart: test accuracy peaks at 49.0% for a weight of 0.003, versus 7.1% at 0.1 and 24.4% at 0.0001. The evidence supports an intermediate setting for this setup; simply making masks smaller or larger is not sufficient.
Where the evidence stops. The paper does not describe a separate validation procedure for selecting the weight. These masks and offline test scores cannot establish a broadly optimal value across robot embodiments, camera visibility patterns, or execution tasks.
7. Analysis & limitations
7.1 What the evidence leaves open
Appendix F requires imagery to contain enough information to recover controls. The target camera is repositioned to show both arms; hidden joints can violate this assumption. Figure 8 attributes failures to predicted-motion misalignment and gripper mechanical misalignment. e17e18
Section 2.1 mentions proprioceptive traces, embodiment tokens, and contact-consistent objectives. Detailed observation and loss definitions leave these additional mechanisms unspecified. It also describes short video windows, whereas MIDM’s equation uses one frame; temporal input assembly and action cadence remain unclear. e20
Real-world tables omit explicit trial counts and uncertainty. MIDM test-set construction, action units, and a separate validation procedure for choosing λ are unspecified. At λ=0.1, test accuracy is only 7.1%, limiting the claimed broad sparsity robustness. e11e13e15e16
7.2 Questions for discussion
- Does sparse masking still help when joint visibility, data quantity, and background shift are controlled independently?
- Does the evaluator select videos with better executed success, beyond better visual plausibility?
8. Reproducibility audit
8.1 Requirements and known gaps
Recover filtering and camera layouts: Appendix A segments Agibot episodes, filters clips shorter than four seconds or with fewer than three views, and samples datasets proportionally to size. Table 6 also lists one-view EgoDex, leaving the filtering rule’s scope across sources unclear. e06
Reported training resources: Vidu uses 64 Ampere-series 80GB GPUs for about 64 hours; MIDM uses eight Hopper-series 80GB GPUs for 60,000 iterations, about five hours. MIDM has 92M parameters, five U-Net down/up layers, ResNet-50, learning rate 0.0005, and 6,000 warm-up steps. e07e19
Exact Vidu checkpoint access, temporal/action preprocessing, evaluator version, and missing evaluation denominators need clarification. Section 7 says code was submitted as supplemental material; Appendix D says public release is planned. Neither establishes independently verified availability. e19e20
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Separate mask robustness from camera observability
Reader-proposed check, not performed: train ResNet and MIDM on the same target episodes, then evaluate a controlled two-by-two test crossing background change with full versus partial arm visibility. Keep action labels, camera calibration, training quantity, and preprocessing fixed within each comparison. Report both offline errors and executed success, documenting physical units before applying the paper’s tolerances. Select the sparsity weight on a separate validation set. If MIDM improves performance only when joints are fully visible, or loses its background advantage after visibility is controlled, the claimed masking benefit needs a narrower interpretation. This tests the connection between Figure 3 and Appendix F rather than conflating it with extra demonstrations. e05e06e13e15e17
Check 2: Test whether video ranking predicts executable success
Reader-proposed check, not performed: freeze one generator and one MIDM, save three candidate videos per initial scene and instruction, and compare GPT-4o selection with uniform random selection from those same candidates. Pin the evaluator version, prompt, frame sampling, and seeds. Execute each candidate after resetting the scene, with repeated trials and uncertainty estimates; use the resulting success rates to assess whether the ranker favors executable plans. Record generation, ranking, and total latency separately. If ranking improves visual judgments but not executed success under matched candidates, the proposed actionability benefit is unsupported even if videos appear better. An execution-based oracle over the same candidates would also expose the remaining selection headroom. e04e08e11e14e18
8.3 Reading coverage
Visual audit: Visually inspected the title/byline/version, method equations and Figure 1, all original Figures 2–9, and all Tables 1–16. Supporting pages cover dataset filtering, conditioning, training resources, evaluator prompt, simulation and real-world protocols, per-task results, the sparsity sweep, additional backbones, and hardware/visibility assumptions. Cross-checked Figure 1’s mask multiplication and action-regression arrows against Section 2.3; no directional conflict was found. Its Stage 1/2 labels concern embodied pre-training and fine-tuning, following the Internet checkpoint stage in the text. Six original crops were inspected at their final dimensions; the narrow RoboTwin table was rendered at 300 DPI. All remaining supporting pages were viewed at 200 DPI. Reference-only pages 11–13 were read as text. No supplemental continuous videos or code were inspected.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 14, 15, 16, 17, 18, 19, 20, 21, 22. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract
- 1 Introduction
- 2 Method: 2.1–2.3
- 3 Experiments: 3.1.1–3.3
- 4 Related Work
- 5 Conclusion
- 6 Ethics Statement
- 7 Reproducibility Statement
- References
- A Dataset Details
- B Training and Inference Details
- C Experimental Results
- D Additional Real-World Experiments
- E Additional Visualizations
- F Hardware Details
- G Large Language Model Usage
Outside the original text pass
- Identity: the inspected title and eight authors match the catalog. PDF p. 1 identifies arXiv:2507.12898v4 [cs.LG], 20 December 2025, and “Preprint. Under review.” This report covers v4, later than the catalog submission date of 17 July 2025; earlier revisions were not supplied or compared.
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- The extraction limitation was addressed by inspecting every original PDF figure and table. All six text chunks, including references and Appendices A–G, were read. Visual inspection covered PDF pp. 1–10 and 14–22; reference-only pp. 11–13 were read as text.
- No separate supplements, code, datasets, external checkpoints, or continuous demonstration videos were inspected. No experiments were reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title/byline, affiliations, arXiv margin and footer
The title is Vidar: Embodied Video Diffusion Model for Generalist Manipulation. Authors are Yao Feng, Hengkai Tan, Xinyi Mao, Chendong Xiang, Guodong Liu, Shuhe Huang, Hang Su, and Jun Zhu. This is arXiv:2507.12898v4, 20 December 2025, a preprint under review. Affiliations name Tsinghua University units and Shengshu Tech.
Go to primary source ↓e02PDF pp. 1–2, Abstract, Section 1 and Section 2.1 policy definition
The problem is target-embodiment adaptation with limited demonstrations. The introduction distinguishes Internet pre-training, embodied pre-training, and target fine-tuning. Section 2.1 defines language, observation, and action spaces.
Go to primary source ↓e03PDF p. 3, Figure 1/caption and Section 2.1 factorization
The diagram routes image/instruction through the embodied video model, test-time scaling, MIDM, and action. The policy factors as I composed with G. Figure stages cover embodied pre-training and fine-tuning; the MIDM inset gates image pixels before action regression.
Go to primary source ↓e04PDF p. 4, Section 2.2, Equations (1)–(3), fine-tuning and test-time scaling paragraphs
Rectified flow learns noise-to-video velocity. The unified space aggregates resized views and concatenates robot/camera/task language; it excludes actions. Fine-tuning updates all parameters using randomly clipped videos. A pretrained evaluator ranks candidate videos.
Go to primary source ↓e05PDF p. 5, Section 2.3, mask/action definitions and MIDM loss
U predicts a mask in [0,1] over image pixels; rounded masks gate input to R. Training combines Huber action loss and an L1 mask penalty with straight-through estimators, without segmentation labels.
Go to primary source ↓e06PDF p. 5, Section 3.1.1; PDF p. 14, Table 6 and Appendix A
Real-world embodied pre-training uses 746,533 episodes; simulation adds EgoDex. Real-world adaptation uses 232 episodes, 81 tasks, about 20 minutes. Simulation uses 20 adjusted-view or 50 original-view demonstrations per task. Appendix A gives clipping/filtering and proportional sampling. Table 6 includes one-view EgoDex and robot/camera descriptions.
Go to primary source ↓e07PDF p. 6, Section 3.1.2; PDF pp. 14–15, Appendix B and Table 7
Video batch size is 128; 10,000 embodied pre-training steps precede 12,000 Wan2.2 or 13,000 Vidu fine-tuning steps. Videos are 8 fps; MIDM uses U-Net, ResNet and λ=0.003. Table 7 specifies 92M parameters, five down/up layers, ResNet-50, Huber loss, AdamW, learning rate 0.0005 and 6,000-step warm-up.
Go to primary source ↓e08PDF p. 6, Section 3.1.2 inference paragraph; PDF p. 15, Figure 4 and Appendix B; PDF p. 16, Appendix C opening
Real-world control generates once, with K=3 parallel seeds ranked by GPT-4o. A 60-frame video takes about 25 seconds on eight Ampere-series 80GB GPUs. Ranking uses 5–7 frames and a plausibility/instruction/multi-view-consistency prompt. The source reports about $0.003 per pairwise comparison and 25% of latency. Video generation is in the cloud, MIDM local. Simulation allows three 60-step inferences with K=1.
Go to primary source ↓e09PDF p. 6, Section 3.1.3, Pi0.5/UniPi/VPP descriptions
UniPi directly fine-tunes Vidu and trains a ResNet inverse model. VPP shares Vidar’s video checkpoint but decodes single-denoising-pass features into short action sequences and uses closed-loop control. Pi0.5 uses its official checkpoint and multi-task fine-tuning.
Go to primary source ↓e10PDF p. 7, Table 1/caption; PDF p. 16, Appendix C; PDF pp. 17–18, Tables 9–10, Average rows
Vidar low/standard clean success is 60.0%/65.8%; randomized success is 15.7%/17.5%. Pi0.5 scores 25.0%/44.8% and 9.2%/14.2%. The paper specifies 50 tasks and 100 evaluation episodes. Pi0* is from a single-task leaderboard and is not directly comparable. Tables 9–10 give per-task values and repeat these averages.
Go to primary source ↓e11PDF pp. 6–7, Section 3.2 H1/H2 and Table 2; PDF p. 19, Table 11 scenario blocks and Average rows
Real-world evaluation uses six seen tasks, five unseen tasks, and six unseen-background tasks. Vidar scores 68.2%/66.7%/55.6%; UniPi 36.4%/6.7%/22.2%; VPP 4.5%/13.3%/0.0%. Table 11 identifies tasks and arms without explicit trial counts or confidence intervals.
Go to primary source ↓e12PDF p. 7, Section 3.2 H3 and Table 3, both rows and three metric columns
Original Vidu scores 0.565 subject consistency, 0.800 background consistency, 0.345 imaging quality. Embodied pre-training scores 0.855, 0.909, 0.667 on the unseen target domain. No separate data-only versus format-only factorial ablation is shown.
Go to primary source ↓e13PDF p. 8, Section 3.2 H4 and Table 4; PDF p. 9, Figure 3/caption
Prediction success requires maximum joint error below 0.06 and gripper error below 0.6 for both arms. ResNet/MIDM train accuracy is 99.9%/99.9%, test accuracy 24.3%/49.0%, and test L1 error 0.0430/0.0308. Figure 3 shows input/masked-image pairs with unseen reflective backgrounds. Physical units and detailed split construction are unspecified.
Go to primary source ↓e14PDF p. 9, Section 3.3 and Table 5, all rows; PDF p. 19, Table 11 ablation columns
Without TTS, success is 45.5%/33.3%/44.4%; without MIDM, 59.1%/26.7%/22.2%; full Vidar, 68.2%/66.7%/55.6%. The MIDM ablation substitutes ResNet. No row removes both components.
Go to primary source ↓e15PDF p. 16, Figures 5–6 and Appendix C; PDF p. 19, Table 12, all rows
Figure 5 shows masks at λ=0.1, 0.01, 0.003, 0.001, 0.0001. Test accuracies are 7.1%, 39.9%, 49.0%, 40.7%, 24.4%; L1 errors are 0.0670, 0.0331, 0.0308, 0.0338, 0.0461. Figure 6 illustrates segmentation failures. The paper calls 0.003 best without specifying separate validation selection.
Go to primary source ↓e16PDF p. 16, Appendix D; PDF p. 19, Table 13; PDF p. 20, Tables 14–15
Wan2.2 uses 2,307 adaptation episodes and seven seen/seven unseen tasks. Its averages are 69.3%/67.1%, versus Pi0.5 34.3%/12.9%. Table 15 lists HunyuanVideo rates of 75%, 100%, 25%, 50%, 100%, 20%, and average 58.3%; aggregation weights and denominators are absent. Table 13 gives separate backbone configurations.
Go to primary source ↓e17PDF p. 20, Appendix F; PDF p. 22, Figure 9 and Table 16
Video must contain all information needed for action prediction; joints outside camera views can violate this assumption. The target central camera is repositioned to capture both arms. Table 16 specifies 14 degrees of freedom and three RGB cameras.
Go to primary source ↓e18PDF p. 8, Figure 2; PDF p. 21, Figure 7; PDF p. 22, Figure 8 failure rows
The paper juxtaposes predicted and executed frames. Figure 8 attributes failed bowl stacking to motion prediction misalignment and failed dish lifting to gripper mechanical misalignment. These are still sequences, not continuous-video inspection.
Go to primary source ↓e19PDF p. 10, Section 7; PDF pp. 14–15, Appendix B; PDF p. 16, Appendix D; PDF p. 19, Tables 12–13
Vidu training: 23,000 iterations on 64 Ampere-series 80GB GPUs, about 64 hours. MIDM: eight Hopper-series 80GB GPUs for 60,000 iterations, about five hours. HunyuanVideo: 64 Hopper-series 80GB GPUs, about 54 hours. Section 7 states supplemental code submission; Appendix D plans public release. Exact Vidu checkpoint access and evaluator snapshot are unspecified.
Go to primary source ↓e20PDF p. 3, Section 2.1 final paragraphs; PDF p. 4, Section 2.2 Equations (1)–(3); PDF p. 5, Section 2.3 equations
The overview mentions proprioceptive traces, embodiment tokens, contact/flow-consistent objectives, and short video windows. Explicit definitions provide RGB aggregation, language concatenation, flow matching, and single-frame mask/action equations, without explaining additional proprioceptive encoding, a separate contact objective, or temporal window assembly.
Go to primary source ↓8.5 Primary sources
Vidar: Embodied Video Diffusion Model for Generalist Manipulation ↗
PDF · 9,819 extracted words
Source fingerprint
354fc9b0ba31ede2c90872ab6d073e81e4f7a4bd8f365bac848b73bcfa48abe9