VAG: Dual-Stream Video-Action Generation for Embodied Data Synthesis
1. Paper overview
In one sentence: VAG uses synchronized video and action denoising to synthesize robot-training pairs, with useful replay and transfer results but no isolated test of its global-pooling mechanism. e02e03e04e09e11e12
| At a glance | What to know |
|---|---|
| Research problem | Author claim Generated robot videos lack the paired motor trajectories needed for policy learning. The authors target this data bottleneck with joint video–action synthesis, arguing that generating a complete video before regressing actions introduces inconsistency and cumulative errors. Their objective is useful training pairs, with direct trajectory replay as an additional application. e02 |
| Core mechanism | |
| A key reported result | Real-robot tableware pick-and-place after synthetic pretraining: Synthetic-pretrained pi0.5: 11/20 (55%). Physical task success rate. Train VAG on 131 samples, generate X_syn, pretrain pi0.5 to convergence, then fine-tune on 20 real samples for 10,000 iterations; 20 deployment trials per policy. pi0.5 trained on the 20 real samples for 10,000 iterations: 7/20 (35%). The gain is 20 percentage points, or four additional successes. This small study supports usefulness under the tested protocol; it lacks equal-compute and additional-real-data pretraining controls. e12 |
| Reading caution | Source description The authors identify one-way coupling as a limitation and propose action-guided video generation, a larger action DiT, more data and broader task evaluation as future work. e14 |
Core contributions
Figure 3. Video predictions guide action generation through a one-way, globally pooled connection. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the upper panel: text and the initial frame condition the video transformer. Follow its output both rightward to the decoder and downward through 3D Pooling into the Action U-Net. The action branch also receives its own noise. In the lower panel, the snowflake marks the frozen encoder, flames mark trainable denoisers, and separate dashed loss paths compare clean predictions with targets. Crucially, the line from video_pred to pooling is labeled detach. Sections 3.2–3.3 explain that this connection carries predicted clean video information while blocking action-loss gradients into the video model. The diagram summarizes an iterative process whose denoising loop is not drawn. e02e03e04e05e06e14
What it supports. The model contains two distinct denoisers, with information moving from video to action during generation. The detach marker agrees with the training text and the authors' stated lack of reverse action guidance. This is architectural evidence for coupled joint generation; it does not establish a single shared video–action backbone.
Where the evidence stops. The introduction says 'single feed-forward pass,' but Section 3.2 describes repeated denoising and Section 4.1 specifies 35 steps. Read the diagram as a pipeline summary, not evidence of single-step or real-time inference. No timing benchmark is reported.
2. Motivation
2.1 The problem and the proposed response
Generated robot videos lack the paired motor trajectories needed for policy learning. The authors target this data bottleneck with joint video–action synthesis, arguing that generating a complete video before regressing actions introduces inconsistency and cumulative errors. Their objective is useful training pairs, with direct trajectory replay as an additional application. e02
2.2 What this reading follows
A plausible robot video is useful for policy training only if it comes with actions that explain the depicted behavior. VAG tackles this pairing problem by generating video and actions together. Its video transformer predicts a clean future latent at each denoising step; an action U-Net receives a compact summary of that latent while denoising its own trajectory. This edition follows the information flow, then separates video quality, recorded-action accuracy, simulated execution and downstream physical-policy performance. The distinction matters: the paper reports different kinds of success, and its strongest physical transfer comparison involves a separately trained policy evaluated over a small number of trials. e02e03e04e09e11e12
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Not assigned |
| Architecture | Not assigned |
| Prediction paradigm | Not assigned |
| Quadrant | Not assigned |
This table preserves the labels recorded at reading time. The current major category is WAMs. View the current classification.
3.1 Evidence-based assessment
Insufficient evidence to decide
The catalog is entirely unassigned, so there is no recorded quadrant to affirm or contradict. Architecture evidence supports joint future-video/action generation by two distinct denoisers, coupled through detached video conditioning. This is not sequential video-to-IDM extraction, but a unified framework does not establish a single shared backbone or a 'One Model' classification. Video prediction is used during action inference, not solely as a training auxiliary. e03e04e05
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Predict both modalities, but follow the actual direction of dependence
During inference, an initial frame and instruction specify the intended rollout, while Gaussian noise initializes both video and action states. The video transformer predicts a clean latent for the current noise level. Adaptive pooling then compresses that prediction into the global condition used by the action U-Net at the corresponding step. Repeating this sequence produces both outputs after the stated 35 denoising steps. The action branch does not wait for a fully decoded final video and then run inverse dynamics. Nevertheless, the streams are not symmetric: video predictions guide actions, and the authors acknowledge that actions do not guide video generation. When the generated actions are later replayed on a robot, that execution is downstream of the learned prediction procedure; the source does not document an intervening feedback or replanning loop. e03e04e06e13e14
5.2 Understand what pooling and detachment preserve—and what remains untested
The training diagram distinguishes two forms of dependence. The action prediction depends on the video prediction as an input, but detachment prevents the action loss from updating the video branch through that path. Video reconstruction therefore has its own supervision, while the U-Net learns to use an evolving visual summary. Section 3.2 averages every spatial and temporal position within each latent channel. As reader analysis, this removes explicit position indexing from the transmitted vector, although its channel values may still encode useful global information. The reported expansion from 16 channels to a 132-dimensional condition is described as repetition without an exact construction rule. These details motivate an ablation, but the supplied experiments do not independently test pooling structure, gradient detachment or matched-noise synchronization. Close trajectory overlays alone cannot settle those mechanisms. e03e04e05e06e17
5.3 Trace the evidence from synthetic pairs to a deployed policy
The evaluation ladder answers progressively different questions. Table 1 asks whether generated videos score well; Table 2 asks whether actions resemble recorded trajectories under a numerical threshold. Table 3 then asks whether replayed trajectories complete simulator tasks. Finally, Figure 11 evaluates a separate pi0.5 policy trained with the synthetic pairs. For that last experiment, VAG first learns from 131 real trajectories, and its generated data pretrains pi0.5 before the same 20-example real-data fine-tuning used by the baseline. The four additional successful physical trials are encouraging, but reader analysis cannot attribute the entire gain to synthesis itself: the comparison also adds pretraining and access to information from the generator's larger real-data allocation. A compute-matched, direct-real-data pretraining control would distinguish those explanations. e07e08e09e11e12
5.4 Training and inference
During training
Qwen2.5-VL extracts instructions from training videos. A VAE tokenizer compresses time, height and width by 4×8×8; Figure 3 marks the encoder frozen and both denoisers trainable. The video and action branches receive separate clean-reconstruction MSE objectives, with the video prediction detached before action conditioning. e03e05
The reported configuration uses Cosmos-Predict2 2B-Video2World, 93 frames at 10 Hz, input resizing to 432×768, 16 latent channels and a 132-dimensional global embedding. General training is 40,000 iterations on eight NVIDIA H20 GPUs, batch size one per GPU; the LIBERO experiment instead reports 20,000 iterations. e06e10
AgiBot G1 uses 1,794 training and 200 test pairs with 16-dimensional actions. LIBERO uses 400 training and 50 test pairs selected across four suites, with seven-dimensional actions. Agilex data allocates 131 samples to VAG and 20 to VLA training, with 14-dimensional actions. Real datasets use head-camera video; LIBERO stitches head and wrist views. e07
During inference
Initialize video and action noise, repeatedly predict a clean video latent and condition the action denoiser on its pooled embedding, then return decoded video and actions after 35 steps. The introduction's 'single feed-forward pass' wording should not be read as single-step generation. e02e03e04e06
Synthetic pairs pretrain a separate pi0.5 policy, which is subsequently fine-tuned on real demonstrations. Alternatively, VAG-generated trajectories are replayed on Agilex. The source describes generation followed by replay, without a documented feedback or replanning loop for VAG itself. e12e13
5.5 Implementation flow
- Generate a conditional video latent
Cosmos-Predict2 supplies the video DiT. The initial image is a latent prefix; T5-XXL text embeddings enter through cross-attention with classifier-free guidance. Video noise is scaled by the paper's weight sigma. Each denoising step predicts a clean latent, and the final latent is decoded to video. e03e04
- Compress video context into action conditioning
Adaptive 3D pooling averages the clean latent over space and time, retaining one value per latent channel. The vector is repeated to form the global embedding and concatenated with the encoded diffusion timestep for a modified Diffusion Policy 1D U-Net. This transfers video information to actions without an inverse-dynamics stage. e04
- Synchronize the two denoisers
Both streams use the same noise intensity during training and synchronized denoising during generation. The U-Net consumes the current video prediction, so the coupling is directional. There is no action-to-video feedback in this version, as the authors explicitly acknowledge. e04e05e14
6. Experiments & results
VAG synthesizes robot videos and action sequences from an initial image and instruction. A video diffusion transformer supplies pooled clean-latent predictions to an action U-Net during synchronized denoising. The evidence supports improved action prediction and simulation replay, plus a small real-robot policy-pretraining benefit. It does not establish guaranteed alignment or general closed-loop control.
The supplied PDF contains no controlled ablation of the pooling bridge, detachment or synchronized denoising. Figure 9 is therefore included as a trajectory diagnostic in the ablation slot, explicitly without causal attribution. Training-loss curves and two-stage baseline comparisons do not isolate these components; the report preserves that evidence gap rather than inventing an ablation. e03e04e10e16e17
6.1 Read the original evidence
Table 1. VAG improves several visual metrics while Wan2.2 retains two column wins. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the arrows before comparing rows: lower values are preferred for FVD, FID and LPIPS, while higher values are preferred for SSIM and PSNR. CP2 abbreviates Cosmos-Predict2, the video foundation model underlying VAG. The experiment generates from the first frame and corresponding prompt of AgiBot test examples; the dataset section reports 200 test pairs. Compare VAG with CP2 to assess the reported change from that baseline, then check Wan2.2 to avoid treating bold entries as a universal ranking. The five columns measure different aspects of generated visual output and contain no robot-execution score. e07e08e16
What it supports. VAG reports FVD 965, LPIPS 0.320 and PSNR 15.1, leading those columns. Wan2.2 has better FID, 129 versus 130, and SSIM, 0.612 versus 0.512. The table therefore supports competitive video quality with specific improvements, rather than dominance on every reported measure.
Where the evidence stops. The PDF does not establish matched post-training, data exposure or compute for every video baseline. These visual scores also cannot verify action alignment or task completion; those require the separate action and replay experiments.
Table 2. Joint generation improves recorded-action accuracy, under a threshold-based success definition. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Treat each dataset as its own pair of columns. ED is Euclidean distance between predicted and ground-truth actions; SR is the proportion judged successful by the paper's error rule. Section 4.2 defines that rule on the next page: every action dimension must have error below 0.2. The first two rows use the same VAG-generated videos but then regress actions with ResNet or AnyPos. The final row instead uses the synchronized action denoiser. This helps compare action-generation pipelines, although the regressors and the U-Net are different architectures. Do not read the LIBERO SR column as the simulator's task-completion result. e07e09e10e16
What it supports. VAG lowers ED from AnyPos's 0.98 to 0.81 on AgiBot and from 0.55 to 0.38 on LIBERO. Its corresponding threshold success rates are 45% and 79%, compared with 29% and 66%. These results support improved agreement with recorded trajectories within each reported dataset.
Where the evidence stops. Action units, normalization and temporal aggregation of the threshold are not specified. The baselines report 40,000 training iterations, whereas VAG's LIBERO experiment reports 20,000. Architecture and training differences prevent this comparison from isolating synchronized conditioning alone.
Table 3. Simulation replay provides a separate execution test of the generated trajectories. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Begin with the heading 'Success Rate (%) — Replay': this is the execution-oriented table, unlike Table 2's error-threshold evaluation. Read across each method's Spatial, Object, Goal and Long results before consulting Avg. All three methods are evaluated by replaying their generated trajectories in simulation. The accompanying text and Figure 7 compare generated camera views with views from replay, linking visual generation to actual simulator behavior. Section 4.1 describes selected LIBERO training/test pairs; the table does not enumerate the replay episode counts or fully specify their relationship to those selected pairs. e07e09e11
What it supports. VAG's reported average replay success is 62%, versus 54% for AnyPos and 25% for ResNet. The advantage over AnyPos varies across suites: Long rises only from 39% to 42%, whereas Spatial rises from 59% to 70%. Performance therefore depends substantially on the task suite.
Where the evidence stops. The paper does not supply replay trial counts, seed variability or detailed termination rules. These simulator results do not establish physical-robot success or performance on a fully specified standard LIBERO evaluation protocol.
Figure 11. Synthetic pretraining adds four successful trials in a small physical-policy study. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Separate the left loss plot from the right deployment bars. Both policies are fine-tuned on the same 20 real demonstrations for 10,000 iterations, but the blue policy first receives synthetic pretraining to convergence. VAG generated those training pairs after learning from a separate 131-sample allocation. The loss axis is logarithmic; small visual differences between curves should not be treated as proportional improvements in physical performance. Section 4.4 gives the deployment denominator missing from the bars: 20 tableware pick-and-place trials per policy, with seven successes for the baseline and eleven after synthetic pretraining. e07e12e13
What it supports. The source reports 35% versus 55% task success, an absolute gain of 20 percentage points. This is evidence that VAG-generated data can help the tested pi0.5 training pipeline. The deployed policy in this figure is pi0.5; direct replay of VAG actions is a separate experiment illustrated in Figure 12.
Where the evidence stops. Only 20 trials per policy are reported, without uncertainty or a matched-compute control. Synthetic-set size and pretraining duration are unspecified. Additional real-data exposure through the generator is also not controlled by direct pretraining on those 131 demonstrations.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| AgiBot video generation AgiBot G1 test split: 200 pairs; generation conditioned on first frame and corresponding prompt. | VAG: 965, 130, 0.320, 0.512, 15.1, respectively. FVD, FID, LPIPS (lower); SSIM, PSNR (higher) | Cosmos-Predict2: 988, 135, 0.352, 0.427, 14.2. Wan2.2 leads FID (129) and SSIM (0.612). VAG leads three of five metrics. Video-quality scores do not measure executed action success; matched baseline post-training details are incomplete. e07e08e16 |
| Action prediction against recorded trajectories AgiBot and LIBERO test pairs; two-stage baselines regress actions from VAG-generated videos. | VAG: AgiBot ED 0.81 / SR 45%; LIBERO ED 0.38 / SR 79%. Euclidean distance (ED, lower) and threshold-based success rate (SR, higher) | AnyPos: 0.98 / 29% and 0.55 / 66%; ResNet: 1.54 / 8% and 0.87 / 37%. SR requires error below 0.2 in every action dimension. It measures agreement with recorded actions, not environment task completion; normalization and temporal aggregation remain unspecified. e07e09e16 |
| LIBERO trajectory replay Generated trajectories replayed in Spatial, Object, Goal and Long suites; trial counts and repeated-seed uncertainty are not specified. | VAG: Spatial 70, Object 72, Goal 64, Long 42; average 62. Executed-task success rate (%) | AnyPos: 59, 62, 56, 39; average 54. ResNet: 33, 34, 23, 10; average 25. The reported average exceeds AnyPos by 8 percentage points, but Long remains weakest. This replay result is distinct from the 79% action-threshold score. e09e11 |
| Real-robot tableware pick-and-place after synthetic pretraining Train VAG on 131 samples, generate X_syn, pretrain pi0.5 to convergence, then fine-tune on 20 real samples for 10,000 iterations; 20 deployment trials per policy. | Synthetic-pretrained pi0.5: 11/20 (55%). Physical task success rate | pi0.5 trained on the 20 real samples for 10,000 iterations: 7/20 (35%). The gain is 20 percentage points, or four additional successes. This small study supports usefulness under the tested protocol; it lacks equal-compute and additional-real-data pretraining controls. e12 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 9. A selected trajectory reveals local errors hidden by aggregate scores. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow the six small panels as separate action coordinates labeled Arm Joint 1 through Arm Joint 6; compare the blue GT Action trace with the orange Pred Action trace. The lower, wider Gripper Joint panel makes switching behavior easier to inspect. Horizontal axes count frames, and vertical axes are labeled only Value. The caption identifies a LIBERO task that moves the black bowl between the plate and ramekin onto the plate. Inspect both plateau agreement and transitions: the arm traces follow similar shapes, while the gripper prediction changes more gradually than the abrupt ground-truth signal and briefly overshoots its levels. e04e17
What it supports. The selected example supports the claim that VAG can reproduce detailed trajectory trends, but also shows why broad visual agreement is insufficient. The gripper trace contains localized discrepancies around transitions despite close plateau agreement. This is a useful diagnostic observation, not evidence that these errors caused any particular replay failure.
Where the evidence stops. This is one selected example with no component-removal comparison, population error distribution or stated physical action units. It cannot validate guaranteed alignment or attribute improvements specifically to pooling, synchronization or the detached connection.
7. Analysis & limitations
7.1 What the evidence leaves open
The authors identify one-way coupling as a limitation and propose action-guided video generation, a larger action DiT, more data and broader task evaluation as future work. e14
No controlled ablation isolates pooling, detachment or synchronized noise. Baseline comparisons change the action architecture and generation procedure together. Training-loss curves and selected trajectory overlays diagnose behavior but cannot identify the cause of improvement. e03e16e10e17
Global averaging discards explicit spatial and temporal positions in the conditioning vector. The source's strong alignment language is not a mathematical guarantee; Figure 9 visibly retains gripper-transition errors despite close arm-trajectory agreement. e02e04e17
The physical replay examples are selected demonstrations, not a VAG deployment success-rate study. The separate pi0.5 comparison has only 20 trials per condition, unspecified synthetic-set size and pretraining duration, and no reported uncertainty. e12e13
7.2 Questions for discussion
- Would temporally structured conditioning reduce the gripper-transition errors visible in Figure 9?
- Would the transfer advantage persist against equal-compute pretraining directly on the 131 real demonstrations?
8. Reproducibility audit
8.1 Requirements and known gaps
Reproduction requires the stated model components, embodiment-specific data splits, camera preprocessing and replay interfaces. The PDF omits optimizer, learning rate, precise noise/guidance settings, software versions and exact action normalization; these must be resolved before interpreting ED or the 0.2 threshold. e03e04e06e07e09
The 16-to-132 channel-repetition rule is not fully specified. The relation between velocity MSE and clean-reconstruction MSE needs implementation clarification. A useful proposed check is to vary pooled conditioning while fixing denoisers and noise schedules; a second is to match pretraining compute and real-data exposure in the transfer comparison. e04e05e06e12e15
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Test whether temporally structured conditioning improves replay
Reader-proposed, not performed: on the same selected LIBERO split, train matched action U-Nets with VAG's global pooled condition, a learned constant condition, and a condition that retains temporal bins before projection. Hold the video predictions, action initialization, optimizer, training budget and denoising schedule fixed; document how each condition reaches the required embedding width. Measure held-out ED, gripper-transition error and replay success across repeated seeds. If global conditioning does not beat the constant control, its claimed contribution is weakened. If temporal bins reduce transition error without improving replay, that would limit the practical importance of the diagnostic discrepancy. e04e06e07e09e11e17
Check 2: Separate synthesis benefit from extra pretraining and real-data exposure
Reader-proposed, not performed: compare pi0.5 pretraining on VAG pairs with pretraining directly on the same 131 real demonstrations available to the generator, plus a control spending the same additional updates on the 20-example fine-tuning set. Match policy initialization, update budget and final 10,000-step fine-tuning; declare synthetic sample counts. Evaluate identical held-out color/position conditions over repeated seeds and more physical trials, reporting binomial uncertainty and overlap checks. An advantage over both matched controls would support a specific benefit from synthesis; parity would suggest that extra training or access to additional demonstrations explains the original gain. e07e12
8.3 Reading coverage
Visual audit: Page 1 verifies title, all authors, affiliations and v1/date; page 2 supports motivation and the single-feed-forward wording; page 3 supplies flow-matching equations; page 4 supplies the architecture, detach/frozen/trainable markings and conditioning; page 5 supplies losses, preprocessing, dimensions, hardware and splits; page 6 supplies video/action tables and baseline training; page 7 supplies threshold definition, LIBERO training and trajectory diagnostics; page 8 supplies simulation replay, synthetic-pretraining protocol and physical trial counts; page 9 supplies selected physical replays and author limitations. All six final original crops were inspected, with full labels and legends retained. Figures 1–12 and Tables 1–3 were viewed on their original pages. Reference pages 10–13 were text-read but not visually inspected; no scientific claim relies on inspecting a cited external work. No appendix is present and separate supplements remain unverified.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9. Appendix coverage: not present.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Title, authors, affiliations, version and abstract (p. 1–2)
- 1. Introduction (p. 2)
- 2. Related Work (p. 2–3)
- 3. Method; 3.1 Flow Matching; 3.2 Dual-Stream Generation; 3.3 Training (p. 3–5)
- 4.1 Experimental Setup (p. 5–6)
- 4.2 Real-World Evaluation (p. 6–7)
- 4.3 Simulation Evaluation (p. 7–8)
- 4.4 VLA Generalization and 4.5 World-Action Policy (p. 8–9)
- 5. Conclusion; Limitations and Future Work (p. 8–9)
- References [1]–[86] (p. 10–13)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- All five supplied text chunks were read individually. PDF pages 1–9 and all six final crops were visually inspected; this resolves the figure-image extraction omission for the reviewed PDF. Reference pages 10–13 were read as text.
- Identity matches the catalog title and all 13 authors. The title page identifies arXiv:2604.09330v1 [cs.RO], 10 April 2026; no different revision or edition was supplied or compared.
- No appendix is present in the supplied 13-page PDF. Separate supplements remain unverified.
- Code and external resources were not inspected, and no experiments were reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title, complete byline, affiliation block and arXiv margin stamp
Title and all 13 authors match the catalog; the artifact is labeled arXiv:2604.09330v1 [cs.RO], 10 April 2026. Five affiliations are listed.
Go to primary source ↓e02PDF p. 2, Section 1, motivation, proposed method and contribution bullets
The authors motivate aligned synthetic training pairs, claim rigorous alignment and describe single-feed-forward generation; these are author claims to read alongside the iterative method.
Go to primary source ↓e03PDF p. 4, Figure 3 and Section 3.2, Video Prediction
The diagram shows image/text-conditioned video DiT, a decoder, pooling into a separate action U-Net, frozen encoder, trainable denoisers and a detach marker in training. Text specifies Cosmos-Predict2, noise scaling, T5-XXL and classifier-free guidance.
Go to primary source ↓e04PDF p. 5, Section 3.2, continuation of Video Prediction and Action Prediction
Each denoising step sends predicted clean video latents through global adaptive 3D average pooling and channel repetition to a time-conditioned 1D action U-Net; both finish after N steps.
Go to primary source ↓e05PDF p. 5, Section 3.3, Video Branch and Action Branch, Equations (4)–(5)
Training uses Qwen2.5-VL-derived instructions, T5-XXL, pretrained video weights, VAE compression of 4×8×8, clean-reconstruction losses and detached predicted-video conditioning with matched noise intensity.
Go to primary source ↓e06PDF p. 5, Section 4.1, Implementation Details
Configuration: Cosmos-Predict2 2B-Video2World, 10 Hz, 93 frames, resize 432×768, C=3, C'=16, C''=132, 35 denoising steps; 40,000 iterations on eight H20 GPUs with batch size one per GPU.
Go to primary source ↓e07PDF p. 5, Section 4.1, Datasets
Used splits are AgiBot G1 1,794/200 and LIBERO 400/50 training/test pairs; actions have 16 and 7 dimensions. Agilex allocates 131 VAG-training and 20 VLA-training samples with 14-dimensional actions; camera views are specified.
Go to primary source ↓e08PDF p. 6, Table 1, all rows and metric columns; Section 4.2, video test inputs
VAG reports FVD 965, FID 130, LPIPS 0.320, SSIM 0.512, PSNR 15.1. CP2 reports 988/135/0.352/0.427/14.2; Wan2.2 has best FID 129 and SSIM 0.612.
Go to primary source ↓e09PDF p. 6, Table 2, all rows; p. 7, Section 4.2, action-evaluation paragraph
VAG ED/SR is 0.81/45% for AgiBot and 0.38/79% for LIBERO; AnyPos is 0.98/29% and 0.55/66%; ResNet is 1.54/8% and 0.87/37%. Success is defined by error below 0.2 in every dimension.
Go to primary source ↓e10PDF p. 6, Figure 4; p. 7, Section 4.3 and Figure 8
AgiBot loss curves cover 40,000 iterations; LIBERO training and curves cover 20,000 iterations. These are optimization diagnostics, not component-removal comparisons.
Go to primary source ↓e11PDF p. 8, Table 3, all rows; Section 4.3 continuation
LIBERO replay success for VAG is 70/72/64/42, average 62; AnyPos 59/62/56/39, average 54; ResNet 33/34/23/10, average 25. This table reports replay rather than action-error thresholding.
Go to primary source ↓e12PDF p. 8, Section 4.4, Figures 10–11
VAG learns from X_a of 131 samples; synthetic pairs pretrain pi0.5 to convergence before 10,000-step fine-tuning on X_b of 20 samples. Real tableware trials yield 11/20 versus 7/20; figures illustrate relocation/color cases and loss/success comparisons.
Go to primary source ↓e13PDF p. 8, Section 4.5; p. 9, Figure 12 and caption
Head-camera image and instruction generate trajectories replayed on Agilex. Three selected examples cover left-arm, right-arm and bimanual manipulation; no aggregate VAG physical-replay success rate is supplied.
Go to primary source ↓e14PDF p. 9, Limitations and Future Work
Video generation currently receives no action-branch influence. Proposed extensions include reverse conditioning, replacing the action U-Net by a DiT, larger data and wider tasks.
Go to primary source ↓e15PDF p. 3, Section 3.1, Equations (1)–(2); p. 4, Equation (3) and symbol definitions
Flow matching interpolates data and Gaussian noise using a logit-normal timestep; the target velocity is noise minus data, trained by conditional velocity squared error.
Go to primary source ↓e16PDF p. 6, Section 4.1, Baselines
SVD and Wan2.2 video baselines have 1.5B and 5B parameters. Action baselines use ResNet50 plus MLPs or AnyPos to regress actions from VAG videos and are trained for 40,000 iterations.
Go to primary source ↓e17PDF p. 7, Figure 9, axes, GT Action/Pred Action legend and caption
The selected LIBERO bowl-to-plate example overlays six arm-action channels and a gripper channel over frames. Arm curves largely overlap; predicted gripper transitions are smoother and overshoot the ground-truth steps. No action units or component ablation are shown.
Go to primary source ↓8.5 Primary sources
VAG: Dual-Stream Video-Action Generation for Embodied Data Synthesis ↗
PDF · 8,171 extracted words
Source fingerprint
99cde0af549a4de56c722a3ba9d0096e976730c0d9b6b8ed793bc65cffa0a51c