CoVAR: Co-generation of Video and Action for Robotic Manipulation via Multi-Modal Diffusion
1. Paper overview
In one sentence: CoVAR couples pretrained video diffusion to a dedicated action diffusion branch, improving reported manipulation success while relying on additional action refinement for low-resolution LIBERO90. e-probleme-architecturee-bridgee-liberoe-reale-ablation
| At a glance | What to know |
|---|---|
| Research problem | Source description Generated robot videos usually lack executable action labels. The authors seek paired visual futures and joint-state actions from one initial observation and instruction, retaining pretrained video knowledge while enabling bidirectional interaction with control representations. Video and actions are not spatially aligned, motivating separate processing streams. e-probleme-architecture |
| Core mechanism | Source description A parallel action DiT extends an OpenSora video DiT, with modality-specific Bridge Attention projections and a UNet action decoder. e-architecturee-bridge |
| A key reported result | UR5 small-object manipulation: Nut 0.64; screw 0.74; dowel 0.70. Success rate (fraction). Self-collected real dataset; nut, screw and dowel tasks; trial counts and uncertainty are unstated. RoboEnvision: 0.04 / 0.10 / 0.12; Unipi: 0.00 / 0.06 / 0.02. Physical execution evidence for these tasks, without a reported statistical significance analysis or broad real-world joint-model comparison. e-real |
| Reading caution | Source description The authors identify monocular video as limiting full 3D scene geometry. The supplied PDF does not test the proposed remedy of incorporating 3D foundation-model perception. e-limit |
Core contributions
- Source description
A parallel action DiT extends an OpenSora video DiT, with modality-specific Bridge Attention projections and a UNet action decoder. e-architecturee-bridge
- Source description
An image- and text-conditioned transformer refines coarse actions on low-resolution LIBERO90; experiments separately report video fidelity, simulation success, physical success, and component removals. e-refinemente-videoe-liberoe-reale-ablation
Figure 2. Separate diffusion streams exchange features before decoding video and action. Original paper, p. 3 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start at panel A's left edge: image/video information passes through a VAE, text through its encoder, and action information through a separate encoder. Follow the upper and lower DiT streams into Bridge Attention and their residual additions. Their output heads differ: video uses a video head, action a UNet. Panel B expands the bridge: each modality forms its own Q, K and V, while matching projections are concatenated before attention and split afterward. This agrees with Equation (3) on page 4. Panel C is a later refinement stage: rough actions and image tokens enter self-attention, followed by text-conditioned cross-attention and decoding. e-architecturee-bridgee-refinemente-training
What it supports. The diagram supports joint video–action generation through two interacting backbones. Information can flow between modalities during denoising, and the action branch has its own representation and decoder. LIBERO90 adds a downstream refiner, so results for that dataset describe more than the paired diffusion backbone alone.
Where the evidence stops. The overview omits noise schedules and the exact injection of initial-state conditioning. It provides no frozen-module or stop-gradient markers. Preserving pretrained knowledge should therefore not be read as evidence that the video DiT is frozen.
2. Motivation
2.1 The problem and the proposed response
Generated robot videos usually lack executable action labels. The authors seek paired visual futures and joint-state actions from one initial observation and instruction, retaining pretrained video knowledge while enabling bidirectional interaction with control representations. Video and actions are not spatially aligned, motivating separate processing streams. e-probleme-architecture
2.2 What this reading follows
A plausible video of a robot grasping an object does not supply the joint commands needed to execute that grasp. CoVAR learns to generate both outputs together. Its design keeps video and action in separate diffusion transformers, then lets them exchange information through Bridge Attention. The distinction matters: an action stream can use pretrained visual knowledge without requiring every token to share the same projection parameters. The experiments support this combined design on simulation and physical manipulation, but also reveal a substantial dependence on action refinement. This reading follows the architecture, distinguishes video quality from executed success, and examines what the ablations can establish. e-probleme-architecturee-bridgee-liberoe-reale-ablation
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Not assigned |
| Architecture | Not assigned |
| Prediction paradigm | Not assigned |
| Quadrant | Not assigned |
This table preserves the labels recorded at reading time. The current major category is WAMs. View the current classification.
3.1 Evidence-based assessment
Insufficient evidence to decide
The recorded classification is entirely unassigned, so there is no substantive catalog judgment to confirm. Architecture evidence supports joint future-video/action prediction through two coupled DiTs, with optional downstream refinement. It does not support describing the architecture as a single shared DiT. A final quadrant assignment requires a catalog definition for coupled multi-module co-generation; the snapshot is preserved. e-architecturee-bridgee-refinement
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Generate paired futures without a single shared transformer
Begin with the conditioning interface: an initial RGB observation, robot joint states and an instruction. CoVAR predicts future video frames and a corresponding action sequence, so action generation is not postponed until after a finished video has been decoded. The video branch inherits an OpenSora backbone, while an MLP embeds low-dimensional actions for a dedicated action DiT. The two streams exchange features through Bridge Attention, and each retains a distinct output head. Training supervises both components of a multimodal flow field. Equation (1) labels data at time zero and noise at time one; the reverse direction needed for generation follows from those endpoints, although the numerical solver is unstated. Equation (2) is reproduced as printed, with L2 norms rather than silently adding squared exponents. e-architecturee-flowe-training
5.2 Locate what Bridge Attention changes
The bridge does more than concatenate raw video and action tokens. It first applies separate query, key and value parameters to each modality. It then concatenates corresponding projections, computes joint attention and separates the outputs again. Figure 2(B)'s connections agree with Equation (3): both streams contribute to the shared attention operation and both receive an output. The authors argue that this balances cross-modal communication with modality-specific structure. Table V supports the design empirically because replacing the bridge with the tested self- or cross-attention alternatives lowers success and worsens video metrics. Reader analysis: this is evidence for these particular architectural replacements, not a direct measurement of interference. Parameter-matched controls and feature-level diagnostics would be needed to establish the proposed causal explanation more narrowly. e-bridgee-architecturee-ablation
5.3 Follow coarse actions all the way to execution
A paired prediction is not yet a guarantee of successful contact. On LIBERO90, the authors add a transformer that combines rough action tokens with the initial image, then uses instruction-conditioned cross-attention before decoding refined actions. Table III makes this extra stage central: the raw backbone trails UVA in every reported task group, while refinement reverses that comparison. The real rollout has a separate practical constraint. It generates a 35-frame pair in about four seconds using 30 flow steps and interpolates the action sequence for a 100 Hz robot controller. Reader analysis: controller frequency does not remove generation latency or create visual feedback within the open-loop sequence. The physical successes are meaningful, but robustness to disturbances and continuous replanning remain unestablished by this protocol. e-refinemente-liberoe-rolloute-real
5.4 Training and inference
During training
Co-train the two modalities from OpenSora-1.2 using 35-frame samples and the multimodal flow objective. The model has 1.4B parameters: 1.1B video and 0.3B new modules. Training is reported as about one day on four GPUs; GPU models are unspecified. Preserving pretrained knowledge does not establish that the video DiT is frozen. e-training
Data comprise approximately 20k CALVIN demonstrations at 200×200, LIBERO90's 90 tasks with 50 demonstrations each at 128×128, and 1K self-collected real demonstrations. Real training uses 180×320 videos. LIBERO90 refinement is fine-tuned with 450 video–action pairs; their selection is unspecified. e-datae-training
During inference
The real rollout obtains the current image and joint state through ROS and generates a 35-frame video–action pair with 30 rectified-flow sampling steps in approximately four seconds. Actions are interpolated to the robot's 100 Hz control frequency and executed open-loop. e-rollout
The video branch participates in inference-time co-generation. The description gives neither a receding-horizon execution schedule nor feedback corrections within an action sequence. The 100 Hz figure concerns robot control, not fresh model predictions. e-architecturee-rollout
5.5 Implementation flow
- Encode separate modalities
The video branch uses the pretrained video model and VAE representation. A lightweight MLP embeds actions without training an action VAE. Parallel video/action DiTs process their respective features; text conditions the action DiT through cross-attention. e-architecture
- Exchange information through Bridge Attention
Video and action features each have their own query, key, and value projections. Concatenated projected tokens undergo joint attention, then split back into their streams. Thus both intra-modal and cross-modal interactions remain available. A UNet supplies the action output head. e-bridgee-architecture
- Refine coarse control when needed
For LIBERO90, embed rough actions and the initial image, concatenate their tokens, apply self-attention and text-conditioned cross-attention, then decode precise actions. The source attributes the need to a resolution mismatch; it does not give a separate refinement loss. e-refinemente-training
6. Experiments & results
CoVAR co-generates instruction-conditioned video and robot actions through separate, interacting diffusion transformers. Bridge Attention connects a pretrained video branch to an action branch; a UNet decodes actions, with an additional refinement model for LIBERO90. Reported simulation and UR5 successes support the complete system, while refinement dependence, incomplete evaluation details, and roughly four-second sequence generation limit broader conclusions.
6.1 Read the original evidence
Table I. CoVAR improves the joint baselines, while comparison with OpenSora is mixed. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read CALVIN and LIBERO90 as separate four-column blocks. Higher PSNR and SSIM indicate better agreement; lower LPIPS and FVD are preferable. The accompanying evaluation uses 100 generated CALVIN videos and 180 LIBERO90 videos from a validation partition. First compare CoVAR against UVA, PAD and UWM: it improves every displayed metric on both datasets. Then compare against OpenSora, the video-only backbone. Its row is essential to the pretraining argument: CoVAR's CALVIN SSIM is slightly lower, its LIBERO90 PSNR is lower, and its FVD is higher on both datasets. The bold cells therefore belong to more than one method. e-video
What it supports. CALVIN FVD is 72.42 for CoVAR versus OpenSora's 61.00; LIBERO90 FVD is 70.64 versus 63.33. CoVAR nevertheless improves LPIPS on both datasets and outperforms the listed joint baselines throughout. The supported conclusion is competitive video generation alongside action prediction, with metric-dependent tradeoffs against the pretrained video baseline.
Where the evidence stops. These are generated-video measurements, not robot success rates. The source does not provide uncertainty, the precise validation partition, or enough baseline training details to establish a fully matched comparison.
Table II. CoVAR leads every reported CALVIN task group under the paper's rollout protocol. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Treat each column as a task group rather than a stage in a long-horizon chain. Drawer and Cabinet combine opening and closing; Light includes switching via handles and buttons, while Pick and Push concern object manipulation. The paper trains with 20k ABCD videos and evaluates 200 randomly generated novel test scenes. Read the strongest baseline separately for each column: UVA leads the comparison methods in four groups, whereas UWM has the higher Cabinet baseline. CoVAR's bottom row is highest across all five. The table reports fractions; a value of 1.000 describes the evaluated Drawer group, not universal success on CALVIN. e-calvin
What it supports. The reported CoVAR rates are 1.000, 0.800, 0.867, 0.909 and 0.929 from Drawer through Push. These support improved grouped manipulation success in this experiment. The Cabinet comparison is 0.800 versus UWM's 0.733; Pick is 0.909 versus UVA's 0.758.
Where the evidence stops. The paper gives neither per-group trial counts nor uncertainty. This grouped ABCD protocol is not evidence of standard CALVIN chain-length performance or a held-out-environment transfer score.
Table IV. Physical small-object manipulation gives evidence beyond generated-video quality. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Each column is a real manipulation task involving a small nut, screw or dowel on a UR5 platform. Compare within a column: the table's two baselines are Unipi and RoboEnvision, both described as two-stage methods. CoVAR's largest displayed rate is 0.74 for screws; the corresponding RoboEnvision rate is 0.10. The dataset description reports 1K demonstrations including these object tasks and bowl stacking, but that is a training-data scale, not the number of test trials. The real rollout paragraph separately describes approximately four seconds to generate a sequence, followed by interpolation of open-loop actions to the robot's control frequency. e-reale-datae-rollout
What it supports. CoVAR achieves reported success fractions of 0.64 for nuts, 0.74 for screws and 0.70 for dowels, above both listed baselines. Unlike Table I, these measurements concern executed manipulation. They provide concrete physical-task support for the architecture under the reported setup.
Where the evidence stops. The PDF supplies no real evaluation trial counts, uncertainty or explicit split definition. This table compares two-stage baselines only; it cannot establish physical superiority over every joint-model baseline used in simulation.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Video generation on CALVIN and LIBERO90 Validation partition; 100 generated CALVIN videos and 180 LIBERO90 videos. Exact split construction is unspecified. | CALVIN: 19.95 / 0.766 / 0.156 / 72.42. LIBERO90: 20.09 / 0.826 / 0.143 / 70.64. PSNR / SSIM / LPIPS / FVD | CoVAR beats UVA, PAD and UWM on all listed metrics. OpenSora has better FVD: 61.00 / 63.33; it also has better CALVIN SSIM and LIBERO90 PSNR. Supports improved fidelity over these joint baselines, with mixed results against the video-only backbone. It does not measure execution success. e-video |
| CALVIN grouped manipulation success Training on 20k ABCD videos; 200 randomly generated novel rollout scenes, grouped by task. | Drawer 1.000; cabinet 0.800; light 0.867; pick 0.909; push 0.929. Success rate (fraction) | Best baseline per group: 0.875 UVA; 0.733 UWM; 0.711 UVA; 0.758 UVA; 0.785 UVA. Highest reported rate in each group. This custom grouped protocol does not establish standard CALVIN long-horizon chain performance. e-calvin |
| LIBERO90 manipulation with action refinement 450 rollouts across pick, open/close, and compositional task groups; refinement is used. | Pick 0.873; open/close 0.860; combination 0.711. Success rate (fraction) | Without refinement: 0.592 / 0.520 / 0.422. UVA: 0.676 / 0.640 / 0.489. Refinement adds 28.1, 34.0 and 28.9 percentage points, calculated from reported fractions. Raw CoVAR trails UVA in every group. e-libero |
| UR5 small-object manipulation Self-collected real dataset; nut, screw and dowel tasks; trial counts and uncertainty are unstated. | Nut 0.64; screw 0.74; dowel 0.70. Success rate (fraction) | RoboEnvision: 0.04 / 0.10 / 0.12; Unipi: 0.00 / 0.06 / 0.02. Physical execution evidence for these tasks, without a reported statistical significance analysis or broad real-world joint-model comparison. e-real |
| Component ablations on the self-collected dataset Replace Bridge Attention with self- or cross-attention; replace UNet with ResNet; or remove video DiT. | Full: 0.68 success, 133.89 FVD. Action success rate (fraction); video FVD | Self-attention: 0.32 / 137.66; cross-attention: 0.20 / 145.26; no UNet: 0.24 / 141.62; action-only: 0.08 with no video metric. Supports the tested components together. Removing video also removes substantial capacity and pretraining, so it does not isolate video information alone. e-ablation |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table III. The refinement module changes the LIBERO90 conclusion. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the two bottom rows together before comparing methods. They separate raw CoVAR actions from the complete system with action refinement. The columns cover picking, opening/closing articulated objects, and compositional tasks; the paper reports 450 rollouts overall. Its dataset description places LIBERO90 at 128×128 resolution, and the training paragraph states that the refiner uses 450 video–action pairs. Those training pairs must not be conflated with the evaluation rollouts merely because the counts match. The upper baseline rows show why the distinction matters: raw CoVAR is below UVA in all three groups, whereas refined CoVAR exceeds every displayed baseline. e-liberoe-datae-traininge-refinement
What it supports. Refinement raises Pick from 0.592 to 0.873, Open & Close from 0.520 to 0.860, and Combination from 0.422 to 0.711. These are calculated gains of 28.1, 34.0 and 28.9 percentage points. The table supports the complete pipeline and a strong contribution from the refiner.
Where the evidence stops. Adding a trained, image/text-conditioned module changes both processing and capacity. The table does not isolate resolution mismatch as the cause, and the source leaves refinement-pair selection and train/test overlap unspecified.
Table V. The full combination performs best; removing video is the largest reported success drop. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read Rate as action success and the preceding four columns as video-quality measurements on the self-collected dataset. The first two rows replace Bridge Attention with self-attention (SA) or cross-attention (CA). The third replaces the UNet action head with the ResNet used in the cited baseline. The no-video row removes the video DiT and trains action generation conditioned on the initial image and instruction; its dashes indicate that video metrics do not apply. Compare each replacement with the bottom row, keeping metric directions separate. The full system has higher PSNR, SSIM and success, and lower LPIPS and FVD, than the applicable ablations. e-ablatione-training
What it supports. Full CoVAR reports 0.68 success versus 0.32 with SA, 0.20 with CA, 0.24 without UNet and 0.08 without video. Its FVD is 133.89, compared with 137.66, 145.26 and 141.62 for the three video-producing ablations. The pattern supports the tested design as a combination.
Where the evidence stops. The table does not report uncertainty or matched parameter and compute budgets. Removing the video branch also removes pretrained capacity; the result alone cannot attribute the entire difference specifically to bidirectional information exchange.
7. Analysis & limitations
7.1 What the evidence leaves open
The authors identify monocular video as limiting full 3D scene geometry. The supplied PDF does not test the proposed remedy of incorporating 3D foundation-model perception. e-limit
No error bars, repeated-seed results, or per-group rollout counts are supplied. Validation membership, real-data splits, success criteria, and baseline training budgets remain insufficiently specified for a tightly controlled comparison. e-videoe-calvine-liberoe-reale-ablation
A four-second generation time and open-loop execution leave responsiveness to disturbances unestablished. Selected generated/rollout images illustrate alignment but do not quantify general video–action consistency. e-rolloute-qualitative
7.2 Questions for discussion
- Would capacity- and compute-matched controls preserve Bridge Attention's gains? (e-ablation)
- Does action refinement correct systematic coarse plans or mostly learn an image-conditioned policy of its own? (e-refinement, e-libero)
8. Reproducibility audit
8.1 Requirements and known gaps
A reproduction needs the OpenSora-1.2 backbone, paired demonstrations, initial image/joint-state conditioning, separate DiTs, Bridge Attention, UNet, and the LIBERO90 refiner. Optimizer, learning rate, batch size, random seeds, checkpoint choice, action normalization, precise head configuration, and refinement training objective are not specified. e-architecturee-bridgee-refinemente-training
Proposed first checks: compare attention variants under matched training budgets and measure refinement gains on identical held-out rollouts. These would test the reported mechanisms, provided splits, trial counts, and uncertainty are explicitly recorded. e-ablatione-libero
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Separate Bridge Attention structure from capacity and training budget
Reader-proposed check, not performed: train the full bridge and the paper's SA and CA alternatives on the same fixed split of the self-collected paired data. Keep backbone initialization, UNet head, frame sampling, optimization budget, action normalization and rollout starts identical; disclose any unavoidable parameter-count differences. Repeat seeds and report success with trial counts and intervals, plus all four video metrics. Add a parameter-matched bridge variant with cross-modal attention blocked while retaining both streams. If the full bridge's advantage disappears under matched budgets, or blocking cross-modal exchange leaves success unchanged, the proposed communication mechanism is weakened; a stable gain would strengthen it. e-bridgee-traininge-ablation
Check 2: Test whether the LIBERO90 refiner uses its coarse action input
Reader-proposed check, not performed: fix a held-out LIBERO90 rollout set and explicitly separate it from the 450 refinement pairs. Evaluate raw actions, the trained refiner with correct coarse actions, and that same refiner with shuffled or zeroed coarse actions while keeping image and instruction unchanged. Train an image/text-only control with comparable capacity and training data to guard against interpreting test-time corruption alone as dependence. Report paired task success, per-group counts and confidence intervals. The claim that refinement improves an informative coarse plan predicts an advantage for correct coarse actions. Similar performance from the image/text-only control would suggest that extra policy learning explains much of the gain. e-refinemente-traininge-libero
8.3 Reading coverage
Visual audit: All four supplied text chunks were read individually and every page of the nine-page v1 PDF was rendered and visually inspected. The pass covered the title/authors/version and Figure 1 on p. 1; background on p. 2; Figure 2 and Eqs. (1)–(2) on p. 3; Eq. (3), refinement, dataset and training details on p. 4; rollout details, Figure 3 and Table I on p. 5; Figure 4, Tables II–III and rollout protocols on p. 6; Figures 5–7, Table IV and ablation descriptions on p. 7; Table V, ablation continuation and limitations on p. 8; and the references through p. 9. All six final original crops were separately viewed. Figure 2's modality projections, concatenations, split outputs and refinement ordering were checked against III-C/D; no claim-relevant figure/text conflict was found. The flow endpoint convention and unsquared printed norms are preserved. Static figures do not constitute viewing full rollout videos. No separate supplement, external code or experiments were inspected.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9. Appendix coverage: not present.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract; I. Introduction (PDF pp. 1–2)
- II. Related Work, A–B (PDF pp. 2–3)
- III. Method, A–D, Eqs. (1)–(3) (PDF pp. 3–4)
- IV. Experiments, A–D (PDF pp. 4–8)
- V. Conclusion (PDF p. 8)
- References [1]–[31] (PDF pp. 8–9)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- Identity: the inspected title and all eight authors match the catalog. The supplied artifact is arXiv:2512.16023v1 [cs.CV], dated 17 December 2025; no revision or edition difference is observed. Other versions were not supplied or compared.
- Acquisition omission: Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout. This reading addressed that omission by visually inspecting all nine PDF pages and all six final crops.
- Separate supplemental material availability has not been fully verified. No separate supplement was supplied; no appendix is present in the nine-page PDF.
- Code and external links were not inspected. No experiments were run or reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e-identityPDF p. 1, title block and arXiv margin
Exact title, eight authors, five affiliations, identifier 2512.16023v1, and 17 December 2025 date are visible.
Go to primary source ↓e-problemPDF pp. 1–3, Abstract; I. Introduction; II-B final paragraph
The work targets action labels for generated videos and describes separate modality spaces because video and action lack spatial alignment.
Go to primary source ↓e-architecturePDF p. 3, Figure 2(A–C), III opening and III-B; p. 4, III-B continuation
Initial image, joint states and text condition video/action co-generation through parallel DiTs; an MLP action encoder and UNet action decoder are described.
Go to primary source ↓e-flowPDF p. 3, III-A, Eqs. (1)–(2)
Data is indexed 0 and Gaussian noise 1; the displacement target is X_1−X_0. The printed loss sums two L2 norms without superscript squares; inference integrates a learned flow.
Go to primary source ↓e-bridgePDF p. 3, Figure 2(B); p. 4, III-C, Eq. (3)
Separate modality projections feed concatenated queries, keys and values to attention; output splits into video and action features. The authors contrast shared self-attention and bidirectional cross-attention.
Go to primary source ↓e-refinementPDF p. 3, Figure 2(C); p. 4, III-D
Rough action and initial-image embeddings feed self-attention, text cross-attention and decoding. The resolution mismatch motivates refinement; this section gives no refinement objective.
Go to primary source ↓e-dataPDF p. 4, IV-A, Datasets
CALVIN has approximately 20k demonstrations at 200×200; LIBERO90 has 90 tasks, 50 demonstrations per task, at 128×128; self-collected data has 1K demonstrations.
Go to primary source ↓e-trainingPDF p. 4, IV-A, Training Details; p. 5, first paragraph beneath Figure 3
OpenSora-1.2, 1.4B total parameters, 35-frame samples, co-training, 180×320 real videos, about one day/four unspecified GPUs, and 450 LIBERO90 refinement pairs are reported; detailed hyperparameters and frozen-module status are not.
Go to primary source ↓e-rolloutPDF p. 5, IV-A, Real Rollout Details
ROS supplies the current observation/joint state; a 35-frame pair uses 30 flow steps in about four seconds, with open-loop actions interpolated to 100 Hz.
Go to primary source ↓e-videoPDF p. 5, IV-B, Metrics and Quantitative Results; Table I, all rows and columns
Validation uses 100 CALVIN and 180 LIBERO90 generated videos. CoVAR improves all four metrics over UVA/PAD/UWM, but OpenSora has lower FVD on both datasets, higher CALVIN SSIM, and higher LIBERO90 PSNR.
Go to primary source ↓e-calvinPDF p. 6, IV-C, Calvin Dataset; Table II, all rows
ABCD training uses 20k videos and evaluation uses 200 new rollout scenes. Five grouped success rates are reported without group counts, uncertainty or chain-length metrics.
Go to primary source ↓e-liberoPDF p. 6, IV-C, Libero90 Dataset; Table III, all rows
450 rollouts cover three task groups. Refined rates are 0.873/0.860/0.711; raw rates 0.592/0.520/0.422; UVA 0.676/0.640/0.489. Group counts and uncertainty are absent.
Go to primary source ↓e-realPDF p. 6, IV-C, Real Dataset; p. 7, Table IV
UR5 evaluation reports nut/screw/dowel success fractions 0.64/0.74/0.70 versus RoboEnvision 0.04/0.10/0.12 and Unipi 0.00/0.06/0.02. Trial counts are not given.
Go to primary source ↓e-ablationPDF p. 7, IV-D and Figure 7; p. 8, Table V and Only Action DiT paragraph
Self-collected-data success is 0.68 full, 0.32 with self-attention, 0.20 with cross-attention, 0.24 with ResNet replacing UNet, and 0.08 without video. No-video retains initial-image/text conditioning; no matched parameter budget is reported.
Go to primary source ↓e-qualitativePDF p. 5, Figure 3 and caption; p. 6, Figure 4 and caption; p. 7, Figures 5–7 and captions
Figures distinguish generated frames from rollout examples; red trajectories denote ground truth, blue predictions, with green cross-attention in Figure 7's BA ablation. These are selected qualitative illustrations.
Go to primary source ↓e-limitPDF p. 8, V. Conclusion, final paragraph
The stated limitation is monocular input restricting full 3D geometry; 3D foundation-model perception is suggested as future work.
Go to primary source ↓8.5 Primary sources
CoVAR: Co-generation of Video and Action for Robotic Manipulation via Multi-Modal Diffusion ↗
PDF · 5,686 extracted words
Source fingerprint
940881ad12926c144ff550475111a2f679f49dcb8deee07b1f1f22f1d3fd5785