Video2Act: A Dual-System Video Diffusion Policy with Robotic Spatio-Motional Modeling
1. Paper overview
In one sentence: Video2Act filters observed-video features into spatial and motion conditions for a separate action diffusion policy, trading expensive perceptual refreshes for faster action-chunk generation. e03e04e05e10e11e13
| At a glance | What to know |
|---|---|
| Research problem | Author claim Static image representations may miss manipulation dynamics, while unprocessed video-diffusion features can retain distracting information and incur expensive updates. Video2Act asks whether explicit structure and motion filtering can make those features more useful and reusable for imitation learning. e02e03e06 |
| Core mechanism | |
| A key reported result | Multi-task manipulation on RoboTwin 1.0 and 2.0: RoboTwin 1.0: 54.6 ± 1.1; RoboTwin 2.0: 54.1 ± 1.4. Mean task success rate (%). Six tasks per benchmark, 100 demonstrations per task; 50 seeded rollouts per task, evaluation repeated three times. Best competing benchmark means: π0.5 at 48.1 ± 0.7 in 1.0 and RDT at 46.0 ± 0.9 in 2.0; differences are 6.5 and 8.1 percentage points. Highest reported mean on each suite, with substantial task variation. The source calls its repeated-evaluation dispersion 'variance'; the ± notation is retained without relabeling it as SD or a confidence interval. e08e09 |
| Reading caution | Source description Failure examples include flower-stem misalignment, closing a gripper above a carrot, premature release during cucumber handover and triangle-pushing overshoot. Similar object/background colors also weaken localization. Autonomous error detection and correction remain future work. e22e23 |
Core contributions
Figure 3. Observed-video structure and motion become slow conditioning for a separate fast action policy. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow the blue paths from the high-resolution and long-horizon observations through the 3D VAE and video transformer. The upper branch uses the spatial filter; the lower branch uses the motion filter. Their separate formers compress the features before they reach action-head cross-attention. The green path carries current image tokens directly into that head, while the black path carries language tokens. The dashed green loop belongs to action denoising. Figure 3's circled-plus junction is schematic: the explicit training definition on PDF p. 6 specifies Cat(S_t, M_t), so it does not justify assuming elementwise addition. The blue frequency label is explained in Section 3.4 as one video-feature update per n fast-system steps. e03e04e05e06e07e24
What it supports. The policy has two distinct neural systems and two temporal cadences. Its slow path extracts features from observed history; its fast path denoises joint-position action chunks using those features and current observations. This architecture supports representation reuse without requiring a decoded future video at every control update.
Where the evidence stops. The figure does not specify frozen modules, the number of denoising iterations, or how robot state enters the network. The problem definition includes state, so the missing state arrow is a documentation gap, not evidence that state is excluded.
2. Motivation
2.1 The problem and the proposed response
Static image representations may miss manipulation dynamics, while unprocessed video-diffusion features can retain distracting information and incur expensive updates. Video2Act asks whether explicit structure and motion filtering can make those features more useful and reusable for imitation learning. e02e03e06
2.2 What this reading follows
A robot can keep acting between expensive video-model updates if its cached features remain useful. Video2Act builds that idea around two views of observation history: a short, detailed stream for spatial structure and a longer, lower-resolution stream for motion. Fixed Sobel and Fourier operators refine the video features before a separate diffusion action head consumes them alongside current image tokens. The experiments connect this design to executed manipulation, including physical robot trials. Read the evidence in two stages: first test whether filtering improves the policy, then ask what performance survives when those features are refreshed less often. The latter is a measurable compromise, not free acceleration. e03e04e05e10e11e13
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Not assigned |
| Architecture | Not assigned |
| Prediction paradigm | Not assigned |
| Quadrant | Not assigned |
This table preserves the labels recorded at reading time. The current major category is VLA. View the current classification.
3.1 Evidence-based assessment
Insufficient evidence to decide
Every recorded taxonomy field is unassigned, so there is no substantive catalog claim to confirm. Architecture evidence supports a modular video-representation network plus a separate action diffusion model. The described policy neither jointly predicts future video and actions in one backbone nor infers actions from a generated video using inverse dynamics. Video-derived dynamics priors are used as inference-time conditioning, rather than as an explicit action-conditioned rollout planner. e03e04e06e07
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Why an edge operator can help a video representation
The spatial branch starts with learned features, not a hand-built object mask. Hunyuan processes a short high-resolution observation window, and Sobel kernels then compute horizontal and vertical gradients independently in each feature channel. Taking their magnitude emphasizes where that channel's activation changes spatially. The spatial former compresses this representation before it conditions the action DiT. This ordering matters: a boundary in a learned feature channel need not be the literal outline of an object in RGB space. The authors interpret the resulting cues as useful structure for manipulation, supported by the fixed-ratio success ablation. Reader interpretation: these operators impose a helpful feature bias rather than supplying guaranteed segmentation. Appendix B.1 strengthens that interpretation because Scharr also works well, suggesting the benefit is not unique to one kernel. e03e04e05e10e16
5.2 What the motion branch measures—and what remains unknown
The motion branch keeps a longer observation window at lower spatial resolution. At each feature-grid location and channel, the method transforms the temporal sequence into Fourier coefficients, retains a high-pass subset, and reconstructs a real-valued sequence. The intended effect is to suppress slowly varying background content while preserving temporal changes useful for action learning. This mechanism differs from predicting optical flow or simulating a future state: the paper defines a filtered latent signal. The large drop when the long window is shortened is consistent with useful temporal context, but it does not isolate which motion information matters. Reader interpretation: camera movement can also change features, so clean separation of object motion from viewpoint motion remains a testable hypothesis. The unspecified numerical mask prevents an exact filter reconstruction from the paper alone. e04e05e18e24
5.3 Cache semantics while refreshing visual feedback
The fast action system has two relevant sources of visual information: cached video-derived conditions and current image tokens. That arrangement allows an action prediction to respond to a new image without recomputing the expensive video branch every time. The action DiT denoises a chunk of joint-position commands, while System 2 refreshes its representations on a slower schedule. This is the operational reason to distinguish model-call frequency from the number of action entries produced. Figure 4 shows increasing inference frequency as video updates become less frequent, but success also falls relative to synchronous operation. Reader interpretation: the data support reuse over a limited interval, not indefinitely stable semantics. The documented premature cucumber release further shows that having temporal features does not guarantee correct timing at contact transitions. e03e06e07e11e22
5.4 Training and inference
During training
System 1 learns conditional noise prediction on demonstrated action chunks. The paper identifies pretrained components, but does not provide a complete staged training recipe or establish which Video2Act modules are frozen versus updated. e03e07e24
Simulation uses 100 expert demonstrations per task and multi-task models. For VPP and VPDD, the authors fine-tune Stable Video Diffusion and replace the action head with their pretrained diffusion head. These are adapted baselines, with a different video backbone from Video2Act, so the comparison does not isolate filtering alone. e08e03
Real-world policies are task-specific, trained from 100 teleoperated demonstrations per task containing synchronized joint positions and three RGB views. Evaluation uses the final checkpoint. e13e14
During inference
System 2 updates once per n fast-system steps; System 1 reuses its latest conditioning while receiving fresh image features and denoising action chunks. This is feature-conditioned action generation, not inverse dynamics applied to generated future frames. e03e06e07
The authors select a 1:8 update ratio as a speed–accuracy compromise. The reported 5.96 Hz is model inference frequency; multiplying by H = 64 motivates their approximately 380 Hz claim. That multiplication does not establish 380 independently observed and replanned control decisions per second. e11e14
5.5 Implementation flow
- Encode observations
SigLIP-ViT-L/14 supplies image tokens and a text encoder supplies instruction embeddings. Hunyuan's pretrained 3D VAE and first 25 of 60 transformer blocks provide System 2 features at inversion timestep t_diff = 0, using observed images rather than an imagined future. e03e04
- Separate structure from temporal variation
The spatial stream uses resolution 512×768 and T_s = 2; the motion stream uses 256×256 and T_l = 16. Channel-wise Sobel gradients produce spatial magnitudes. A temporal DFT, high-pass frequency mask and inverse DFT produce motion features at each spatial location and channel. e04e05
- Compress and condition actions
Two Q-formers compress the branches, with token length 256 selected in the ablation. A separate 1B-parameter action DiT receives image, text and video conditioning through cross-attention. The explicit objective specifies concatenation of spatial and motion features; Figure 3's circled-plus junction should not independently be interpreted as elementwise addition. e03e07e12
6. Experiments & results
Video2Act turns observed video into conditioning for a separate robot action policy. Sobel filtering emphasizes spatial structure in video-model features, while temporal Fourier filtering emphasizes motion. A slow video network refreshes these representations for a faster diffusion action head. The strongest evidence is improved executed manipulation on RoboTwin and a small real-robot evaluation; the speed claims require separating action-chunk throughput from fresh-feedback control.
6.1 Read the original evidence
Table 1. Video2Act leads both benchmark averages, while individual tasks still favor other policies. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Treat the two shaded benchmark blocks as separate six-task evaluations. Read each task column before the Mean column: block handover is much easier for this model than container placement, and the same policy ranking does not hold everywhere. PDF p. 6 specifies 100 demonstrations per task, 50 seeded rollouts, and three evaluation repetitions. It also explains that the VPP and VPDD implementations use fine-tuned Stable Video Diffusion with replaced action heads. Compare the numerical cells rather than relying on bold type: the turn-switch column gives VPP 45.3 and Video2Act 40.7, although the latter is bold. e08e09e03
What it supports. The reported means are 54.6 ± 1.1 for RoboTwin 1.0 and 54.1 ± 1.4 for RoboTwin 2.0. Relative to the strongest competing mean in each suite, the gains are 6.5 percentage points over π0.5 and 8.1 points over RDT. These support a benchmark-average advantage under the stated protocol.
Where the evidence stops. The source describes the repeated-evaluation dispersion as variance but displays ± values; it does not establish a confidence interval. The cross-method comparison changes video backbones as well as feature processing, so the filtering ablation is the closer mechanism test.
Figure 5. Physical robot evaluation shows a large mean gain and a persistent flower-insertion weakness. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. The upper image groups establish the objects and manipulation types; their order differs from the lower chart, so use the chart's task labels to read scores. Pink bars are Video2Act, orange bars RDT, and yellow bars VPP. Each task uses 100 teleoperated demonstrations and ten evaluation rollouts, as stated by the caption and Section 4.3. Read the dual-carrot group carefully: Video2Act and VPP both achieve 60%. The flower task is the weakest for all three policies. The surrounding text says these are task-specific policies, which differs from the multi-task simulation setup. e01e13e14
What it supports. Video2Act averages 73.3% across these six tasks, compared with 50.0% for RDT and 51.7% for VPP. Its flower score is only 20%, despite stronger performance on laptop closure, cucumber handover and single-carrot placement. These are robot-execution outcomes, not video-generation quality scores.
Where the evidence stops. Ten rollouts make each task score change in ten-percentage-point increments, and no uncertainty bars are supplied. Section 4.3 names π0 as another baseline, but the figure contains no π0 bar; no missing value can be inferred.
Table 6. Higher success under visual shifts comes with an unresolved lighting-average inconsistency. Original paper, p. 22 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Within each model block, compare Clean against Background and Light, then check the task cells against the Average row. Appendix F specifies 50 differently seeded tests per scene. Video2Act has a specific inconsistency: its six Light entries sum to 248, giving 41.3% by arithmetic averaging, while the Average cell and appendix prose report 43.0%. Keep both visible as conflicting source information. Task rows also prevent overgeneralization: Video2Act scores 20% on turn switch under lighting variation, versus 30% for π0 and π0.5. This table concerns simulated background and lighting shifts; Figure 7 instead addresses real-world shifts but has conflicting task labels and caption. e19e20
What it supports. The lighting aggregate is inconsistent: 43.0% is printed, but its six task cells average 41.3% (reader arithmetic). Relative to 54.1% clean, the reported 11.1-point loss becomes about 12.8 points using those cells. Background success is 32.7%, a 21.4-point loss. Either lighting value and the background mean exceed π0.5’s 19.7% and 17.7%, respectively, but the discrepancy remains unresolved.
Where the evidence stops. No explanation or alternative weighting is supplied for the lighting mismatch. Randomization distributions and uncertainty are also unspecified. This table compares complete methods; it does not separately ablate the filters under these same shifts, so it cannot isolate their causal contribution.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Multi-task manipulation on RoboTwin 1.0 and 2.0 Six tasks per benchmark, 100 demonstrations per task; 50 seeded rollouts per task, evaluation repeated three times. | RoboTwin 1.0: 54.6 ± 1.1; RoboTwin 2.0: 54.1 ± 1.4. Mean task success rate (%) | Best competing benchmark means: π0.5 at 48.1 ± 0.7 in 1.0 and RDT at 46.0 ± 0.9 in 2.0; differences are 6.5 and 8.1 percentage points. Highest reported mean on each suite, with substantial task variation. The source calls its repeated-evaluation dispersion 'variance'; the ± notation is retained without relabeling it as SD or a confidence interval. e08e09 |
| Six real-world manipulation tasks Agilex Cobot Magic; task-specific policies, 100 demonstrations and 10 final-checkpoint rollouts per task under varied object positions. | 73.3; individual task scores in Figure 5 are 100, 90, 90, 60, 80 and 20 in plotted order. Average task success rate (%) | RDT: 50.0; VPP: 51.7 (means also shown in Figure 1). Improvement over VPP is approximately 21.7 percentage points. Physical execution evidence, but only ten trials per task and no uncertainty bars. Dual flowers remains difficult at 20%; dual carrots ties VPP at 60%. Section 4.3 mentions π0, but Figure 5 supplies no π0 result. e01e13 |
| Spatial and motion feature ablation RoboTwin 1.0 six-task average, fixed 1:1 operating ratio. | No VDM: 44.9; raw VDM: 46.3; +Sobel: 50.3; +FFT: 51.3; +Sobel+FFT: 54.6. Success rate (%) | Combined filtering gains 8.3 percentage points over raw features; individual gains are 4.0 and 5.0 points. Supports complementary feature processing under this configuration, without proving semantic object segmentation or causal dynamics identification. e10 |
| Asynchronous inference tradeoff RoboTwin 1.0 update-ratio sweep; H = 64. | At 1:8: 5.96 Hz; plotted success is approximately 52%, versus 54.6% at 1:1. Model inference frequency and task success | Increasing the interval raises throughput; 1:16 further reduces plotted success. A compromise rather than unchanged accuracy. Approximately 380 action entries per second is the authors' chunk-based rate claim, not a demonstrated fresh-feedback frequency. e11 |
| RoboTwin 2.0 visual-shift generalization Clean, randomized-background and randomized-light conditions; Appendix F specifies 50 seeded tests per scene. | Video2Act: 54.1 clean and 32.7 background. Light: 43.0 is the printed average, but the six displayed task scores (72, 64, 26, 32, 34, 20) sum to 248 and average 41.3 (reader arithmetic); the source does not resolve this discrepancy. Six-task average success rate (%) | π0.5: 42.8, 17.7, 19.7, respectively. The background decline is 21.4 percentage points. Appendix F reports an 11.1-point lighting decline using its printed 43.0; the displayed task cells instead imply about 12.8 points below the printed clean mean. Either lighting value exceeds the competing means, but neither the aggregate nor task cells can be silently corrected. Randomization distributions and uncertainty are not supplied. e19 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 4(a–c). Filtering helps at a fixed update rate; reducing video refreshes introduces a visible accuracy tradeoff. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with panel (a), where every variant uses the same 1:1 system ratio. The progression is no VDM, raw features, spatial filtering, motion filtering, and both filters. Panel (b) asks a different question: its red line uses the left success axis, while its dashed blue line uses the right frequency axis. Moving right makes video updates less frequent. Panel (c) changes the formers' token length; its success axis also starts above zero, so bar heights exaggerate small absolute differences. The text on PDF p. 7 gives the precise 1:8 frequency, and PDF p. 8 explains why 256 tokens were selected. e10e11e12e14
What it supports. At 1:1, both filters reach 54.6%, compared with 46.3% for raw features. At 1:8, the model reaches 5.96 Hz but the plotted success is only about 52%. The evidence therefore supports complementary filtering and a useful operating compromise; it does not support identical success at the fastest settings.
Where the evidence stops. Do not pair 54.6% from the 1:1 feature ablation with 5.96 Hz from 1:8 as one measured operating point. The approximately 380 Hz claim multiplies inference frequency by a 64-action chunk; it does not establish equally frequent fresh observations.
Table 5. The reported sensitivity is stronger to shortening motion history than to extending the spatial window. Original paper, p. 16 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each row as a change from the default, rather than as a joint sweep. T_s is the short spatial window and T_l the long motion window. The Resolution column refers to the high-resolution branch; Section 3.3 on PDF p. 4 separately specifies the motion branch at 256×256. The first row uses T_s = 2, T_l = 16 and 512×768. The next row changes only T_s, the third only T_l, and the fourth lowers spatial resolution. Appendix B.3 reports these as six-task RoboTwin 1.0 averages and chooses the default as an efficiency–accuracy compromise. e04e18e24
What it supports. Extending T_s from 2 to 8 increases reported success by 0.7 percentage points. Shortening T_l from 16 to 8 lowers it by 5.9 points, and reducing spatial resolution lowers it by 2.3 points. Within these tested settings, both longer temporal context and detailed spatial input contribute.
Where the evidence stops. No dispersion or per-variant timing appears in this table, so the small short-window difference cannot establish a significant gain or quantify the efficiency tradeoff. The method's temporal indexing and frame-sampling implementation are not fully specified.
7. Analysis & limitations
7.1 What the evidence leaves open
Failure examples include flower-stem misalignment, closing a gripper above a carrot, premature release during cucumber handover and triangle-pushing overshoot. Similar object/background colors also weaken localization. Autonomous error detection and correction remain future work. e22e23
Grad-CAM examples and action distributions are qualitative diagnostics. The latter use only 40 successful trajectories per configuration, so broader plotted coverage cannot establish the distribution of failures or causal generalization. e02e15
Source inconsistencies remain unresolved: Table 1 bolds Video2Act's turn-switch score below VPP's; Figure 7 labels pick carrot while its caption/text name pick dual flowers, and its cucumber original score differs from Figure 5. No numerical generalization conclusion here uses Figure 7. e09e13e20
Additional lighting and Labubu tests lack explicit trial counts. The toy's absence from video-model pretraining is an author assertion without a supplied data audit. e21
Table 6 has an unresolved arithmetic inconsistency: its six Video2Act Light entries average 41.3%, while the Average cell and Appendix F state 43.0%. The reported 11.1-point decline depends on that disputed aggregate; task-row arithmetic gives about 12.8 points. These calculations diagnose the conflict without establishing which source entries are erroneous. e19
7.2 Questions for discussion
- Does temporal filtering remain beneficial when camera motion and object motion vary independently?
- How much success is lost when cached video features become stale during abrupt contact events?
8. Reproducibility audit
8.1 Requirements and known gaps
A faithful setup needs Hunyuan features, the image/text encoders, two Q-formers, the action DiT and synchronized demonstrations. The paper omits optimizer, learning rate, batch size, training duration/compute, module-freezing policy, exact FFT cutoff, diffusion sampling schedule, and detailed state/multicamera routing. e24
Deployment is reported on one NVIDIA 4090 with 24 GB VRAM: FP16 System 2 requires 21.04 GiB and 144.4 W, with 587.9 ms cold start. Table 4 also reports 587.9 ms for 20 dual-stream plus 5 single-stream blocks, including feature processing; it is not just action-head latency. e14e17
The platform uses four 6-DoF Piper arms for teleoperation, a RealSense D435 head camera and two Orbbec Dabai wrist cameras. Table 3 lists RGB resolution 640×480 and 30 fps. Table 5 supports checking window length and spatial resolution: shortening T_l to 8 lowers success to 48.7%, and reducing spatial resolution lowers it to 52.3%. e14e18
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Test filtering with a shared video backbone and controlled temporal information
Reader-proposed check, not performed: reproduce the no-VDM, raw, Sobel, FFT and combined variants on the same six RoboTwin 1.0 tasks, using identical Hunyuan features, demonstration splits, action-head initialization, token budget and 1:1 update ratio. Evaluate 50 matched seeds per task across three repetitions and report uncertainty on paired success differences. Add a combined-filter control trained and evaluated with temporally shuffled motion windows while leaving spatial inputs unchanged. A gain for filtering with no additional benefit from chronological windows would weaken the interpretation that coherent temporal structure explains the improvement. Record the exact frequency mask and frame sampling so this comparison is reproducible. e04e05e08e10e24
Check 2: Measure stale-feature cost separately from action-chunk throughput
Reader-proposed check, not performed: compare raw and filtered video conditions at update ratios 1:1, 1:4, 1:8 and 1:16, holding H = 64, the action head, hardware and evaluation seeds fixed. Timestamp camera observations, video-feature completion, each action-head call and command execution; report cold start, feature age, calls per second and executed commands per second separately. Measure task success and handover release failures as feature age grows. If filtering preserves success better than raw features specifically at longer intervals, that supports the representation-reuse claim. If only command throughput rises while success or reaction delay deteriorates, the fast-control interpretation needs a narrower boundary. e06e11e14e22
8.3 Reading coverage
Visual audit: Visually inspected the title/byline and v3 stamp, all original Figures 1–16 and Tables 1–6, and supporting method, evaluation, hardware, ablation and failure text on the declared pages. All six final crops were regenerated at their retained bounds and separately viewed. Table 5 uses a 350-DPI render; the other crops use 200 DPI. Figure 3 arrows and fusion junction were checked against Sections 3.3–3.4 and their equations. Table 6 and all of Appendix F on PDF p. 22 were inspected: the six Video2Act Light entries sum to 248 and average 41.3%, conflicting with the printed 43.0% and the prose’s 11.1-point decline. The report preserves the source values and distinguishes reader arithmetic. The previously identified Figure 4 accuracy narrative, Table 1 boldface, Figure 5 baseline coverage and Figure 7 label discrepancies remain disclosed. Reference-only PDF pp. 10–12 were read in the complete eight-chunk text pass. No external videos, project material or separate supplement were inspected.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract; 1. Introduction
- 2. Related Work (2.1–2.2)
- 3. Method (3.1–3.4), including equations (1)–(6) and the diffusion objective
- 4. Experiments (4.1–4.4); 5. Conclusion
- References
- A. Additional Experimental Setup and Data Details (A.1–A.3)
- B. Additional Ablation Study (B.1–B.4)
- C. Additional Visualizations (C.1–C.3)
- D. Additional Real-World Evaluations
- E. Failure Analysis
- F. Additional Quantitative Generalization Experiments
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Identity: the title and all ten authors match the supplied catalog. The inspected title page identifies arXiv:2512.03044v3 [cs.RO], 24 March 2026. The catalog records initial submission on 2 December 2025. This report concerns the supplied v3 artifact; earlier editions and their changes were not supplied or compared.
- All eight supplied text chunks were read individually, covering all 22 PDF pages. The embedded supplement occupies PDF pp. 13–22, with printed page numbers restarting at 1. Evidence locations use PDF page numbers.
- Text extraction does not reconstruct figure images; the original PDF figures and tables were inspected separately on PDF pp. 1–9 and 13–22. Reference-only pages 10–12 were read as text.
- Separate supplemental material availability has not been fully verified.
- External project pages, code, checkpoints, datasets and videos were not accessed. No paper code was inspected and no experiment was reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title/byline, arXiv margin stamp and Figure 1
Exact title and ten-author byline match the catalog; stamp reads arXiv:2512.03044v3 [cs.RO], 24 Mar 2026. Affiliations are Peking University, AI²Robotics and Hong Kong University of Science and Technology. Figure 1 shows real-world averages 50.0 (RDT), 51.7 (VPP), 73.3 (Video2Act).
Go to primary source ↓e02PDF pp. 2–4, Introduction, Figure 2 and Section 3.1
Motivation compares static-image encoders and VDM representations on block handover with head and wrist cameras; foreground-focused Grad-CAM patterns motivate explicit spatial and motion extraction.
Go to primary source ↓e03PDF pp. 4–5, Section 3.2 and Figure 3
Problem conditions joint-position action chunks on images, robot state and language. Architecture names SigLIP-ViT-L/14, a text encoder, Hunyuan 3D VAE and first 25 transformer blocks, spatial/motion Q-formers, and a separate 1B action DiT with cross-attention.
Go to primary source ↓e04PDF p. 4, Section 3.3, equations (1)–(3)
Features are extracted from observed images at t_diff = 0. Branches use T_s = 2 at 512×768 and T_l = 16 at 256×256. Spatial and temporal filters operate on latent features, not on decoded imagined futures.
Go to primary source ↓e05PDF pp. 4–5, Section 3.3, Sobel kernels, equation (4), gradient-magnitude definition and equations (5)–(6)
Spatial filtering applies channel-wise horizontal/vertical Sobel convolution and gradient magnitude. Temporal DFT is followed by a high-pass mask B and real inverse-DFT reconstruction. No numerical frequency cutoff is given.
Go to primary source ↓e06PDF pp. 5–6, Section 3.4, Asynchronous Frequency; Figure 3
Slow System 2 features periodically condition fast System 1, which uses recent observations. The described operating ratio is 1:n; the diagram routes both feature branches and current image tokens into action-head cross-attention.
Go to primary source ↓e07PDF p. 6, Section 3.4, Training Objective
The conditional action DiT predicts Gaussian noise from noisy action chunks, diffusion timestep embedding, image features, textual features and F_VDM = Cat(S_t,M_t), under mean squared noise error. The explicit formulation specifies concatenation despite the diagram's circled-plus junction.
Go to primary source ↓e08PDF p. 6, Section 4.1; PDF p. 13, Appendix A.1
Six tasks each from RoboTwin 1.0/2.0, 100 expert demonstrations per task, multi-task models, 50 seeded evaluation rollouts per task repeated three times. VPP/VPDD use fine-tuned Stable Video Diffusion and the authors' pretrained action head. Appendix A.1 describes automated expert generation.
Go to primary source ↓e09PDF p. 7, Table 1, Mean columns and turn-switch column
Video2Act means are 54.6±1.1 and 54.1±1.4; π0.5 has 48.1±0.7 in 1.0 and RDT 46.0±0.9 in 2.0. Turn switch shows bold 40.7 for Video2Act but 45.3 for VPP. Section 4.1 on p. 6 calls repeated-evaluation dispersion variance.
Go to primary source ↓e10PDF p. 7, Figure 4(a), Section 4.2, Spatio-motional Feature Extraction
At fixed 1:1, no VDM/raw/+Sobel/+FFT/+Sobel+FFT score 44.9/46.3/50.3/51.3/54.6 percent on six RoboTwin 1.0 tasks.
Go to primary source ↓e11PDF p. 7, Figure 4(b) and Section 4.2, Dual-System Operating Frequency Ratio
Ratio sweep spans 1:1–1:16. Text reports 5.96 Hz at 1:8, H = 64 and approximately 380 Hz control; figure shows success declining from 54.6 at 1:1 to about 52 at 1:8 and further at 1:16. Sobel/FFT overheads are 0.56/1.33 ms averaged over 50 runs.
Go to primary source ↓e12PDF p. 7, Figure 4(c); PDF p. 8, Spatial and Motion Former Token Length
Token lengths 128/256/512/1024 score 50.7/54.6/54.0/54.7 percent. The authors choose 256 to balance performance and computation.
Go to primary source ↓e13PDF p. 8, Figure 5 and Section 4.3
Six task-specific real-world evaluations use 100 demonstrations and 10 final-checkpoint rollouts per task. Figure 5 scores Video2Act/RDT/VPP as 100/70/60, 90/70/60, 90/60/70, 60/40/60, 80/50/50, 20/10/10 for laptop/cucumber/carrot/dual carrots/triangle/dual flowers. π0 is named in text but absent from plotted results.
Go to primary source ↓e14PDF p. 8, Section 4.3 hardware paragraph; PDF pp. 13–14, Figure 8, Appendix A.2–A.3 and Tables 2–3
One NVIDIA 4090 (24 GB), FP16 System 2 at 21.04 GiB and 144.4 W, 587.9 ms cold start. Platform has four 6-DoF Piper arms, D435 head and two Orbbec Dabai wrist cameras. Table 3 lists 640×480 and 30 fps; demonstrations synchronize joint positions and three RGB views.
Go to primary source ↓e15PDF pp. 8–9, Section 4.4 and Figure 6; PDF p. 16, Appendix B.4 and Figure 10
Action-distribution visualizations use 40 successful rollouts per configuration in dual bottles pick hard, with controlled initial bottle positions, projecting right- and left-arm end-effector positions into X–Z. Authors interpret broader filtered-feature distributions as better demonstration coverage.
Go to primary source ↓e16PDF p. 15, Appendix B.1 and Figure 9
With FFT fixed, Scharr and Laplacian alternatives reach 53.3 and 52.7 percent versus Sobel+FFT 54.6; uncertainty is not plotted.
Go to primary source ↓e17PDF pp. 15–16, Appendix B.2 and Table 4
Default 20 dual-stream plus 5 single-stream blocks achieve 54.6 percent and 587.9 ms System 2 latency, defined as forward pass plus feature processing. Full 20+40 blocks achieve 54.0 and 1284.6 ms; depth does not monotonically improve success.
Go to primary source ↓e18PDF p. 16, Appendix B.3 and Table 5
Default T_s/T_l/resolution 2/16/512×768 scores 54.6. Increasing T_s to 8 gives 55.3; reducing T_l to 8 gives 48.7; reducing high-resolution input to 256×256 gives 52.3. These are six-task RoboTwin 1.0 averages without reported dispersion.
Go to primary source ↓e19PDF p. 22 (supplement printed p. 10), Table 6, Video2Act Light column (all six task rows and Average), other Average cells, and Appendix F
Appendix F specifies 50 seeded tests per scene. Table 6 prints clean/background/light averages of 54.1/32.7/43.0 for Video2Act and 42.8/17.7/19.7 for π0.5. Video2Act Light task cells are 72.0, 64.0, 26.0, 32.0, 34.0 and 20.0: their sum is 248 and their arithmetic mean is 41.3%, conflicting with the printed 43.0%. Appendix F repeats 43.0 and an 11.1-percentage-point decline; the task-cell mean implies about a 12.8-point decline from printed clean 54.1 (reader arithmetic). The source supplies no explanation or alternative weighting. Randomization distributions and uncertainty are unspecified. Turn-switch Light scores are Video2Act 20.0 and π0/π0.5 30.0.
Go to primary source ↓e20PDF p. 9, Figure 7 plot, caption and Section 4.4 Generalization Experiments; PDF p. 8, Figure 5
Figure 7 plots Handover Cucumber and Pick carrot, while caption/text name pick dual flowers and handover cucumber. Its original cucumber bar is 100 versus 90 in Figure 5; the reason is not supplied.
Go to primary source ↓e21PDF pp. 19–21, Appendix D and Figure 15 on p. 20
Extra cucumber lighting evaluation compares raw, Sobel and combined filters at 1800/1300/800/300 lux; at 300 lux scores are 30/40/70. The text reports 80 percent success on a Labubu pick-and-place task and asserts the toy was unseen during VDM pretraining. Trial counts and pretraining-data verification are absent.
Go to primary source ↓e22PDF pp. 21–22, Appendix E and Figure 16
Reported failures are flower grasp-offset accumulation, incorrect carrot grasp height, premature giver release in cucumber handover, and triangle-push overshoot. More demonstrations, training constraints and System 2 error correction are proposed future directions.
Go to primary source ↓e23PDF p. 17, Appendix C.1 and Figure 11
Real-world Grad-CAM comparisons generally favor stable VDM object focus, but text explicitly notes less precise localization when object and background share nearly identical colors.
Go to primary source ↓e24PDF pp. 4–8, Sections 3.2–3.4 and 4.1–4.3; PDF pp. 13–16, Appendices A–B
The supplied method/setup text specifies the loss, backbone, branch windows, compression choice and evaluation setup, but not optimizer, learning rate, batch size, training duration/compute, frozen-module policy, numerical FFT mask, diffusion sampling schedule or detailed state/multicamera routing. Figure 3 does not show a robot-state path.
Go to primary source ↓8.5 Primary sources
Video2Act: A Dual-System Video Diffusion Policy with Robotic Spatio-Motional Modeling ↗
PDF · 12,292 extracted words
Source fingerprint
83cf1bbbd1a5a9824265da9063c8c4929291faab872be3b812b926dc259821fa