Unified Video-Action Joint Denoising for Dexterous Action and Data Generation
1. Paper overview
In one sentence: Donk reuses one video-action denoising core for observed-scene prediction and text-conditioned paired data generation, with initial hand-camera geometry and an asymmetric visual-to-action information path. e_modese_architecturee_attentione_anchore_action_resultse_video_resultse_text_resultse_gaps
| At a glance | What to know |
|---|---|
| Research problem | Source description Dexterous learning needs fine hand motion aligned with evolving object interactions, but paired visual-action data are costly. The authors ask whether an action-aligned video prior can also generate training pairs from language, while preserving the observation-conditioned prediction mode. This is a proposed data-generation role, not evidence of successful downstream robot learning. e_probleme_gaps |
| Core mechanism | |
| A key reported result | OakInk2 offline dexterous trajectory prediction (TI2VA): Donk: RMSE 0.238; 0.049/0.079/0.046/0.029 m; ROT 14.95°. Hand RMSE; ADE/FDE/DTW-S/DTW-L (m); wrist ROT (degrees), all lower is better. First-person benchmark; ten sampled futures per example; trajectory numbers below use best-of-10. DreamZero-alike: RMSE 0.262; 0.057/0.094/0.054/0.037 m; ROT 19.00°. VITRA has better ROT, 14.64°. Best hand-pose and translation scores among listed methods. Best-of-K evaluation does not establish single-sample policy performance; hand-RMSE units are unspecified. e_action_protocole_action_results |
| Reading caution | Reader analysis There is no closed-loop robot trial or downstream policy trained on generated data. Figures 4–5 offer selected qualitative alignment examples; they do not quantify contact validity, dynamics accuracy or out-of-distribution execution success. No statistical uncertainty is reported. e_qualitativee_gaps |
Core contributions
- Source description
A shared video/action denoiser combines asymmetric attention, initial-state map control, hand-focused supervision and a frozen video teacher. e_architecturee_attentione_anchore_losses
- Author claim
The authors claim the first T2VA exploration for dexterous data generation. The supplied experiments demonstrate generated pairs and offline prediction benefits; priority is not independently established here. e_probleme_qualitative
Figure 2. One denoising core produces the visual rollout and its hand trajectory. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start at the left with the image/video, MANO action sequence and initial hand state. The blue and green latent streams are noised before entering the same DiT; the purple state stream supplies initial geometry. Read the right-hand decoders as outputs of the generative pipeline, while Section 3.3 specifies that the training targets are flow velocities. Snowflakes mark frozen modules. The teacher and Gaze paths supply training losses, and the anchor-map controller supplies spatial hints. Cross-check the central fire symbol with the text: only a small subset of Wan layers is trained, alongside the added interfaces. e_architecturee_anchore_lossese_inference
What it supports. The architecture shares computation across video and action generation instead of extracting a trajectory after rendering a completed video. Its geometric control still matters: the first-frame anchor provides an explicit spatial scaffold. Removing the image changes conditioning, while the core denoiser and paired output interface are retained.
Where the evidence stops. The upward “Context Prior” arrow is not given a separate computation in Eqs. (6)–(7); those equations define map encoding and first-frame injection. Do not infer an additional feedback algorithm from the schematic. The teacher is training-only.
2. Motivation
2.1 The problem and the proposed response
Dexterous learning needs fine hand motion aligned with evolving object interactions, but paired visual-action data are costly. The authors ask whether an action-aligned video prior can also generate training pairs from language, while preserving the observation-conditioned prediction mode. This is a proposed data-generation role, not evidence of successful downstream robot learning. e_probleme_gaps
2.2 What this reading follows
Donk asks whether a dexterous video-action model can do more than predict the next interaction from a camera image. Its second interface starts from a language instruction and generates both a video and a matching MANO hand trajectory. The key qualification is that “text-only” describes the external input: a separate initializer still supplies first-frame hand-camera geometry. Read the architecture alongside its attention mask, then keep the action and video evaluations separate. The evidence is strongest for offline hand prediction and competitive visual quality. Whether the synthetic pairs improve a deployed robot policy is still an open experimental question. e_modese_architecturee_attentione_anchore_action_resultse_video_resultse_text_resultse_gaps
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Foundational work |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
This table preserves the labels recorded at reading time. The current major category is WAMs. View the current classification.
3.1 Evidence-based assessment
Conflicts with the recorded classification
The broad simulator, data-generation and policy themes fit, but architecture/prediction marked Not applicable obscures a concrete shared-transformer method. The denoising core supports a One Model, joint future-video/action prediction assessment: video, action and anchor tokens share a backbone and have separate output heads. This is not post-hoc inverse dynamics. The auxiliary anchor initializer is separate, so One Model describes the core rather than every component. The recorded snapshot is preserved. e_architecturee_attentione_anchore_inference
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Start with the initial condition, not an imagined future plan
The two interfaces differ in what is observed at the beginning. TI2VA receives the image, hand state and camera intrinsics, so both scene appearance and the hand-camera configuration are grounded in an observation. T2VA has no image, but its separately trained initializer samples plausible first-frame geometry from language. Rendering that geometry as a skeleton map makes the hand’s position accessible to the video backbone. The anchor-map adapter and layer-specific MLPs inject gated hints into the first visual frame; the gates start at zero. The source explicitly limits this scaffold to an initial condition. Future hand motion and video are still generated by the shared denoiser. Consequently, a plausible initialized pose should not be read as a supplied future trajectory or proof of correct contact dynamics. e_modese_anchore_inference
Figure 3. Joint token processing uses asymmetric access to the generated video. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the grids together with the adjacent attention paragraph: interpreting rows as queries and columns as readable tokens, colored squares implement allowed access. In the upper panel, later visual rows read the visual columns, whereas the purple state row and green action rows extend across the full sequence. The orange first-frame row reads only itself. This is more specific than assuming that every visual row reads every visual token. The lower panel retains the visual/action asymmetry but removes the first-frame and state slots. Its caption does not identify a mode, so it cannot settle the exact inference layout. e_attentione_anchore_architecturee_ablation
What it supports. The supported rule is directional: action and anchor queries can read the visual rollout; video queries do not directly read action or anchor tokens through this self-attention mask. Geometry can still affect video through the separate anchor-map injection. Joint denoising therefore does not imply unrestricted bidirectional token communication.
Where the evidence stops. The lower panel omits f_0 and s_0 despite prose saying both conditioning modes share a token layout. Preserve this unresolved presentation difference; do not silently label the panels as two fully specified modes. No attention-mask ablation is reported.
5.2 Separate shared denoising from unrestricted communication
The shared sequence contains visual, action and anchor tokens, but the attention rule gives them different permissions. Action and anchor queries can inspect all tokens; visual queries remain within the visual stream. The separate anchor-map controller provides the route by which starting geometry affects video. Training then combines video-flow and valid-hand action-flow errors with hand-focused visual weighting. A frozen teacher regularizes video velocity only on examples retaining the input image. Image dropout exposes the same core to both conditioning regimes, while a small trainable Wan subset limits how much of the pretrained backbone changes. Reader interpretation: these choices make visual preservation and action alignment compatible design goals, but Table 2 does not isolate attention, teacher matching or dropout, so their individual causal contributions remain unmeasured. e_architecturee_attentione_anchore_lossese_inferencee_ablation
5.3 Treat a data engine as a hypothesis with two separate tests
A generator that emits paired modalities has cleared only the first requirement for a useful data engine. OakInk2 evaluates how predicted MANO trajectories match recorded futures under best-of-K selection. EgoDex evaluates generated video fidelity and distributional or semantic quality. The text-only comparison adds a VLM judge, but its baseline has no action output, so it cannot establish a comparative action benefit. Figures 4 and 5 show what visual-action correspondence can look like in selected examples; they do not measure the benefit of training another policy on those pairs. Reader interpretation: the next discriminating test should ask whether correctly paired synthetic actions add value beyond extra images, duplicated real data or shuffled pairings. That separates generative plausibility from useful supervision without assuming either implies robot execution success. e_action_protocole_video_protocole_text_resultse_qualitativee_gaps
5.4 Training and inference
During training
Drop the initial image with probability 0.30. Fit video and masked action flow velocities, add hand-region visual weighting, and match a frozen Wan teacher only when the image is retained. Freeze text encoder, VAE, teacher and most Wan blocks; train the new interfaces and a small Wan subset. Train the text-conditioned anchor initializer separately. e_architecturee_lossese_inference
VITRA-1M training uses 64 NVIDIA Hopper GPUs with 96 GB each, one clip per GPU, effective batch 64, bfloat16 and PyTorch FSDP2. AdamW uses constant learning rate 2×10⁻⁵, betas (0.9, 0.999), epsilon 10⁻⁸, weight decay 0.01 and gradient clipping 1.0. No exact Hopper model or training duration is stated. e_training
During inference
TI2VA clamps the encoded first image at timestep zero and generates future video/actions from language and observed geometry. T2VA obtains plausible geometry from the separate initializer, renders its anchor map, and samples through the same denoiser without image replacement. The initializer provides only a starting scaffold, not a planned trajectory. e_architecturee_anchore_inference
The output is a predicted MANO trajectory. The paper specifies no retargeting, robot actuator execution, receding-horizon feedback, or online candidate-selection loop. Teacher and Gaze terms are training supervision, not inference-time controllers. e_lossese_inferencee_gaps
5.5 Implementation flow
- Encode the paired future
A frozen Wan VAE encodes videos. The Wan2.2 TI2V-5B stem patchifies video latents; lightweight encoders embed future MANO actions and the initial hand-camera anchor. Their tokens enter one sequence with separate video and action output heads. e_modese_architecture
- Protect visual attention
Video queries read only visual tokens, while action and anchor queries read the full sequence. Thus actions can use the evolving visual prediction without a direct action-to-video self-attention path. Figure 3 has additional layout ambiguities described below. e_attention
- Ground the first frame
Render the anchor as a color-coded hand skeleton, encode it with the frozen VAE, and use an adapter plus layer-specific MLPs to form hints. Zero-initialized gates inject these only into first-frame visual tokens; future-frame tokens receive no direct anchor-hint injection. e_anchor
6. Experiments & results
Donk adapts a Wan video denoiser to generate videos and bimanual MANO trajectories through one shared backbone. An observed image yields policy-style future prediction; omitting it yields paired synthetic experience after an auxiliary initializer supplies hand-camera geometry. Offline trajectory and video results support this dual interface, while robot execution and the downstream usefulness of generated data remain untested (e_modes, e_inference, e_action_results, e_text_results, e_gaps).
6.1 Read the original evidence
Table 1. Donk leads hand-pose and translation metrics; VITRA retains the rotation advantage. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Locate the highlighted Donk row, then compare within each column rather than across different error types. ADE and FDE measure trajectory displacement and endpoint error; DTW-S and DTW-L are the evaluator’s short-window and open-end variants. These displacement errors are in meters. ROT is wrist-orientation geodesic error in degrees. Each trajectory group separates best-of-5 and best-of-10, drawn from ten sampled futures per example. The single hand-RMSE column evaluates MANO finger pose and has no separate K heading. Follow the downward arrows: lower values are better throughout. e_action_protocole_action_resultse_gaps
What it supports. At best-of-10, Donk reaches ADE 0.049 m and FDE 0.079 m, compared with DreamZero-alike’s 0.057 m and 0.094 m. Its hand RMSE is 0.238 versus 0.262. However, VITRA’s ROT is lower: 14.64° versus Donk’s 14.95°, so the advantage is not universal.
Where the evidence stops. These are offline best-of-K errors, not robot task success or the reliability of one sampled trajectory. Hand-RMSE units and statistical uncertainty are not stated. The source does not fully specify the DreamZero-alike implementation.
Table 3. Strong frame fidelity coexists with mixed semantic, temporal and distributional results. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Begin with the four metric groups rather than the boldface. PSNR, SSIM and LPIPS describe frame fidelity, CLIP-I and CLIP-S address visual identity and text alignment, tLPIPS addresses temporal flicker, and FVD compares video distributions. Their arrows differ, so larger is not always better. Section 4.2 specifies a LOME evaluation using 1,000 EgoDex samples, 17 frames and 832×480 resolution; the caption says all rows share this protocol. Compare Donk first with Wan2.2-TI2V-5B, then examine which other baseline owns each column’s best score. e_video_protocole_video_resultse_qualitativee_ablation
What it supports. Donk improves LPIPS from 0.3061 to 0.2992 and FVD from 81.87 to 75.13 relative to Wan2.2-TI2V-5B, while also leading PSNR, SSIM and CLIP-I. Wan2.1-I2V-14B still has lower FVD, 68.97; VACE has the best CLIP-S and tLPIPS. The result supports competitive visual preservation, with tradeoffs.
Where the evidence stops. These video-only scores do not directly measure contact correctness or whether the generated MANO hands match the video. The rows share an evaluation protocol, but include different backbone sizes; this is not an isolated attention-mask experiment.
Table 4. The text-only interface improves three visual or semantic scores, with more temporal flicker. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Check the Input column first: both entries are labeled text, even though the baseline’s printed name includes I2V. Retain that source label rather than substituting an assumed checkpoint. FVD and tLPIPS are lower-is-better; the VLM judge and CLIP-S are higher-is-better. Section 4.3 says the judge receives generated videos and their instructions for 100 EgoDex samples, scoring instruction alignment from 0 to 5. It does not identify the judge or fully specify the other metrics’ sampling protocol. The table deliberately compares only video outputs, because the baseline produces no hand trajectories. e_text_resultse_qualitativee_gaps
What it supports. Donk lowers FVD from 306.2 to 191.1 and raises the judge score from 1.59 to 2.37; CLIP-S rises from 0.2508 to 0.2572. Meanwhile tLPIPS increases from 0.0147 to 0.0215. This supports a visual/semantic benefit in the reported setup, accompanied by worse measured flicker.
Where the evidence stops. The judge improvement cannot establish action accuracy, physical feasibility, or usefulness as policy-training data. Figure 5 supplies selected paired examples, not a downstream learning test. The baseline’s precise text-only configuration and the judge rubric require clarification.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| OakInk2 offline dexterous trajectory prediction (TI2VA) First-person benchmark; ten sampled futures per example; trajectory numbers below use best-of-10. | Donk: RMSE 0.238; 0.049/0.079/0.046/0.029 m; ROT 14.95°. Hand RMSE; ADE/FDE/DTW-S/DTW-L (m); wrist ROT (degrees), all lower is better | DreamZero-alike: RMSE 0.262; 0.057/0.094/0.054/0.037 m; ROT 19.00°. VITRA has better ROT, 14.64°. Best hand-pose and translation scores among listed methods. Best-of-K evaluation does not establish single-sample policy performance; hand-RMSE units are unspecified. e_action_protocole_action_results |
| State and Gaze conditioning ablation OakInk2 TI2VA; Table 2 uses the same best-of-K selectors as Table 1. | Base → state-only → full: RMSE 0.262 → 0.258 → 0.238; ADE 0.057 → 0.053 → 0.049. Hand RMSE and best-of-10 ADE (m), lower is better | State-only removes Gaze; base removes both. No Gaze-only control is reported. Both additions help in this sequence. The prose calls Gaze’s extra gain smaller, but its RMSE reduction is larger. This is not a factorial isolation of their interaction. e_ablatione_action_protocol |
| EgoDex image-conditioned video fidelity LOME evaluation: 1,000 samples, 17 frames, 832×480; all Table 3 rows share this evaluation protocol. | Donk: PSNR 19.84, SSIM 0.7908, LPIPS 0.2992, CLIP-I 0.9172, FVD 75.13. PSNR/SSIM/CLIP-I higher; LPIPS/FVD lower | Wan2.2-TI2V-5B: 19.50/0.7855/0.3061/0.9119/81.87. Wan2.1-I2V-14B has better FVD, 68.97. Donk leads frame fidelity and CLIP-I, not every video metric; VACE leads CLIP-S and tLPIPS. This table measures video quality rather than action-video physical consistency. e_video_protocole_video_results |
| Text-conditioned video generation (T2VA) Text input; VLM judge evaluates 100 EgoDex samples on a 0–5 scale. Other metric sample counts are not separately established. | Donk: FVD 191.1, judge 2.37, CLIP-S 0.2572, tLPIPS 0.0215. FVD and tLPIPS lower; VLM judge and CLIP-S higher | Baseline labeled Wan2.2-5B-I2V: 306.2, 1.59, 0.2508, 0.0147, respectively. Distributional and judged semantic scores improve, with worse temporal flicker. The baseline has no actions; action quality and downstream training utility are not measured by this comparison. e_text_resultse_gaps |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 2. State conditioning and the hand-focused component improve the reported errors in sequence. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read this table from the bottom upward. The base row lacks both listed components. The middle row turns on State while leaving Gaze absent; the top row retains State and adds Gaze. This ordering separates two incremental comparisons under the same best-of-K evaluation as Table 1. Use the checkmarks before interpreting the metric changes. The labels name the state expert and Gaze module in Section 4.1, but do not expose a separate experiment for every anchor-map or attention component. A missing Gaze-only row prevents a complete two-factor comparison. e_ablatione_action_protocole_gaps
What it supports. Hand RMSE decreases from 0.262 to 0.258 with State, then to 0.238 with Gaze added. Best-of-10 ADE decreases from 0.057 to 0.053 to 0.049 m. The direction is consistent, but contribution sizes depend on the metric: Gaze’s RMSE reduction is larger than State’s.
Where the evidence stops. The prose calls the additional Gaze gain smaller, whereas hand RMSE shows 0.020 versus 0.004. The crop preserves the reported values. Without the missing factorial control or repeated runs, interaction effects and uncertainty remain unknown.
7. Analysis & limitations
7.1 What the evidence leaves open
There is no closed-loop robot trial or downstream policy trained on generated data. Figures 4–5 offer selected qualitative alignment examples; they do not quantify contact validity, dynamics accuracy or out-of-distribution execution success. No statistical uncertainty is reported. e_qualitativee_gaps
Figure 3 omits first-frame/state slots in its unlabeled lower panel despite text claiming a shared token layout. Its upper first-image row is self-only. Preserve these details and use the explicit prose for the video-only versus full-sequence attention rule; the exact mode-specific layout remains unresolved. e_attention
7.2 Questions for discussion
- Would the offline advantage persist under single-sample evaluation rather than best-of-K selection (e_action_protocol)?
- Would adding generated paired data improve a held-out policy beyond equal amounts of real-data reuse (e_text_results, e_gaps)?
8. Reproducibility audit
8.1 Requirements and known gaps
Reproduction requires Wan2.2 TI2V-5B/VAE, VITRA-1M paired video and MANO data, missing-hand masks, camera intrinsics, the renderer and initializer, and the reported optimizer setup. Obtain exact action normalization, loss weights, trained-layer indices, initializer objective, training length and denoising solver/steps before claiming configuration equivalence. e_modese_architecturee_anchore_traininge_gaps
Evaluation needs OakInk2 sample/split identifiers and best-of-K selection details, the EgoDex/LOME configuration, and the unnamed VLM judge and rubric. Table 4 labels an I2V baseline while listing text input; its precise text-only configuration needs clarification. No executed reproduction is claimed. e_action_protocole_video_protocole_text_results
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Complete the State × Gaze ablation under fixed sampling
Reader-proposed check, not performed: reproduce the three reported variants and add the missing Gaze-only condition, keeping training data, Wan initialization, trainable layers, image-drop probability and training budget fixed. Use the same OakInk2 examples and matched random seeds. Report hand RMSE and trajectory metrics at K=1, 5 and 10, plus seed-level uncertainty; first establish a documented single-sample RMSE convention. If Gaze helps only with State, the result supports an interaction rather than an independent additive gain. If improvements disappear at K=1, the best-of-K advantage does not transfer directly to a single predicted trajectory. e_ablatione_action_protocole_architecturee_traininge_gaps
Check 2: Test whether synchronized generated pairs improve a held-out policy
Reader-proposed check, not performed: train the same downstream MANO predictor with a fixed real-data set under three equal-budget conditions: repeated real examples, added Donk T2VA pairs, and the same generated videos with action sequences shuffled across compatible hand-validity patterns. Generate synthetic pairs using training-side prompts and anchor statistics only, excluding evaluation subjects and scenes from selection. Evaluate single-sample trajectory errors on a fixed held-out set and separately audit video-hand alignment. Improvement unique to correctly paired synthetic data would support the claimed supervision value. No improvement, or equal gains from shuffled pairs, would weaken that claim. Robot execution would still need a separate test. e_probleme_modese_anchore_text_resultse_qualitativee_gaps
8.3 Reading coverage
Visual audit: Visually inspected the title, author/affiliation block and v1 stamp on p. 1; motivation on pp. 2–3; architecture, attention, equations, conditioning and training/inference details on pp. 4–6, including hardware and optimizer settings on p. 6; action comparison/ablation and evaluation protocols on p. 7; video metrics and qualitative examples on pp. 8–9. Figures 1–5 and Tables 1–4 were viewed. All six final original crops were separately viewed at their native rendered resolution. Figure 2 arrows were cross-checked against Eqs. (6)–(10); Figure 3 permissions and its unlabeled panel-layout difference are disclosed. Every retained scientific numerical or method claim and both proposed checks are supported by these inspected pages. All four text chunks, including references on pp. 10–13, were read; reference pages were not visually inspected. No separate supplement or code was reviewed and no experiments were run.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9. Appendix coverage: not present.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract (p. 1)
- 1 Introduction (pp. 2–3)
- 2 Related Work (pp. 3–4)
- 3 Method and 3.1 Unified Video-Action Modeling (pp. 4–5)
- 3.2 Joint Video-Action Architecture (pp. 5–6)
- 3.3 Training Objectives and Inference Modes (p. 6)
- 4 Experiments and 4.1 Action Accuracy for TI2VA (pp. 6–7)
- 4.2 Video Quality for TI2VA (pp. 7–8)
- 4.3 Visual and Semantic Quality for T2VA (pp. 8–9)
- 5 Conclusion (p. 9)
- References (pp. 10–13)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- The extraction-only image caveat was addressed by visually inspecting PDF pp. 1–9, all five figures, all four tables, and six final crops. Reference pages 10–13 were read as text.
- No separate supplement was supplied; no code was inspected and no experiment was reproduced.
- Revision scope: the title page verifies arXiv:2606.03868v1, 2 June 2026. Title and all seven authors agree with the catalog after name-order formatting; no other revision was supplied or compared.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e_identityPDF p. 1, title block and arXiv version stamp
The exact title is Unified Video-Action Joint Denoising for Dexterous Action and Data Generation. Authors, in order: Dingrui Wang, YuAn Wang, Jinkun Liu, Yue Zhang, Mattia Piccinini, Yu Sun, Johannes Betz. Affiliations are Technical University of Munich, ByteDance, and Tsinghua University. The artifact is a preprint marked arXiv:2606.03868v1 [cs.CV], 2 Jun 2026; title and author identity agree with the catalog.
Go to primary source ↓e_problemPDF pp. 2–3, Section 1, motivation and contributions
The authors broaden observation-conditioned video-action prediction into a generator that also produces paired interaction videos and MANO trajectories without an input image. They motivate this as a potential source of dexterous training data and claim a first exploration of T2VA for this purpose.
Go to primary source ↓e_modesPDF pp. 4–5, Section 3.1, Eqs. (1)–(4)
The distribution conditions on language c, optional image I_star, and first-frame hand-camera anchor g_0. TI2VA predicts future video/actions from an observed image and anchor; T2VA generates the full video with an initialized anchor. Videos use Wan VAE latents; actions are normalized continuous bimanual MANO trajectories with missing hands masked.
Go to primary source ↓e_architecturePDF p. 4, Figure 2; p. 5, Section 3.2, Tokenization and Conditioning, Eq. (5)
A Wan2.2 TI2V-5B initialized denoiser concatenates video, action, and anchor tokens. Video and action heads supply separate outputs. The encoded first image replaces the first video latent frame at timestep zero; image conditioning is dropped with probability 0.30 during training. Figure 2 depicts encoders, a shared DiT, the anchor controller, teacher/Gaze supervision, and output decoders.
Go to primary source ↓e_attentionPDF p. 5, Section 3.2, Video-Preserving Joint Attention and Figure 3
The prose permits video queries to read video tokens only and action/anchor queries to read the full sequence. Figure 3 has an upper grid with f_0 attending only to itself, other visual rows reading visual columns, and full state/action rows. Its lower grid omits f_0 and s_0. The caption does not label the two regimes; the surrounding text says both modes share a token layout.
Go to primary source ↓e_anchorPDF pp. 5–6, Section 3.2, Anchor-Map Controller, Eqs. (6)–(7)
The anchor g_0=(s_0,K) contains initial MANO state and camera intrinsics. A rendered color-coded skeleton map is encoded with the frozen VAE, patchified, and passed through a shared adapter and layer-specific MLPs. Zero-initialized gates inject hints into first-frame video tokens only. A separately trained text-conditioned initializer supplies plausible first-frame geometry for prompt-only generation.
Go to primary source ↓e_lossesPDF p. 6, Section 3.3, Eqs. (8)–(10)
Training predicts video/action flow velocities. The action loss masks invalid hand dimensions and normalizes by the valid-mask sum, floored at one. Gaze weights visual flow error near rendered hands. A frozen Wan teacher provides a video-velocity prior only for examples retaining image conditioning. Weighted video, gaze, action and prior losses form the objective.
Go to primary source ↓e_inferencePDF p. 6, Section 3.3, final two paragraphs
TI2VA clamps the first video latent from the initial image and generates future video/actions. T2VA initializes hand-camera geometry from language and renders its map before using the shared denoiser. The text encoder, VAE, teacher and most Wan blocks are frozen; action/anchor interfaces, adapter, action head and a small subset of Wan layers are trained. No robot execution or feedback procedure is described.
Go to primary source ↓e_trainingPDF p. 6, Section 4, training setup
Training uses VITRA-1M, 64 NVIDIA Hopper GPUs each with 96 GB VRAM, one clip per GPU and effective batch size 64, PyTorch FSDP2, bfloat16, AdamW, constant learning rate 2e-5, betas (0.9,0.999), epsilon 1e-8, weight decay 0.01, and gradient clipping 1.0. The paragraph does not identify an exact Hopper model, training duration, or software version.
Go to primary source ↓e_action_protocolPDF p. 7, Section 4.1, Offline action accuracy; Table 1 caption
OakInk2 first-person offline evaluation samples ten futures per example and reports best-of-5/best-of-10. ADE, FDE, DTW-S and DTW-L are in meters; ROT is geodesic wrist-orientation error in degrees. DTW-S is short-window and DTW-L open-end. Hand RMSE evaluates MANO finger pose, with no unit or separate K heading specified.
Go to primary source ↓e_action_resultsPDF p. 7, Table 1, Donk-TI2VA, DreamZero-alike and VITRA rows
Donk reports hand RMSE 0.238 and best-of-10 ADE/FDE/DTW-S/DTW-L/ROT of 0.049/0.079/0.046/0.029/14.95. DreamZero-alike reports 0.262 and 0.057/0.094/0.054/0.037/19.00. VITRA has the lowest ROT: 15.15 at K5 and 14.64 at K10, versus Donk 16.05 and 14.95. Donk leads the table on hand RMSE and translation metrics.
Go to primary source ↓e_ablationPDF p. 7, Table 2, all rows; Section 4.1, Conditioning ablation
Base, state-only (wo Gaze), and full models have hand RMSE 0.262, 0.258, 0.238 and K10 ADE 0.057, 0.053, 0.049. Every reported metric improves along this sequence. The text calls the additional Gaze gain smaller, but its hand-RMSE change is 0.020 versus 0.004 from adding state. There is no Gaze-only row or attention/teacher/dropout ablation.
Go to primary source ↓e_video_protocolPDF p. 7, Section 4.2, EgoDex protocol
LOME evaluation on EgoDex uses 1,000 samples, 17 frames and 832 by 480 resolution. PSNR, SSIM and LPIPS measure frame fidelity; CLIP-I and CLIP-S assess visual identity and text-video alignment; tLPIPS assesses flicker and FVD the video distribution.
Go to primary source ↓e_video_resultsPDF p. 8, Table 3, all rows
Donk-TI2VA reports PSNR 19.84, SSIM 0.7908, LPIPS 0.2992, CLIP-I 0.9172, CLIP-S 0.1982, tLPIPS 0.0340, FVD 75.13. Wan2.2-TI2V-5B reports 19.50, 0.7855, 0.3061, 0.9119, 0.2004, 0.0429, 81.87. Wan2.1-I2V-14B has best FVD 68.97; Wan2.1-VACE-14B has best CLIP-S 0.2187 and tLPIPS 0.0197. All rows share the stated evaluation protocol.
Go to primary source ↓e_text_resultsPDF p. 8, Section 4.3; p. 9, Table 4, both rows
Table 4 lists text as input for both Donk-T2VA and the baseline labeled Wan2.2-5B-I2V. Their FVD is 191.1 versus 306.2, VLM judge 2.37 versus 1.59, CLIP-S 0.2572 versus 0.2508, and tLPIPS 0.0215 versus 0.0147. The VLM assessment uses 100 EgoDex samples on a 0–5 scale. The baseline produces no actions; the comparison measures video quality only. The judge model and detailed rubric are unspecified.
Go to primary source ↓e_qualitativePDF p. 8, Figure 4 and caption; p. 9, Figure 5 and caption
Figure 4 juxtaposes generated MANO hands and interaction frames for lemon placement, plate scrubbing and pan lifting, including missing/occluded hands. Figure 5 shows text-conditioned paired rollouts, including cloth placement, trowel use, elephant interaction and a fire scenario. These selected images provide qualitative correspondence examples, without execution success rates or quantitative physical-validity measurements.
Go to primary source ↓e_gapsPDF pp. 5–6, Sections 3.2–4; pp. 7–9, Sections 4.1–5 and Tables 1–4
The method and setup omit loss weights, selected trainable-layer identities, exact action normalization/dimensionality, initializer architecture/objective, training length, and inference solver/step count. Experiments provide offline action, video and qualitative generation evaluations, without confidence intervals, closed-loop robot trials, or downstream policy training on generated pairs. No separate limitations section or appendix appears in the supplied paper; the conclusion on p. 9 is followed by references on pp. 10–13.
Go to primary source ↓8.5 Primary sources
Unified Video-Action Joint Denoising for Dexterous Action and Data Generation ↗
PDF · 6,561 extracted words
Source fingerprint
4d6f2dbdb70481740fd9b504f9d84cbb8c67c5426e43fdbe24050ac8a705e382