DiT4DiT: Jointly Modeling Video Dynamics and Actions for Generalizable Robot Control
1. Paper overview
In one sentence: Video-generation features can condition effective robot actions after one backbone pass, but the resulting dual-transformer policy trades deployment speed for task performance. e02e03e04e07e08e11e12e13e15e16
| At a glance | What to know |
|---|---|
| Research problem | |
| Core mechanism | |
| A key reported result | RoboCasa-GR1 24-task manipulation: 50.8 Mean task success rate (%). 24,000 demonstrations; 29-dimensional GR1 control, ego-view input; 50 rollouts/task, maximum 720 environment steps. Qwen3DiT 36.2; GR00T-N1.5 41.8; GR00T-N1.6 40.8. Gains are 14.6, 9.0 and 10.0 percentage points respectively. Matched training steps do not equal matched pretraining; several tasks favor baselines. e09e12e18 |
| Reading caution | Source description Single-camera deployment is vulnerable to arm/object occlusion. Wrist cameras and tactile feedback are proposed extensions; large-scale cross-embodiment generality remains future work. e20 |
Core contributions
- Author claim
The separate proxy study reports up to 7× faster convergence and over 10× sample efficiency relative to its semantic baselines. These are curve-based claims for the tested RoboCasa setting, not universal scaling laws. e03
Figure 1. Video generation is the strongest proxy in this controlled downstream-policy study. Original paper, p. 2 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the three proxy objectives: object detection, future-feature alignment and video denoising. In the FLARE-style branch, the snowflake marks the frozen future-frame VLM; the paper removes the original query-diffusion process. Then read the right-hand plots separately: one varies training period, the other training-data quantity. Orange denotes video generation, green FLARE-style and blue grounding. Section 3 explains that each proxy backbone is trained first and frozen while its action expert is fitted. These curves therefore test a staged proxy comparison across 24 RoboCasa tasks; they are not learning curves for the final jointly optimized DiT4DiT policy. e03
What it supports. The figure labels convergence advantages up to 7× and data-efficiency advantages above 10×, with video generation highest over the displayed budgets. These are relative curve comparisons. Their relevance is that dynamics-oriented pretraining produces more useful downstream features in this setting, before the paper introduces joint video/action optimization.
Where the evidence stops. FLARE-style arrows point toward the VLM boxes, whereas Section 3 describes queries attending to VLM features and aligning to future embeddings. Treat the drawing as schematic, not an exact attention/gradient graph. Backbone pretraining and absolute compute are not controlled by parameter count alone.
2. Motivation
2.1 The problem and the proposed response
The authors ask whether learning future visual dynamics supplies a more data-efficient foundation for manipulation than static image–text representations, and how a policy can access those dynamics without waiting for a complete generated video. e02e03
2.2 What this reading follows
DiT4DiT asks a practical question: which part of a video generator is useful for controlling a robot? Its answer is an internal representation, accessed before a full future video is reconstructed. A video transformer supplies features to a separate action transformer, and action supervision helps shape the shared training process. The paper supports this design with simulation benchmarks, physical Unitree G1 trials and feature-extraction ablations. Read the figures in that order: first establish what the proxy comparison tests, then follow the two-model mechanism, and finally distinguish measured task success from the authors’ broader claims about physical understanding. Several timestep and implementation details remain unresolved. e02e03e04e07e08e11e12e13e15e16
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | WAMs |
| Architecture | Dual-system |
| Prediction paradigm | IDM |
| Quadrant | Q4 · Dual-system × IDM |
3.1 Evidence-based assessment
Supports the recorded classification
Dual-system × IDM is supported by distinct DiTs and a directed hidden-feature interface: predicted-dynamics representations condition an inverse-dynamics action decoder. Joint optimization and Equation (8)’s joint distribution do not imply one shared transformer. ‘Visual planning’ needs qualification: inference uses one noise-conditioned feature pass, without decoded video plans or explicit search. e04e05e06e07e08
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Separate the proxy experiment from the final policy
Begin with Figure 1’s experimental question: can a backbone learn useful control features before an action decoder is trained? Section 3 answers using staged training. A proxy objective first adapts each backbone; that backbone is then frozen while the action expert is fitted. Video generation wins this comparison against object grounding and a modified FLARE-style alignment objective. DiT4DiT subsequently changes the optimization regime: both its video and action transformers receive updates in a joint framework. Reader interpretation: the proxy experiment motivates a video foundation, but it does not independently establish the benefit of joint optimization. The two questions need different controls. The final model also remains architecturally dual-system, because the action transformer reads features from a distinct video transformer rather than sharing one token-processing backbone. e03e04e06e07
Figure 2, upper architecture panel. Two distinct transformers connect video dynamics to continuous control. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow the observation through the VAE encoder into the first diffusion transformer, together with language tokens. The lower-left decoder illustrates the optional route from predicted latents to future images. The right transformer receives robot state and action tokens and produces a trajectory. The crucial connection between the two transformers is stated in the caption and Section 4.2: internal video features condition action cross-attention. This cropped architecture panel does not draw that connection explicitly, so the two boxes alone are an incomplete wiring specification. Layer 18 provides the default extracted representation; the future-image decoder is not on the required action-inference path. e04e06e08e13e15e17e21
What it supports. The architecture supports a dual-system reading: a video model and a separate action model retain distinct roles even when trained together. Robot control uses generative hidden features, so the policy can access the video backbone without paying for the illustrated full future-frame reconstruction pathway.
Where the evidence stops. The action arrow is a schematic output, not a measured robot rollout. Figure 10 contains generated videos, while Figures 4–5 document physical tasks and success. Neither the architecture drawing nor qualitative future images establish accurate physical dynamics by themselves.
5.2 Follow both losses while keeping the three times distinct
During training, encode the observation and demonstrated future video with the frozen VAE. Interpolate the future latent with Gaussian noise at the sampled video time, then train the video transformer to predict noise minus the clean latent. Separately, obtain internal video features for action conditioning. The action transformer receives the robot state and a noisy demonstrated action, and predicts its own noise-minus-action velocity. Algorithm 1 combines the masked action error with the weighted video error and updates both transformers; the text encoder remains frozen. Video time, feature time and action time therefore have different jobs. The paper’s fixed-feature-time explanation is understandable as a way to stabilize conditioning, but its training algorithm samples that time instead. Reproduction must expose this choice explicitly rather than silently choosing one description. e05e06e07e17
Figure 3. Three timestep roles separate video learning, feature access and action learning. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the purple line as uniform video training times, the orange curve as the action-time sampling distribution, and the green marker as the feature-extraction operating point described by the caption. Section 4.3 defines action time as one minus a Beta sample. The lower strip illustrates hidden representations moving from noisy h¹ toward clean h⁰. Use Equation (1), not the horizontal arrow alone, for the numerical convention: noise is at time one and clean data at zero. This is a conceptual diagram of separate roles, not a calibrated density plot or a prescription for the exact numerical feature timestep. e05e06e07e08e17
What it supports. Decoupling these roles lets video learning cover its full noise range while action learning uses a different time distribution. In the prose, fixed feature time is intended to stabilize conditioning. That intended design explains why the action decoder need not share the video sampler’s current iteration.
Where the evidence stops. The rightward timestep axis is ambiguous beside a strip running from h¹ to h⁰. More substantively, Algorithm 1 line 11 samples feature time from a grid, conflicting with this figure’s fixed-time caption and Sections 4.3–4.4. No implemented choice or numerical fixed time can be established here.
5.3 Trace the action path through to executed control
At deployment, current observations and the instruction condition a video-transformer forward pass with fresh future noise. The policy extracts internal features once, then keeps them as context while its action transformer denoises a trajectory; Table 4 specifies four action sampling steps and a horizon of 16. The full video sampling loop and image decoder form a separate capability. Reader interpretation: the useful ‘future’ here is a learned generative representation, not a rendered plan that must be followed frame by frame. Figure 8’s one-pass result fits that distinction. Physical task success comes from separate robot trials. Table 3 reports 6 Hz on an A100, while the appendix describes an RTX 4090 deployment workstation; neither account specifies the executed action-chunk length before feedback and replanning. e08e13e15e16e17e19
5.4 Training and inference
During training
Joint fine-tuning updates both DiTs while freezing the text encoder and VAE. Video time is uniform; action time is one minus a Beta sample, with α=1.5 and β=1.0. Algorithm 1 masks and normalizes the action error. e06e07e17
Simulation uses only target-benchmark action data: ‘from scratch’ retains pretrained foundation weights. Physical deployment first uses 241,450 GR1 episodes, then 1,400 G1 demonstrations. Qwen3DiT follows the same physical-data pipeline; GR00T starts from a larger pretrained corpus. e04e10e18
During inference
Encode the observation, sample fresh future noise, and take one video-DiT feature pass at the feature timestep. Hold those features while Euler integration denoises actions. Full video generation runs its own iterative loop and VAE decoder; it is not required for actions. e08
The source reports closed-loop robot execution without specifying the executed chunk length before replanning. Fixed time alone does not ensure deterministic features with freshly sampled input noise, despite Section 4.4’s wording. e08e16e17
5.5 Implementation flow
- Encode observations and instruction
Cosmos-Predict2.5-2B initializes the video backbone. Its causal spatiotemporal VAE compresses current and future frames; Cosmos-Reason1 supplies language embeddings. The video DiT predicts a latent flow velocity conditioned on the current observation and instruction. e04e05
- Expose a control representation
A forward hook extracts video hidden states at the feature timestep. Layer 18 is the default; averaging layers is an evaluated alternative. These features feed cross-attention in a separate action DiT adapted from GR00T-N1. e04e06e15e17
- Decode an action chunk
The action sequence combines proprioceptive embeddings, noisy action tokens and learnable future tokens. AdaLN injects action flow time; cross-attention supplies visual context. A linear output predicts action velocity. Table 4 specifies action horizon 16 and four inference timesteps. e06e17
6. Experiments & results
DiT4DiT conditions a separate action diffusion transformer on internal features of a video diffusion transformer and trains both with flow matching. One video-feature pass supports action inference without decoding future frames. Strong simulation and Unitree G1 results accompany slower deployment and unresolved timestep/configuration details (e04, e06, e07, e08, e11, e12, e13, e16, e17, e19).
6.1 Read the original evidence
Table 2. DiT4DiT leads the average RoboCasa-GR1 success rate, with task-specific exceptions. Original paper, p. 11 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each row across as a success-rate comparison for one task, then use the final row for the mean across all 24 tasks. Bold marks the best reported value, including ties; yellow shading highlights DiT4DiT rather than statistical significance. Evaluation uses 50 rollouts per task and a 720-step maximum episode horizon. The caption defines ‘from scratch’ as no action data beyond this benchmark; pretrained foundation backbones remain. It also states that GR00T models are fine-tuned for the same number of steps as the two custom policies. This equalizes downstream step count but not the provenance or amount of prior training. e04e09e10e12e16e18
What it supports. The mean is 50.8%, versus 36.2% for Qwen3DiT and 41.8%/40.8% for the GR00T variants. The 14.6-point gain over the direct baseline is substantial within this protocol. Yet FromPlacematToTieredshelf scores 18.0% versus Qwen3DiT’s 30.0%, illustrating why the overall gain should not be described as universal task superiority.
Where the evidence stops. No confidence intervals or seed variation accompany the table. Qwen3DiT matches the action architecture and roughly matches parameter count, but changes the foundation backbone. The comparison supports system-level effectiveness without isolating video supervision as the sole cause.
Figure 5. Physical G1 trials show gains beyond generated-video plausibility. Original paper, p. 12 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use the legend before comparing each task group: orange is DiT4DiT, green GR00T-N1.5 and blue Qwen3DiT. These are executed physical-robot success rates on the seven tasks illustrated in Figure 4. Each task has 200 collected demonstrations and 20 evaluation rollouts. DiT4DiT and Qwen3DiT share a simulation-pretraining and physical-fine-tuning pipeline, while GR00T begins with its larger official pretrained model. Compare precision tasks such as Arrange Flower and Stack Cup separately from Drawer Interaction and Box Packing, which require multiple subgoals. The bars provide success outcomes; they do not directly measure the fidelity of the model’s internal predicted dynamics. e09e10e13e14e18e20
What it supports. Arrange Flower reaches 75% versus 25% for GR00T, and Stack Cup reaches 60% versus 25%. DiT4DiT also reports 90% on Drawer Interaction and 50% on Box Packing. These results demonstrate successful physical action execution in the tested setup, complementing the simulation tables and the qualitative video-generation examples.
Where the evidence stops. Twenty trials per task give limited precision, and the chart provides no uncertainty bars. These are the standard physical tasks, not the separate zero-shot shifts in Figure 7. The single ego camera also leaves the policy exposed to occlusions.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| LIBERO four-suite manipulation Franka simulation; target-benchmark action data only; Spatial, Object, Goal and Long suites. | 98.6 average; 98.4 Spatial, 99.6 Object, 98.6 Goal, 97.6 Long. Success rate (%) | Qwen3DiT: 96.6 average, 93.6 Long; CogVLA: 97.4 average, 95.4 Long. Average gain over Qwen3DiT is 2.0 percentage points. Baseline pretraining differs; no uncertainty intervals are given. e09e10e11 |
| RoboCasa-GR1 24-task manipulation 24,000 demonstrations; 29-dimensional GR1 control, ego-view input; 50 rollouts/task, maximum 720 environment steps. | 50.8 Mean task success rate (%) | Qwen3DiT 36.2; GR00T-N1.5 41.8; GR00T-N1.6 40.8. Gains are 14.6, 9.0 and 10.0 percentage points respectively. Matched training steps do not equal matched pretraining; several tasks favor baselines. e09e12e18 |
| Unitree G1 Arrange Flower 16-DoF physical robot, ego camera; simulation pretraining then 200 demonstrations/task; 20 real rollouts/task. | 75 Success rate (%) | GR00T-N1.5 25; Qwen3DiT 0. A 50-point advantage over GR00T on this precision task establishes physical execution evidence, with limited trial count. e09e10e13 |
| Zero-shot object and category shifts Simulation trains on bottle-only drawer/cabinet/microwave tasks, then substitutes can, cup, milk and wine; separate physical flower/vase category change. | Simulation ToDrawerClose 54.5; physical Arrange Flower (Category) 70. Success rate (%) | Qwen3DiT: 32.0 and 0; GR00T-N1.5: 10 for physical flower generalization. Separate protocols support targeted transfer, not unrestricted physical invariance. e14 |
| Feature extraction and joint-training diagnostics Layer/step sweeps on five selected RoboCasa tasks; joint-versus-decoupled feature visualization. | Best extraction at layer 18 and one feature pass; silhouette score 0.17 joint. Success-rate trend; silhouette score | Success degrades across 1–32 feature-denoising steps; decoupled score 0.09. The diagnostic favors early-access features. The visualization does not isolate the cause of improved executed success. e15 |
| Physical deployment efficiency Table 3 specifies one NVIDIA A100; Appendix A.3 separately describes an RTX 4090 workstation. | 6 Hz; 2.2B Control frequency; trainable parameters | Qwen3DiT: 9 Hz, 2.3B; GR00T-N1.5: 13 Hz, 2.7B. Fewer trainable parameters accompany slower control. The hardware accounts are unreconciled; 6 Hz should not be attributed to the RTX 4090. e16e19 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 8. Useful action features appear before full video reconstruction. Original paper, p. 14 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read panels (a) and (b) as separate interventions on feature extraction, evaluated on five selected RoboCasa tasks rather than the full suite. In (a), layer 18 is best; the ‘all’ star represents averaging hidden features across layers. In (b), the x-axis counts video denoising passes used for action conditioning, not action-DiT integration steps. In (c), blue, orange and red identify early, middle and late execution phases, and the two panels compare decoupled with joint training. The displayed silhouette scores summarize separation in the feature analysis; they are not manipulation success rates and should not be compared numerically with panels (a) or (b). e15e17
What it supports. The layer and step sweeps favor a middle-to-deep representation available after one video pass. Extra reconstruction steps reduce reported control success. Joint training also raises the displayed silhouette score from 0.09 to 0.17, consistent with clearer temporal organization, though it does not by itself prove improved causal reasoning.
Where the evidence stops. The explanation that later denoising over-specializes to pixels is a hypothesis. No matched-distribution control or uncertainty bands isolate it here. t-SNE organization is a diagnostic, and a five-task sweep does not establish that layer 18 is optimal for every embodiment or task.
7. Analysis & limitations
7.1 What the evidence leaves open
Single-camera deployment is vulnerable to arm/object occlusion. Wrist cameras and tactile feedback are proposed extensions; large-scale cross-embodiment generality remains future work. e20
Figure 3 and Sections 4.3–4.4 fix τf, but Algorithm 1 line 11 samples a discrete grid. The figure places noisy h¹ left and clean h⁰ right beneath a rightward timestep axis; Equation (1) defines denoising as decreasing τ. The implemented schedule remains unresolved. e05e06e07e08
Benchmarks omit confidence intervals and training-seed variability. Different backbone pretraining confounds a pure objective comparison; ‘physics-aware’ representations are an author interpretation stronger than task success and t-SNE evidence alone. e03e10e11e12e13e15
7.2 Questions for discussion
- Does the one-pass advantage survive training a decoder for each tested feature distribution?
- Does joint training improve executed success beyond a frozen video backbone when data and compute are matched?
8. Reproducibility audit
8.1 Requirements and known gaps
Table 4 lists 32 GPUs, batch 8/device, 100,000 maximum steps, AdamW, video/action learning rates 10⁻⁵/10⁻⁴, and flash_attention_2. It gives 5,000 warmup steps alongside ratio 0.1 without precedence. Training GPU models, numerical λ, fixed τf, software versions and deployment chunk scheduling are unspecified. e07e08e17
Resolve data accounting: Section 5.1 describes 500 LIBERO demonstrations for each of four suites, but Table 5 reports 1,693 used episodes without an exclusion recipe. Table 4’s 32-dimensional action interface needs an explicit embodiment mapping/mask. e09e17e18
Proposed five-task checks should match data, seeds and action sampling: compare fixed versus sampled feature time, and test whether the one-pass advantage survives matching training to inference. Separately compare joint training against stopped action-to-video gradients using executed success. e07e08e15
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Resolve fixed versus sampled feature time
Reader-proposed, not run: on the five Figure 8 tasks, compare training with fixed τf=1 against uniform sampling from {0, 0.25, 0.5, 0.75, 1}; this value and grid are proposed controls, not recovered paper settings. Keep initialization, data, layer 18, optimizer, action-time sampling and four action inference steps identical. Evaluate both policies with one feature pass at τf=1 using paired episode seeds and paired future-noise seeds. Repeat training with multiple seeds and report per-task success with intervals, plus feature/action variance when only future noise changes. A consistent gap would make the prose/algorithm discrepancy consequential; negligible variance and equivalent success would weaken the claim that fixing feature time is essential to stable control. e06e07e08e15e17
Check 2: Isolate action supervision reaching the video backbone
Reader-proposed, not run: compare normal joint training with an otherwise identical variant that stops gradients only between the extracted video features and the action loss. Continue the same video-loss updates in both arms, retaining the same data, initialization, feature-time rule, layer 18, action head and training budget. Use CanToDrawerClose, FromCuttingboardToBasket, FromPlacematToBowl, FromPlateToCardboardbox and FromTrayToPot, with matched rollout seeds. Measure executed task success as the primary outcome and temporal-feature silhouette as a secondary diagnostic. If joint gradients improve silhouette without a reproducible success gain, the representation plot would not establish the claimed control benefit. Improvement in both would support action-shaped video features more directly than a t-SNE comparison alone. e07e09e15e17
8.3 Reading coverage
Visual audit: Visually inspected the title/byline/version on page 1; all Figures 1–10 and Tables 1–5; method equations and Algorithms 1–2; benchmark setup, generalization and efficiency pages; Appendix A configurations, dataset counts, hardware and limitations. All six final original PDF crops were viewed at their native generated dimensions, with axes, legends, headers and all table rows retained. Table 2’s caption-defined meaning of ‘from scratch’ and training-step comparison are explained in its reading guide. Figure 2 omits an explicit inter-transformer feature arrow; its caption and Section 4.2 supply the connection. Figure 3’s timestep ambiguity and Algorithm 1 conflict remain explicit. Page 22 is a qualitative generated-video montage, not an omitted text page or executed-success metric. Reference-only pages 16–18 were read as text; separate supplements, code and linked resources were not inspected.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 19, 20, 21, 22. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract; Section 1 Introduction
- Sections 2.1–2.2 Related Works
- Section 3 Validation of Video Generation as a Scaling Proxy
- Sections 4.1–4.4: flow matching, architecture, joint training, inference; Algorithms 1–2
- Sections 5.1–5.5: setup, benchmark results, generalization, ablations, efficiency
- Section 6 Conclusion; References
- Appendix A.1–A.4: configurations, datasets, robot setup, limitations; Figure 10
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- Pages with little extracted text require visual inspection: 22
- All seven supplied text chunks were read individually, including references and Appendix A. The extraction warnings above are preserved; Figures 1–10, Tables 1–5, equations and algorithms were checked in PDF page images, including low-text page 22.
- Identity: page 1 verifies the catalog title, all seven authors, and arXiv:2603.10448v2 [cs.RO], dated 22 March 2026. The catalog submission date is 11 March 2026; this report describes v2, not the original submission. The manifest identifies the retained previous HTML as v2 too; that HTML and v1 were not separately read or compared.
- Separate supplements remain unverified. Code, checkpoints and project links were not inspected, and no experiments were reproduced. Reference-only pages 16–18 were read as text but not rendered.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title, byline, affiliations and arXiv margin
Title matches DiT4DiT: Jointly Modeling Video Dynamics and Actions for Generalizable Robot Control. Byline: Teli Ma; Jia Zheng; Zifan Wang; Chunli Jiang; Andy Cui; Junwei Liang; Shuo Yang. Affiliations: Mondo Robotics; HKUST(GZ); HKUST. Margin identifies 2603.10448v2, 22 March 2026.
Go to primary source ↓e02PDF pp. 1–2, Abstract and Section 1
Motivates video generative priors for action learning and extracting denoising features instead of reconstructed future frames.
Go to primary source ↓e03PDF p. 2, Figure 1 and caption; pp. 3–4, Section 3
Compares grounding, modified FLARE-style alignment without query diffusion, and video generation. Qwen3-2B and Cosmos-Predict2.5-2B supply the respective backbones. Proxy-pretrained backbones are frozen for downstream action-expert training on 24 RoboCasa tasks. Figure labels show up to 7× convergence and >10× data-efficiency comparisons; absolute compute and statistical uncertainty are not reported.
Go to primary source ↓e04PDF p. 4, Figure 2 and caption; pp. 5–6, Section 4.2
Separate video and action transformers are coupled through hidden features. Video initialization is Cosmos-Predict2.5-2B with a causal VAE and Cosmos-Reason1 language embeddings. The action transformer cross-attends to video features rather than decoded frames.
Go to primary source ↓e05PDF p. 5, Section 4.1, Equations (1)–(8)
Linear interpolation defines clean data at τ=0 and Gaussian noise at τ=1; velocity target is noise minus data, sampled by backward Euler steps. Equations (6)–(8) describe video-dynamics prediction followed by inverse-dynamics action prediction and a joint training distribution.
Go to primary source ↓e06PDF p. 6, Figure 3, Equation (9), Sections 4.2–4.3; p. 7, first paragraph
Hook H extracts video features at τf; action tokens, state embeddings and future tokens use AdaLN and cross-attention. The prose and caption specify fixed feature time and uniform video time, with action time 1−σ, σ drawn from Beta(α,β). The figure’s strip runs from h¹ at left to h⁰ at right.
Go to primary source ↓e07PDF p. 7, Equation (10) and Algorithm 1, especially lines 11–13, 20–23
Both DiTs are optimized while VAE/text encoder stay frozen. Action/video velocity targets are ε−aₜ⁰ and z−zₜ₊₁⁰. The masked action loss is added to λ times video loss. Algorithm line 11 samples τf on a grid, in tension with the fixed-timestep prose; no numerical λ is given.
Go to primary source ↓e08PDF p. 8, Section 4.4 and Algorithm 2
Separate video sampling iterates and decodes future latents. Action conditioning samples fresh future noise and takes one video feature pass, then iteratively denoises actions. The prose calls this deterministic despite newly sampled noise; executed chunk length and replanning schedule are not specified.
Go to primary source ↓e09PDF pp. 9–10, Section 5.1
LIBERO standard data are described as 500 trajectories per suite across ten tasks. RoboCasa uses 24 tasks, 1,000 demonstrations/task, 29-dimensional control, ego-view input, 50 rollouts/task and a 720-step limit. G1 uses 16 DoF, ego input, seven tasks, 200 demonstrations and 20 rollouts/task.
Go to primary source ↓e10PDF p. 10, Policy Setup and Baselines; Table 1 caption
Qwen3DiT pairs a Qwen3-VL 2B backbone with the same action DiT. Physical comparisons match the Qwen3DiT/DiT4DiT pretraining and fine-tuning pipeline; GR00T uses its official pretrained weights. ‘From scratch’ means no action data outside the benchmark, not randomly initialized foundation backbones.
Go to primary source ↓e11PDF p. 10, Table 1, DiT4DiT, Qwen3DiT and CogVLA rows
DiT4DiT success rates are 98.4/99.6/98.6/97.6 for Spatial/Object/Goal/Long and 98.6 average. Qwen3DiT averages 96.6 with 93.6 Long; CogVLA averages 97.4 with 95.4 Long. The table supplies no confidence intervals.
Go to primary source ↓e12PDF p. 11, Table 2, all task rows, Average row and caption
Mean success is 41.8/40.8/36.2/50.8 for GR00T-N1.5/N1.6/Qwen3DiT/DiT4DiT. All train for the same number of steps, with different pretraining. DiT4DiT reaches 74.0 on CanToDrawerClose; it scores 18.0 on FromPlacematToTieredshelf versus Qwen3DiT’s 30.0. The table has no uncertainty intervals.
Go to primary source ↓e13PDF p. 9, Figure 4; pp. 11–12, real-world results; p. 12, Figure 5
Figure 5 shows DiT4DiT/GR00T-N1.5/Qwen3DiT success: Insert Plate 85/80/10, Drawer Interaction 90/80/0, Arrange Flower 75/25/0, Move Spoon 40/15/10, Pick & Place 100/90/10, Box Packing 50/40/0, Stack Cup 60/25/5. Figure 4 illustrates these physical tasks.
Go to primary source ↓e14PDF pp. 12–13, Section 5.3, Figures 6–7
Simulation substitutes can/cup/milk/wine after bottle-only training; ToDrawerClose success is 54.5 versus Qwen3DiT 32.0. Physical Figure 7 includes category/object/number shifts: flower category 70/10/0 and cup-number 50/20/0 for DiT4DiT/GR00T/Qwen3DiT. Box Packing (Object) ties DiT4DiT and GR00T at 30.
Go to primary source ↓e15PDF pp. 13–14, Section 5.4, Figure 8(a–c) and caption
Five tasks: CanToDrawerClose, FromCuttingboardToBasket, FromPlacematToBowl, FromPlateToCardboardbox, FromTrayToPot. Layer 18 peaks; averaging all layers is competitive. Success decreases as feature denoising increases from 1 to 32 steps. Joint/decoupled t-SNE panels label silhouette scores 0.17/0.09. Pixel over-specialization is offered as a hypothesis.
Go to primary source ↓e16PDF pp. 14–15, Section 5.5 and Table 3 with caption
Trainable parameters and frequency: GR00T-N1.5 2.7B/13 Hz, Qwen3DiT 2.3B/9 Hz, DiT4DiT 2.2B/6 Hz. The caption specifies a single A100 for the 6 Hz test. Fixed-task language features can be cached; the speedup is not quantified.
Go to primary source ↓e17PDF p. 19, Appendix A.1, Table 4
Video features: dimension 2048, layer 18, flash_attention_2. Action DiT: 16 layers, action/state dimensions 32/64, horizon 16, four inference timesteps, Beta parameters 1.5/1.0. Training: 32 GPUs, batch 8/device, 100000 steps, warmup 5000 and ratio 0.1, video/action LR 1e-5/1e-4, AdamW. GPU model, library versions, λ, feature-time value and embodiment mapping are not supplied here.
Go to primary source ↓e18PDF p. 20, Appendix A.2 and Table 5
Datasets: Fourier GR1 Unified 1K 24,000 episodes/29 DoF; Pretrain 10K 241,450/29; LIBERO 1,693/7; real robot 1,400/16. The smaller LIBERO count is not reconciled with the standard per-suite counts in Section 5.1. The authors describe their physical pretraining corpus as about 15% of GR00T’s scale.
Go to primary source ↓e19PDF pp. 20–21, Appendix A.3 and Figure 9
G1 has dual 7-DoF arms, ALOHA 2 grippers and a head-mounted RealSense D435i supplying 640×480 RGB. The workstation is described as a single NVIDIA GeForce RTX 4090. Teleoperation uses PICO VR and XRoboToolkit; this hardware account is separate from Table 3’s A100 measurement.
Go to primary source ↓e20PDF p. 21, Appendix A.4
Authors identify single-ego-camera occlusion as a limitation and propose auxiliary wrist cameras/tactile feedback. Larger cross-embodiment pretraining is proposed future work.
Go to primary source ↓e21PDF p. 22, Figure 10 and caption
The low-text final page contains multiple generated future-video sequences. These are qualitative generation examples, with no rollout-success counts or video-quality metrics on this page.
Go to primary source ↓8.5 Primary sources
DiT4DiT: Jointly Modeling Video Dynamics and Actions for Generalizable Robot Control ↗
PDF · 10,757 extracted words
Source fingerprint
64f21b940ff2ee1bccf4e1e7910dc22c207c0cb87ba4e6e16456610c5606b9cd