Dual-Stream Diffusion for World-Model Augmented Vision-Language-Action Model
1. Paper overview
In one sentence: DUST lets action and future-vision streams exchange information while retaining separate denoising dynamics, improving manipulation with an optional latency cost for extra vision refinement. identitysetuparchitecturetrainingreal-resultsablationszero-vision
| At a glance | What to know |
|---|---|
| Research problem | Source description A policy must choose an action chunk from the current image, instruction and robot state while anticipating its visual consequences. The authors argue that a shared action–vision pathway compromises modality specialization, whereas sequential world-model-to-policy pipelines restrict information exchange. problemsetup |
| Core mechanism | Source description DUST combines modality-specific processing with shared attention, independently noised flow-matching targets, and optional asynchronous vision refinement. architecturetrainingsampling |
| A key reported result | Franka Research 3 seven-task manipulation: 59.9 Reported average task score (%) including partial credit. 60 demonstrations per task; 24 evaluations per task; predefined configurations. Adapted FLARE 49.5; GR00T-N1.5 46.5; π0 40.2. A 10.4-point gain over FLARE, with higher reported scores on every task. This is not a purely binary episode-success average. real-resultsreal-scoring |
| Reading caution | Author claim Appendix A.11 attributes a sponge-grasp failure to self-occlusion and missing wrist-camera visibility. Additional views, stronger proprioception and observation history are proposed remedies, not demonstrated fixes. failure |
Core contributions
- Source description
DUST combines modality-specific processing with shared attention, independently noised flow-matching targets, and optional asynchronous vision refinement. architecturetrainingsampling
- Source description
Action-free BridgeV2 pretraining and joint training with robot/human data improve downstream RoboCasa performance under separate protocols. video-transfermixture-transfer
Figure 2. A frozen semantic conditioner feeds a predictor with two interacting output streams. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Begin at the lower left: the instruction and current observation supply the VLM's semantic context. Follow its rightward arrows into the diffusion network. The lower-right paths represent robot state plus actions and the encoded future observation; MMDiT exchanges information before the paths finish in separate action and vision DiTs. Section 4.1 supplies the stream-routing details that the simplified drawing omits. During training, actions and future embeddings are independently corrupted targets. During ordinary inference, both predicted quantities start as noise. The clean future image pictured here is therefore a training target illustration, not an extra sensor input that the deployed policy can access. setuparchitecturetokenstrainingsampling
What it supports. The world model is part of action generation: its predicted representation participates in the same diffusion process as the action chunk. The vision decoder outputs future embeddings, so this architecture does not require generating photorealistic RGB images before choosing motor commands.
Where the evidence stops. The diagram does not specify the closed-loop action execution schedule or establish physical prediction accuracy. Its generic future-image pathway should be read alongside the embedding definition and training/inference distinction in Sections 3–4.
2. Motivation
2.1 The problem and the proposed response
A policy must choose an action chunk from the current image, instruction and robot state while anticipating its visual consequences. The authors argue that a shared action–vision pathway compromises modality specialization, whereas sequential world-model-to-policy pipelines restrict information exchange. problemsetup
2.2 What this reading follows
When a robot reaches for a cup, its action and the scene that follows are tightly related, yet their representations look very different. DUST trains a policy to predict both: compact motor commands and a semantic embedding of the future image. Its central choice is to share attention while keeping other transformations and noise levels separate. This reading follows the architecture into its training objective, then asks which measured gains come from learning with future-state targets and which require generating those targets at deployment. The evidence includes simulated manipulation, a real Franka arm and ablations that reveal both useful interactions and important limits. identitysetuparchitecturetrainingreal-resultsablationszero-vision
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Not assigned |
| Architecture | Not assigned |
| Prediction paradigm | Not assigned |
| Quadrant | Not assigned |
This table preserves the labels recorded at reading time. The current major category is WAMs. View the current classification.
3.1 Evidence-based assessment
Insufficient evidence to decide
The snapshot is entirely unassigned, so no existing quadrant can be confirmed. Architecture evidence supports a single coupled dual-stream diffusion predictor for joint action/future-embedding generation, with modality-specific branches and a separate frozen VLM conditioner. This is stronger evidence than joint training alone; it is neither only inverse dynamics nor merely an auxiliary training loss. architectureblocksamplingzero-vision
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 1. Predict the consequence in a representation useful for action
At environment time t, DUST receives an instruction, a current image and proprioception. The policy target is the chunk A_t, while its paired world target is the embedding of the observation at t+k, after the chunk. Eagle-2 supplies semantic conditioning; SigLIP-2 supplies the future visual target. This pairing ties policy learning to an observed consequence without requiring pixel reconstruction. The future image in Figure 2 belongs to the supervised training example. At deployment it is unavailable: the action and future-feature streams begin from noise and are refined together. Their coupling is therefore inside the learned sampler, rather than a second environment observation or an external trajectory-search procedure. The paper reports robot execution separately through benchmark task scores. setuparchitecturetokenssamplingreal-results
Figure 5. Separate clocks condition separate transformations around a shared attention layer. Original paper, p. 15 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read downward from the two token rows. The left timestep, labeled tau_a in this drawing, enters the action AdaLN layers; the right tau_o enters the vision AdaLN layers. Attention spans both modalities, while feed-forward transformations remain separate. These arrows agree with Section 4.1 and the Appendix A.3 description: shared attention does not imply a shared noise level or a single feed-forward pathway. Independent time conditioning lets the same block process combinations such as almost-clean visual features with very noisy actions. The actual corruption convention must come from the equations: zero is noise and one is clean data. blockarchitecturetrainingnotationsamplingablations
What it supports. DUST separates two decisions that are often coupled: where modalities exchange information and how far each has progressed through denoising. That separation is necessary for its proposed asynchronous inference, and the architecture/noise ablation tests whether it also matters during training.
Where the evidence stops. Section 4.2's illustrative inverse-dynamics example reverses the endpoints used by the explicit interpolation on p. 5. The crop is faithful; this guide follows the equations and preserves that unresolved prose discrepancy.
5.2 2. Independent corruption creates different conditional prediction problems
Consider two training examples with identical underlying actions and future state but different corruption levels. If the future embedding is nearly clean and the action is noisy, action reconstruction can use information about the desired consequence. Reversing which modality is cleaner encourages prediction of consequences from actions. This is the authors' forward/inverse-dynamics interpretation of decoupled noising. The explicit equations make tau=1 clean, despite the reversed example in Section 4.2. Figure 5 shows why separate timestep inputs matter: each pathway must know its own corruption level even while attending to the other. My interpretation of Table 7 is that architecture and corruption work together. However, the dependency diagnostic demonstrates sensitivity to future-state information; it does not establish causal identification independently of the training distribution. trainingnotationblockablationsdependency
5.3 3. Separate a useful training target from expensive deployment refinement
Training with future states and generating them at deployment are distinct sources of benefit. Table 13 exposes the distinction: disabling vision denoising lowers RoboCasa's average from 50.1% to 47.9%, but the latter still exceeds the reported GR00T and adapted FLARE baselines. Thus the learned policy retains a substantial benefit without explicit vision refinement. For additional test-time computation, Table 6 holds action updates fixed and spends more steps on vision. Table 11's synchronous control instead worsens results when both modalities receive more steps. My reading is that compute allocation matters more than a blanket recommendation to increase solver steps. The GR-1 reversal at 64 steps and measured latency growth make this an operating-point decision, not an unconditional scaling law. zero-visionrobocasascalingsynchronouslatency
5.4 Training and inference
During training
GR00T-based comparisons freeze the pretrained VLM and randomly initialize the diffusion expert. The paper's FLARE variant replaces its original Q-Former target with SigLIP-2 and an MLP alignment module; it is an adapted reimplementation. The π0 baselines start from PaliGemma, without robot-pretrained VLA checkpoints. baselines
BridgeV2 pretraining optimizes only world-modeling loss with randomly initialized action tokens, followed by policy finetuning. Joint training instead mixes RoboCasa, GR-1 and 46,000 EgoDex trajectories, with estimated MANO poses retargeted to Fourier hands. video-transfermixture-transfer
During inference
Initialize both predicted modalities from Gaussian noise. Forward Euler updates vision every global step and actions every q steps, with N_o=qN_A. Main comparisons use N_A=N_o=4; scaling holds N_A=4 and increases N_o. Generated future embeddings guide action denoising; they are not observations received from the environment. samplingscaling
The policy predicts from the current state; a precise chunk-execution/replanning cadence is not specified. Zero vision denoising retains 47.9% RoboCasa success versus 50.1% with four steps, showing that inference-time vision refinement explains only part of the trained policy's benefit. setupzero-visionfailure
5.5 Implementation flow
- Ground the instruction
Frozen Eagle-2 supplies features from its 12th layer. The target future observation is encoded with SigLIP-2; 2×2 pooling reduces 256 image tokens to 64. The stated diffusion input contains one state token, 16 action tokens and 64 future-image tokens. architecturetokens
- Exchange information without sharing every transformation
Twelve MMDiT blocks concatenate modalities only for shared attention, then split them for separate processing. Each stream receives its own timestep through AdaLN. Four modality-specific DiT blocks per stream follow; VLM conditioning alternates self-attention and cross-attention layers. architectureblock
- Learn two velocity fields
Independently sample action and vision corruption levels and Gaussian noises. Predict action-minus-noise and future-embedding-minus-noise; sum their squared-error flow-matching objectives with world-model weight 1.0. Clean/noisy combinations teach conditional prediction in both directions. trainingtokens
6. Experiments & results
DUST augments a frozen vision-language backbone with jointly denoised actions and future visual embeddings. Separate streams exchange information through attention; independent noise levels and unequal sampling budgets accommodate their different dynamics. Controlled comparisons support improved manipulation, but do not establish general causal world understanding.
6.1 Read the original evidence
Table 1. DUST improves reported RoboCasa averages at all three demonstration budgets. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each demonstration budget as its own comparison. PnP covers pick-and-place, OP/CL opening and closing, and Other the remaining tasks; green cells mark the reported averages. Compare the final three rows first because they share the GR00T-based frozen-backbone setup. The dagger identifies results reproduced by the paper's authors, including an adapted FLARE implementation with the same SigLIP-2 target as DUST. The upper rows provide wider context but do not all share that implementation. Dashes mean unreported results, not zero success. Increasing demonstrations also changes training duration, so cross-budget improvements are not a controlled test of data volume alone. robocasabaselinestraining-configsimulation-prose
What it supports. At 100 demonstrations per task, DUST reports 50.1% against 44.6% for adapted FLARE and 41.7% for GR00T-N1.5. The respective gains are 5.5 and 8.4 percentage points. The FLARE gap narrows to 1.7 points at 1,000 demonstrations, despite a higher absolute DUST score.
Where the evidence stops. The p. 7 prose's 18% GR00T improvement does not match these cells. Simulation trial counts and uncertainty are not supplied; the table supports reported point estimates, not statistical significance.
Table 3. The real-robot gain spans seven tasks, with task-specific partial-credit scoring. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the task columns, then the highlighted average. The first four tasks move objects between different source–destination pairs; Cord requires insertion; Eraser and Brush test tool use. All models train with 60 demonstrations per task. Appendix A.5 describes 24 evaluations for each task under predetermined configurations. These entries need careful metric interpretation: partial placements can earn half credit, erasing earns half or full credit at stated removal thresholds, and brushing earns one-sixth per bolt collected. Cord insertion alone has no partial score. The dagger again denotes the paper authors' reimplementation of FLARE, not a reproduction performed for this reading. real-resultsreal-scoringbaselinesfailure
What it supports. DUST has the highest reported score in every task column and averages 59.9%, compared with 49.5% for adapted FLARE. Nevertheless, its insertion score is only 29.2%, and the fourth pick-and-place task reaches 45.8%. The improvement coexists with substantial remaining failures.
Where the evidence stops. The average mixes binary and partial-credit tasks and should not be called an unqualified episode-success rate. This small tabletop suite has no reported uncertainty; Appendix A.11 also documents a grasp failure under occlusion.
Table 6. More vision denoising helps selectively while the action-update budget stays fixed. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. The left column is N_o, the number of vision denoising steps, not the number of action updates. Section 5.3 fixes N_A at four throughout. Compare rows within each benchmark and training budget, using the green average columns as a summary and the task-category columns to see uneven changes. Figure 3 explains the intended schedule: vision updates at every fine step, actions every q steps, where N_o=qN_A. Appendix Table 11 is the relevant control because it increases both step counts together and reports deterioration. This table measures task outcomes; consult Figure 4 separately for hardware-specific latency. scalingsamplingsynchronouslatency
What it supports. GR-1 improves from 42.0% at four vision steps to 47.1% at 32, then falls to 45.0% at 64. RoboCasa with 1,000 demonstrations rises from 66.3% to 69.7% at 64 steps. Figure 4 reports 24.6 ms for four-step DUST and 246.1 ms for 64-step DUST on the stated GPU/compiler setup.
Where the evidence stops. More computation is not uniformly beneficial. Equation (5)'s modulo condition is written using the action clock, leaving indexing ambiguous relative to the global-step description; an implementation must resolve that discrepancy.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| RoboCasa manipulation across demonstration budgets 24 simulated tasks; 100 / 300 / 1,000 demonstrations per task; default four-step sampling. | 50.1 / 58.5 / 66.3 Reported average success (%) | Adapted FLARE: 44.6 / 55.3 / 64.6; GR00T-N1.5: 41.7 / 45.0 / 50.8. Gains over FLARE are 5.5 / 3.2 / 1.7 percentage points. Evaluation trial counts and uncertainty are not reported. robocasabaselinessampling |
| GR-1 humanoid manipulation 24 simulated tasks; 300 / 1,000 demonstrations per task. | 36.0 / 42.0 Reported average success (%) | Adapted FLARE: 33.7 / 36.3. Gains are 2.3 / 5.7 points. Task-category counts conflict between Table 2 and Appendix A.4. gr1 |
| Franka Research 3 seven-task manipulation 60 demonstrations per task; 24 evaluations per task; predefined configurations. | 59.9 Reported average task score (%) including partial credit | Adapted FLARE 49.5; GR00T-N1.5 46.5; π0 40.2. A 10.4-point gain over FLARE, with higher reported scores on every task. This is not a purely binary episode-success average. real-resultsreal-scoring |
| Transfer to RoboCasa Two separate experiments: 300-demo robot/human mixture training; 100-demo finetuning after BridgeV2 video pretraining. | Mixture: 58.5→64.4; video pretraining: 50.1→58.5. Reported average success (%) | FLARE: 55.3→57.6 and 44.6→55.1, respectively. Both routes help; additional data and training compute prevent attributing all gains solely to the architecture. mixture-transfervideo-transfertraining-config |
| Asynchronous inference scaling GR-1, 1,000 demos/task; N_A=4; increase N_o from 4 to 32 and 64. | 42.0→47.1→45.0 Reported average success (%) | RoboCasa 1,000-demo success increases 66.3→69.7 at N_o=64. Improvement is non-monotonic. Figure 4 reports 24.6 / 127.1 / 246.1 ms for DUST at 4 / 32 / 64 steps on RTX 5090 with TorchInductor. scalinglatency |
| Architecture and independent-noise ablation RoboCasa; 100 demonstrations per task. | MMDiT + decoupled noise: 50.1. Reported average success (%) | DiT + decoupled: 42.5; MMDiT + joint: 38.2; DiT + joint: 38.0. Both components matter in this setting; separate streams alone barely improve the joint-noise baseline. ablations |
| Additional simulation benchmarks LIBERO four suites; CALVIN training on ABC and evaluation on D. | 96.2; 3.91 LIBERO average success (%); CALVIN average completed chain length | Adapted FLARE: 96.2; 3.75. LIBERO ties FLARE; CALVIN improves average chain length. Some other baselines are quoted from prior papers under differing protocols. additional-benchmarks |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 7. The strongest ablation result combines separate processing with independently sampled noise. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Panel (a) is the central comparison: move between DiT and MMDiT while holding the noise scheme fixed, then compare joint and decoupled noise within an architecture. MMDiT with joint noise barely changes the baseline average, but MMDiT with decoupled noise reaches the shaded 50.1 row. Panel (b) varies the number of shared MMDiT layers while keeping total path depth at 16, trading shared processing against modality specialization. Panel (c) varies lambda_WM, the relative world-model loss weight. All panels use RoboCasa with 100 demonstrations per task; the shading identifies the chosen configuration rather than an uncertainty interval. ablationsarchitecturetraining
What it supports. With decoupled noise, replacing DiT by MMDiT raises average success from 42.5% to 50.1%. With MMDiT, replacing joint noise by decoupled noise raises it from 38.2% to 50.1%. The two choices interact: separate pathways alone do not reproduce the full gain.
Where the evidence stops. These are point estimates from one benchmark regime. Depth changes also change the allocation of specialized layers; the paper does not establish a parameter-matched, multi-seed optimum or that every weight outside the tested range fails.
7. Analysis & limitations
7.1 What the evidence leaves open
Appendix A.11 attributes a sponge-grasp failure to self-occlusion and missing wrist-camera visibility. Additional views, stronger proprioception and observation history are proposed remedies, not demonstrated fixes. failure
The noisy-future diagnostic increases action MSE from 0.0318 to 0.0865 as noise rises from 0 to 1. It establishes conditioning sensitivity, not identifiable physical causality. Rollout stills do not measure future-prediction fidelity. dependencyrollouts
Source inconsistencies remain unresolved: Section 4.2 reverses noise endpoints; Equation (5)'s action-clock modulo guard is ambiguous relative to the global-step schedule; Table 2 lists 18/6 GR-1 task categories versus 16/8 in A.4. Table 1 gives an 8.4-point RoboCasa gain over GR00T at 100 demos, not the prose's 18%. Tables 8 and 11 label percentages but print fractions. notationsamplinggr1robocasasimulation-proseembeddersynchronous
7.2 Questions for discussion
- Does independent-noise training still help when parameter count and training compute are matched across architectures?
- How much of the vision-refinement gain survives realistic occlusion and latency-constrained replanning?
8. Reproducibility audit
8.1 Requirements and known gaps
For RoboCasa use batch 32 on two A100 GPUs for 60k / 420k / 600k updates at 100 / 300 / 1,000 demos. GR-1 uses batch 960 on eight H200 GPUs for 60k updates. AdamW uses learning rate 1e-4, betas (0.95,0.999), epsilon 1e-8, weight decay 1e-5 with bias/LayerNorm exclusions, cosine decay and 5% warmup. training-config
Required controls include the exact adapted FLARE target, frozen-backbone initialization, data subsets and normalization. Simulation trial counts, seed variance, precise action execution cadence and several preprocessing details remain unspecified; these must be resolved before treating a rerun as equivalent. baselinestraining-configsimulation-settingssetup
A minimal proposed study should repeat the architecture/noise factorial and compare asynchronous, synchronous and zero-vision inference on identical checkpoints, paired initial states and seeds, reporting task success and measured latency. ablationsscalingsynchronouszero-vision
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Match capacity and compute in the architecture–noise factorial
Reader-proposed check, not performed: train the four Table 7(a) variants on identical RoboCasa 100-demo subsets, with frozen Eagle-2 conditioning and the same future target, optimizer and update budget. Match trainable parameter counts where feasible and report any residual difference. Use multiple training seeds and paired evaluation configurations, then report average and per-task success with uncertainty. The discriminating observation is whether MMDiT's advantage remains larger under decoupled than joint noise after these controls. If the interaction disappears, the original table would not isolate the proposed architecture–corruption mechanism from capacity or optimization effects. ablationsarchitecturetokenstraining-config
Check 2: Measure what vision refinement buys at an equal inference budget
Reader-proposed check, not performed: hold a trained checkpoint, initial noise seeds and evaluation configurations fixed. Compare zero vision steps, four-step joint sampling, asynchronous N_A=4 with N_o=16/32/64, and synchronous N_A=N_o at the same global step counts. First resolve the Equation (5) clock indexing and explicitly define zero-step token handling. Measure actual end-to-end policy latency on the same GPU and compiler setup, task success and future-feature error where ground truth is available. A falsifiable result is that asynchronous sampling improves the success–latency tradeoff over the synchronous control; failure to do so would weaken the claimed benefit of modality-specific inference allocation. samplingscalingsynchronouszero-visionlatencysetup
8.3 Reading coverage
Visual audit: All seven supplied text chunks were read individually, including references and Appendix A.1–A.11. Original PDF pages 1–9 and 14–22 were rendered and visually inspected: identity, Figures 1–7, Tables 1–13, implementation/training/evaluation details, rollout stills and failure analysis. Every one of the six final original crops was viewed. Figures 2 and 5 were cross-checked against Sections 4.1–4.2 and A.3; Figure 3 was checked against Section 4.3 and Equation (5). The noise-endpoint prose conflict and ambiguous sampling guard are disclosed. Table captions that carry reproduced-result notes are retained. Reference-only pages 10–13 were read as text; external supplemental videos, code and earlier versions were outside the supplied scope.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 14, 15, 16, 17, 18, 19, 20, 21, 22. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract; Sections 1–2: introduction and related work
- Section 3: preliminaries and world-modeling target
- Sections 4.1–4.3: architecture, joint training, joint sampling
- Sections 5.1–5.4: main results, transfer, scaling and ablations
- Section 6; acknowledgements; impact statement; references
- Appendix A.1–A.5: additional benchmarks, synchronous sampling, implementation, simulation and real-world protocols
- Appendix A.6–A.11: rollout examples, dependency diagnostic, zero-step ablation and failure analysis
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- The extraction limitation was addressed by inspecting original PDF pages and six final crops. All seven supplied text chunks were read individually, including references and Appendix A.1–A.11.
- Version scope: the inspected title page identifies arXiv:2510.27607v3, 28 May 2026, and ICML 2026 / PMLR 306. Title and all five authors match the catalog. The catalog submission date is 31 October 2025; earlier revisions were not supplied or compared, so this report describes v3 only.
- No external project material, repository, code, checkpoints or videos were inspected; no experiments were reproduced. Reference-only pages 10–13 were read as text, not rendered.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
identityPDF p. 1, title, author line, affiliations footnote and arXiv margin
Exact title and authors John Won, Kyungmin Lee, Huiwon Jang, Dongyoung Kim and Jinwoo Shin; v3 dated 28 May 2026; ICML 2026, PMLR 306. Affiliations are printed as Kim Jaechul Graduate School of AI, Korea Advanced Institute of Technology, Seoul, Republic of Korea; RLWRLD, Seoul, Republic of Korea.
Go to primary source ↓problemPDF pp. 1–3, Sections 1–2; Figure 1 and caption on p. 2
The paper contrasts shared joint diffusion and one-way causal diffusion with attention-linked modality streams; it uses VLM-derived representations rather than a pretrained video-generation backbone.
Go to primary source ↓setupPDF p. 3, Section 3, problem setup and world-modeling paragraphs, Eqs. (1)–(2)
Inputs are instruction, current visual observation and proprioception; A_t contains actions a_t through a_{t+k-1}; the world target is the encoded observation at t+k. Euler integration samples actions from noise. The execution/replanning cadence is not specified here.
Go to primary source ↓architecturePDF p. 4, Figure 2/caption and Section 4.1; p. 5, Section 5, VLM backbone and diffusion architecture
Separate modality pathways share attention, receive separate AdaLN timesteps and finish in modality-specific DiTs. Eagle-2 is frozen, conditioning uses layer 12, and the stated stack is 12 MMDiTs followed by four modality-specific blocks per stream.
Go to primary source ↓tokensPDF p. 6, World-modeling target paragraph
SigLIP-2 future embeddings are pooled 2×2 from 256 to 64 tokens per image; the stated module input is one state token, 16 action tokens and 64 future-image tokens. Lambda_WM is 1.0.
Go to primary source ↓trainingPDF pp. 4–5, Section 4.2, decoupled noise scheduling and Eqs. (3)–(4)
Action and vision times and Gaussian noises are independent; corruption linearly interpolates noise at zero to clean data at one. Two velocity squared-error losses are combined with a positive world-model weight.
Go to primary source ↓notationPDF p. 4, Section 4.2 inverse-dynamics example; p. 5, explicit corruption definitions
The example calls tau_o near zero clean and tau_A near T fully noised; the explicit interpolation formulas on p. 5 instead make tau=0 noise and tau=1 clean. T in that example is not reconciled with the later [0,1] convention.
Go to primary source ↓samplingPDF p. 5, Section 4.3, Figure 3/caption and Eq. (5); p. 8, Section 5.3
Initialize both streams from Gaussian noise, use N_o=qN_A, update vision each global step and actions every q steps. Default is four steps each. Eq. (5) prints the guard tau_A N_o mod q=0, leaving the clock indexing ambiguous relative to the stated global schedule.
Go to primary source ↓baselinesPDF p. 6, Baselines; p. 15, Appendix A.3, Baselines
GR00T-based models freeze VLMs and initialize experts randomly. FLARE is reimplemented with SigLIP-2 and a REPA-like MLP instead of its original Q-Former. π0 and π0-FAST initialize only PaliGemma; PAD/VPP use released default configurations.
Go to primary source ↓robocasaPDF p. 6, Table 1, 100/300/1,000-demo Avg. columns; p. 16, Appendix A.4, RoboCasa
DUST averages 50.1/58.5/66.3, FLARE 44.6/55.3/64.6 and GR00T-N1.5 41.7/45.0/50.8. The suite has 8 pick-and-place, 6 open/close and 10 other tasks; trial counts and uncertainty are not supplied.
Go to primary source ↓gr1PDF p. 6, Table 2, caption and Avg. columns; p. 16, Appendix A.4, GR-1
DUST averages 36.0/42.0 against FLARE 33.7/36.3 for 300/1,000 demos. Table 2 describes 18 PnP and 6 articulated tasks, while A.4 describes 16 and 8; both total 24.
Go to primary source ↓real-resultsPDF p. 6, Table 3, all task columns and Avg.; p. 7, Real-world setup
Franka Research 3 has seven tasks and 60 teleoperated demonstrations per task. DUST scores 83.3/79.2/62.5/45.8/29.2/56.3/65.3, average 59.9; FLARE averages 49.5, GR00T 46.5 and π0 40.2.
Go to primary source ↓real-scoringPDF pp. 17–18, Appendix A.5, evaluation and partial-credit paragraphs
Each task has 24 evaluations. PnP can receive half credit; insertion has none. Erasing receives half credit above 50% removed and full credit above 90%; brushing scores one-sixth per bolt collected. Locations and orientations are predetermined.
Go to primary source ↓mixture-transferPDF p. 7, Table 4 and Joint training setup; p. 16, joint-training bullet
At 300 RoboCasa demos/task, adding 300-demo GR-1 and 46k retargeted EgoDex trajectories changes DUST 58.5→64.4 and FLARE 55.3→57.6. MANO poses are retargeted to Fourier hands; mixture training uses batch 512, eight A100s and 60k updates.
Go to primary source ↓video-transferPDF p. 7, Table 5; p. 8, pretraining continuation; p. 16, video-pretraining bullet
BridgeV2 video-only pretraining uses world-model loss with randomly initialized action tokens. RoboCasa finetuning uses 100 demos/task; DUST changes 50.1→58.5 and FLARE 44.6→55.1. Pretraining is 120k updates then finetuning 60k, batch 32 on two A100s.
Go to primary source ↓scalingPDF p. 8, Table 6, Avg. columns and Section 5.3
With four action steps, vision steps 4/16/32/64 give RoboCasa 100-demo averages 50.1/50.4/50.8/51.8, RoboCasa 1,000-demo 66.3/66.8/68.6/69.7 and GR-1 1,000-demo 42.0/45.1/47.1/45.0.
Go to primary source ↓ablationsPDF p. 8, Table 7(a–c); p. 9, Section 5.4 architecture, depth and loss-weight discussion
On RoboCasa 100-demo data, DiT/joint=38.0, DiT/decoupled=42.5, MMDiT/joint=38.2 and MMDiT/decoupled=50.1. Depth 6/10/12/14 gives 47.4/48.3/50.1/49.3 at total path depth 16; weights 0.2/0.5/1/2 give 34.3/48.9/50.1/49.6.
Go to primary source ↓latencyPDF p. 9, Figure 4, caption and Inference time discussion
RTX 5090 with TorchInductor: DUST 4/16/32/64 steps has 24.6/72.6/127.1/246.1 ms latency. These measurements have a specified hardware/compiler context; they do not measure full task completion time.
Go to primary source ↓embedderPDF p. 9, Table 8 and latent-embedder discussion
Table 8 labels success rates as percentages but prints fractional averages 0.501 SigLIP-2, 0.516 DINOv2 and 0.491 Flux.1 VAE, with the SigLIP baseline corresponding to 50.1% elsewhere.
Go to primary source ↓additional-benchmarksPDF p. 14, Tables 9–10 and Appendix A.1
DUST and FLARE both average 96.2 on LIBERO; CALVIN ABC-D chain averages are 3.91 and 3.75. Reported baselines are distinguished from authors' runs; own LIBERO runs use batch 32 for 60k iterations and own CALVIN runs batch 32 for 200k iterations.
Go to primary source ↓synchronousPDF pp. 14–15, Appendix A.2 and Table 11
Increasing both modality step counts worsens performance. Table 11 prints fractions despite a percent caption; RoboCasa 100-demo averages are 0.501/0.425/0.424/0.397 for 4/16/32/64 synchronous steps.
Go to primary source ↓blockPDF p. 15, Figure 5/caption and Appendix A.3 implementation paragraphs
The modified MMDiT shows separate action/vision AdaLN timestep inputs, shared attention and separate feed-forward paths. Vision uses a three-layer SiLU MLP encoder with 2D sinusoidal positions and a two-layer ReLU MLP decoder; actions use linear encoding/decoding and 1D positions.
Go to primary source ↓training-configPDF p. 16, Appendix A.3, batch-size bullets and common training details
RoboCasa uses batch 32, two A100s, 60k/420k/600k updates; GR-1 uses batch 960, eight H200s, 60k updates. Real-world training uses batch 32, two A100s, 60k updates. Optimizer and warmup settings are stated, but seed variability and several preprocessing details are not.
Go to primary source ↓simulation-settingsPDF p. 16, Appendix A.4, RoboCasa and GR-1 paragraphs
RoboCasa uses MimicGen/MuJoCo, Panda and left/right/wrist images with end-effector pose plus gripper actions. GR-1 uses DexMimicGen/MuJoCo, a head view and 29-DoF state/action space. This section does not specify simulation evaluation trial counts or detailed held-out configuration sampling.
Go to primary source ↓simulation-prosePDF p. 7, Simulation results paragraph; p. 6, Table 1
The prose claims an 18% RoboCasa gain over GR00T at 100 demos; Table 1 prints 50.1 versus 41.7, a difference of 8.4 percentage points, or about 20.1% relative.
Go to primary source ↓dependencyPDF p. 21, Appendix A.9, Table 12 and accompanying discussion
Conditioning on increasingly corrupted ground-truth future states yields action MSE 0.0318/0.0436/0.0507/0.0642/0.0816/0.0865 at noise 0/0.05/0.1/0.2/0.5/1.0. The authors interpret this as causal dependency; the diagnostic directly measures action prediction sensitivity.
Go to primary source ↓zero-visionPDF p. 21, Appendix A.10, Table 13 and discussion
RoboCasa 100-demo default four vision steps average 50.1%; zero steps average 47.9%, still above GR00T 41.7 and FLARE 44.6. PnP falls 29.5→19.7 while open/close and other scores increase slightly. Exact zero-step token handling and measured latency are not specified.
Go to primary source ↓failurePDF p. 22, Appendix A.11, failure rollout and all three analysis paragraphs
The authors attribute a failed sponge grasp to arm occlusion of the side ZED camera and no target visibility in the wrist camera. Current-state prediction limits temporal context; extra cameras, proprioceptive priors and history are proposed future directions.
Go to primary source ↓rolloutsPDF pp. 19–20, Appendix A.6–A.8, original rollout sequences and labels
Selected GR-1/RoboCasa execution sequences use 1,000-demo models and real-robot sequences a 60-demo model. Panels are qualitative execution examples, without a ground-truth-versus-predicted future-embedding fidelity metric.
Go to primary source ↓8.5 Primary sources
Dual-Stream Diffusion for World-Model Augmented Vision-Language-Action Model ↗
PDF · 12,678 extracted words
Source fingerprint
52a74b858657a79765359575d93442f1f8b947dca3b8997f9cfc4a1556b3455a