DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledge

English reading report: Method, equations, original figures, experiments and reproducibility.

Authors: Wenyao Zhang; Hongsi Liu; Zekun Qi; Yunnan Wang; Xinqiang Yu; Jiazhao Zhang; Runpei Dong; Jiawei He; Fan Lu; He Wang; Zhizheng Zhang; Li Yi; Wenjun Zeng; Xin Jin

Affiliations: SJTU; EIT; THU; Galbot; PKU; UIUC; USTC

Source: arXiv preprint · 2507.04447 ↗ · Catalog record

Reading: 345 / 558 · 6 original figures & tables · ~19 min ·

1. Paper overview

In one sentence: DreamVLA improves manipulation by supervising selective future knowledge inside a shared backbone, but its action-conditioning mechanism and reproduction settings retain unresolved reporting ambiguities. premise architecture attention calvin real-results knowledge routing-ablation

At a glance	What to know
Research problem	Author claim The authors argue that full-image forecasting wastes capacity on unchanged background while omitting explicit geometry and semantic supervision. They seek useful future representations for language-conditioned manipulation without image generation during execution. The human-reasoning analogy motivates the design; the experiments test robot control. premise architecture
Core mechanism	Source description Dream queries organize future knowledge inside a shared backbone. Training-only decoders supervise these latents, while an action query conditions a diffusion transformer. architecture diffusion
A key reported result	CALVIN ABC-D five-instruction chains: 4.44; 78.1% Average completed instructions; five-task success. Train A/B/C; test held-out D. Table 1 reports 1000 rollouts and consecutive completion of up to five instructions. VPP: 4.29 and 75.0%; Seer: 4.28 average length. Reader calculation: +0.15 tasks and +3.1 percentage points versus VPP. This tests simulated execution generalization. Baseline training budgets are not standardized; uncertainty is absent. calvin
Reading caution	Reader analysis Evaluation remains RGB-centric parallel-gripper tabletop manipulation with limited geometry/material diversity. Multi-view ablation, dexterous hands, navigation and humanoid control remain future work. Coarse depth illustrations provide no forecast-error or collision-safety guarantee. scope qualitative

Core contributions

Source description
Dream queries organize future knowledge inside a shared backbone. Training-only decoders supervise these latents, while an action query conditions a diffusion transformer. architecture diffusion
Reader analysis
Motion-focused supervision, separated queries and structured attention improve reported manipulation performance. Ablations support selective target choice more strongly than the claim that every knowledge signal helps. knowledge routing-ablation

Language, state and image encoders feed a shared language-model backbone with dream and action queries. Colored world-prediction branches appear inside a dashed training-only box; an action diffusion transformer appears on the right. — **Figure 2.** Prediction heads teach the shared latent state; diffusion turns action conditioning into controls. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.

2. Motivation

2.1 The problem and the proposed response

Author claim

The authors argue that full-image forecasting wastes capacity on unchanged background while omitting explicit geometry and semantic supervision. They seek useful future representations for language-conditioned manipulation without image generation during execution. The human-reasoning analogy motivates the design; the experiments test robot control. premise architecture

2.2 What this reading follows

A robot placing an object needs to anticipate what will move and where contact will occur. DreamVLA trains this anticipation through dynamic-region reconstruction, depth and semantic-feature prediction, then uses the resulting latent context to generate actions with diffusion. The useful distinction is between teaching a representation to forecast and rendering a future image at deployment: only the former remains necessary here. Read the architecture and attention mask first, then compare simulated chains, physical trials and knowledge ablations. The evidence favors motion-centered supervision and selective semantic cues; it also exposes conflicting mask descriptions and numerical inconsistencies that matter when interpreting the gains. premise architecture attention calvin real-results knowledge routing-ablation

3. Research context

We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.

Catalog dimension	Recorded classification
Major category	Not assigned
Architecture	Not assigned
Prediction paradigm	Not assigned
Quadrant	Not assigned

This table preserves the labels recorded at reading time. The current major category is WAMs. View the current classification.

3.1 Evidence-based assessment

Insufficient evidence to decide

Reader analysis

The recorded taxonomy is unassigned, so no existing quadrant can be endorsed or rejected. Architecture evidence supports a shared backbone with specialized prediction heads and a separate diffusion action decoder. The authors frame control as predictive inverse dynamics through future latents, rather than external video-generator/policy stages or candidate-action-conditioned simulation. A One Model assessment requires this shared information path, not merely joint optimization; mask details remain ambiguous. architecture encoders diffusion attention

This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.

4. Problem formulation

4.1 Inputs and outputs

Inputs	Outputs
Language instruction l Fixed and wrist RGB observations o_t End-effector pose and gripper state s_t	Sequences of seven-dimensional end-effector displacement/gripper actions Future world embedding; explicit dynamic-region, depth and semantic predictions during training

4.2 Equations and their role

\mathbf{w}_{t+n}=\mathcal{M}(l,o_t,s_t\mid\text{<dream>}),\qquad \hat{p}_{t+n}=\mathcal{P}(\mathbf{w}_{t+n})=[\hat{f}_{t+n},\hat{d}_{t+n},\hat{c}_{t+n}]

Eqs. (1)–(2): M maps instruction l, observation o_t and state s_t into future embedding w; n is the forecast offset. Training predictor P decodes dynamic regions f, depth d and semantic features c. Hats denote predictions. architecture

\mathcal{L}_{\mathrm{DiT}}=\mathbb{E}_{\tau,\epsilon}\left\|\epsilon-\epsilon_\theta\left(\sqrt{\bar\alpha_\tau}\,a_{t:t+n-1}+\sqrt{1-\bar\alpha_\tau}\,\epsilon,\tau,c\right)\right\|_2^2

Eq. (9): denoiser epsilon_theta predicts Gaussian noise epsilon from corrupted actions at diffusion step tau, conditioned on action embedding c. Bar-alpha_tau is the noise-schedule coefficient. Here c differs from semantic c above; the schedule choice conflicts with Table 11. diffusion

\mathcal{L}=\lambda_{\mathrm{dyn}}\mathcal{L}_{\mathrm{dyn}}+\lambda_{\mathrm{depth}}\mathcal{L}_{\mathrm{depth}}+\lambda_{\mathrm{sem}}\mathcal{L}_{\mathrm{sem}}+\lambda_{\mathrm{DiT}}\mathcal{L}_{\mathrm{DiT}}

Eq. (10) weights dynamic reconstruction, depth regression, semantic forecasting and action denoising by 0.1, 0.001, 0.1 and 1. train-config

5. Method in detail

5.1 Teach the robot where prediction effort should go

Source description

Begin with target construction rather than action generation. Appendix A.2 samples tracking points every eight pixels and uses CoTracker displacement magnitude to find moving regions. A threshold, illustrated with one pixel per frame, and a one-pixel dilation turn these tracks into a coarse binary mask. The mask weights reconstruction, so the model learns the visual content in motion-relevant areas rather than regressing a dense flow field. Depth and semantic teachers then provide complementary targets: geometry is compared after scale normalization, while semantic forecasting distinguishes corresponding future features from spatially shifted alternatives. These targets are prepared for training, with teacher features precomputed to save device memory and time at the cost of storage. Figure 6 warns that this supervision must be selective: stronger teachers do not guarantee better downstream control. preprocessing targets train-config knowledge

A nine-by-nine attention diagram labels text, state, image, dynamic, depth, semantic and action-query blocks. Cross-type dream cells are white, modality diagonals are shaded, and both action rows are shaded across all columns. — **Figure 4.** The mask separates knowledge types, but its action-query connectivity needs clarification. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.

5.2 Keep prediction supervision separate from deployed decoding

Reader analysis

The action pathway is easier to understand if three things remain distinct: future latent representations, explicit predictions used for supervision, and executed controls. Dream queries produce the first; task-specific decoders produce the second during training; an action query and DiT produce the third. At inference, the decoders can disappear while the learned latent pathway remains. Reader interpretation: forecasting may improve both the backbone’s representation and the information available to its action readout. The present ablations do not cleanly separate those effects. Table 5 compares future forecasting with current reconstruction during training, whereas the latency table measures removing dream-query computation without reporting the corresponding success rate. Neither alone establishes a causal inference-time dependence on accurate future latents. The attention-mask discrepancy makes that distinction especially important for reproduction. architecture diffusion forecast-ablation latency attention

5.3 Read the experiments as complementary tests with different boundaries

Reader analysis

Use CALVIN to ask whether a policy sustains instruction following across a held-out environment, then use the physical table to ask whether its controls work on the reported Franka setup. These protocols support different conclusions. CALVIN reports five-instruction chains and average completed length; real trials allow repeated attempts and use task-specific success criteria. The knowledge ablation then narrows the mechanism claim: dynamic prediction helps most, and retaining every semantic teacher can hurt. Structured attention also improves the displayed chain-success columns, but its average-length inconsistency prevents accepting the entire numerical story without qualification. Reader interpretation: the evidence supports a promising predictive representation for manipulation, not a validated general-purpose physical simulator. Coarse qualitative depth predictions and proposed extensions to navigation or humanoids should be read as illustration and future scope, respectively. calvin real-protocol real-results knowledge routing-ablation qualitative scope

5.4 Training and inference

During training

Source description

Pretraining uses whole-frame prediction on language-free CALVIN and DROID; LIBERO uses LIBERO-90 before suite finetuning. Finetuning introduces comprehensive targets. Eight A800 GPUs, batch 64, AdamW, learning rate 0.001 and 20 epochs are reported; the best validation-success checkpoint is selected. Teacher features are precomputed, incurring storage overhead. training train-config

Reader analysis

Depth uses scale-normalized squared error with fitted global scale; semantic InfoNCE contrasts matching future features against spatially shifted negatives. Dynamic reconstruction and action denoising complete the objective. The depth loss does not establish calibrated metric geometry. targets train-config

During inference

Reader analysis

Encode fresh observations/state, compute dream/action latents and reverse-diffuse Gaussian noise through ten action-head steps. Skip explicit world decoders and external teachers. Outputs are executable controls; no candidate-action world-rollout search is described. How much of each chunk executes before observations refresh is not clearly specified. architecture diffusion latency

5.5 Implementation flow

Encode context
Frozen CLIP text and MAE visual encoders feed a trainable backbone. A Perceiver Resampler compresses image tokens; an MLP embeds robot state. Appendix A.1 identifies GPT-2 Medium as the shared backbone. architecture encoders
Construct future targets
CoTracker masks restrict reconstruction to dynamic regions. Ground-truth depth or Depth-Anything v2 supplies geometry; SAM and optionally DINOv2 supply feature targets. Masks select where reconstruction is supervised, rather than serving as dense optical-flow targets. targets preprocessing
Separate knowledge
Dynamic, depth and semantic dream queries share observation context while cross-type links are masked. Nine queries per modality perform best in Table 9. Exact temporal/action-query connectivity conflicts between Figure 4 and prose. attention query-ablation
Decode actions
An action embedding conditions DiT-B denoising. Appendix A.1 specifies two future steps and no past-action context; real-world Appendix B.4 specifies observation history seven and three future visual/action steps. diffusion real-config

6. Experiments & results

DreamVLA trains a shared backbone to anticipate motion-relevant regions, depth and semantic features, then conditions action diffusion on its latent predictions. Explicit visual decoders disappear at inference. It reports 4.44 completed instructions on CALVIN ABC-D and 76.7% real-robot success under an attempt-limited protocol. Its lesson is selective forecasting, tempered by unresolved mask, configuration and table inconsistencies.

6.1 Read the original evidence

CALVIN table lists fourteen methods, success percentages for completing one through five consecutive tasks, and average completed length. DreamVLA’s highlighted row ends at 78.1 and 4.44; VPP ends at 75.0 and 4.29. — **Table 1.** The clearest headline is sustained execution on held-out CALVIN environment D. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.

Four policies are compared for bottle/doll picking, banana/chili placement, drawer opening/closing and category averages. DreamVLA’s overall mean is 76.7, versus Diffusion Policy 50.8, Octo-Base 45.0 and OpenVLA 35.0. — **Table 3.** Physical trials show a broad gain, with drawer manipulation remaining the weakest category. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.

6.2 Results and evaluation conditions

Task & protocol	Reported result	Comparison & interpretation
CALVIN ABC-D five-instruction chains Train A/B/C; test held-out D. Table 1 reports 1000 rollouts and consecutive completion of up to five instructions.	4.44; 78.1% Average completed instructions; five-task success	VPP: 4.29 and 75.0%; Seer: 4.28 average length. Reader calculation: +0.15 tasks and +3.1 percentage points versus VPP. This tests simulated execution generalization. Baseline training budgets are not standardized; uncertainty is absent. calvin
LIBERO suite evaluation LIBERO-90 pretraining, then Spatial/Object/Goal/Long finetuning.	97.5% / 94.0% / 89.5% / 89.5%; reported mean 92.6% Success rate	SpatialVLA mean 78.1%; CoT-VLA Goal 91.6%. DreamVLA does not lead every suite. CoT-VLA’s printed 69.0 mean conflicts with its four entries; no overall margin against that row is inferred. libero
Franka pick, place and drawer manipulation Two RGB cameras; 100 finetuning demonstrations per task; up to twenty attempts per trial; drawer displacement over 10 cm.	76.7% overall; pick 82.5%, place 80.0%, drawer 67.5% Task-averaged success rate	Diffusion Policy 50.8%, Octo-Base 45.0%, OpenVLA 35.0% overall. Reader calculation: +25.9 percentage points over Diffusion Policy. This is physical execution with retries, not established first-attempt reliability. Independent trial counts and confidence intervals are unclear. real-protocol real-results
Knowledge-target ablation on CALVIN ABC-D Figure 6 single-target and leave-one-out variants.	Dynamic 4.32; depth 2.98; SAM 3.25; DINO 3.13; All 4.27; All-DINO 4.44 Average completed instructions	All-Dynamic falls to 3.74. Dynamic supervision supplies the clearest benefit; removing DINO improves the combination. Gradient interference is an author explanation, not a direct measurement. knowledge
Future forecasting versus current-state reconstruction CALVIN ablation, Table 5.	Prediction 78.1% and 4.44 Five-task success; reported average length	Auxiliary reconstruction 74.2% and printed 4.14. Five-task success favors forecasting by 3.9 percentage points. The auxiliary average fails the cumulative-column consistency check; it is retained as reported, not independently verified. forecast-ablation
Structured attention ablation CALVIN, Table 7; Q4 says only the mask changes.	Structured 78.1% and 4.44 Five-task success; reported average length	Causal 62.7% and printed 3.75. Five-task success improves by 15.4 percentage points. The causal percentages imply approximately 3.93 tasks, not 3.75; exact mask semantics also remain unresolved. routing-ablation attention
Two-camera inference latency RTX 4090; five timing tests averaged in Table 13.	91 ms, described as 11 Hz End-to-end latency	88 ms without dream queries; action head alone 60 ms. Dream queries add 3 ms here. This timing comparison does not establish retained success after query removal or hard real-time control. latency

6.3 Ablations and diagnostic examples

Read component removals and qualitative examples within their stated evaluation conditions.

Bar chart of CALVIN average length shows individual Dynamic, Depth, SAM and DINO targets, then All and leave-one-out variants. Dynamic is 4.32, All is 4.27, and All-DINO is 4.44; a green dashed baseline crosses the chart. — **Figure 6.** More prediction targets do not automatically produce better control. Original paper, p. 10 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.

Two rows compare Causal and Structure attention over five cumulative task-success columns and average length. The structured row is higher throughout, ending at 78.1 and 4.44; causal ends at 62.7 and 3.75. — **Table 7.** Structured attention improves the displayed success columns, but the causal average is inconsistent. Original paper, p. 11 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.

7. Analysis & limitations

7.1 What the evidence leaves open

Reader analysis

Evaluation remains RGB-centric parallel-gripper tabletop manipulation with limited geometry/material diversity. Multi-view ablation, dexterous hands, navigation and humanoid control remain future work. Coarse depth illustrations provide no forecast-error or collision-safety guarantee. scope qualitative

Reader analysis

Figure 4 conflicts with causal/query-isolation prose. Reported averages are inconsistent in Table 1 Pi0, Table 2 CoT-VLA, Table 5 Auxiliary, Table 6 Optical and Table 7 Causal. Figure 6 refers to five models while naming four knowledge types. These ambiguities limit precise conclusions. attention calvin libero forecast-ablation routing-ablation knowledge

7.2 Questions for discussion

Does forecasting still beat current-state supervision after equalizing loss scales, targets and model capacity?
How much control improvement depends on dream latents at inference rather than representation learning during training?

8. Reproducibility audit

8.1 Requirements and known gaps

Reader analysis

Required resources include pretrained encoders, CALVIN/DROID or LIBERO-90, task demonstrations and precomputed teachers. Recreate the eight-pixel tracking grid and dilation, but confirm the threshold: one pixel/frame is only an example. Teacher checkpoints, data versions, validation split and execution/replanning cadence are insufficiently pinned down. encoders preprocessing training real-protocol diffusion

Reader analysis

Resolve conflicting settings: cosine versus constant learning-rate schedule; weight decay 0.0001 versus 0.01; cosine versus linear diffusion noise; convolutional versus ViT decoders; and 64 dream tokens versus nine per each of three modalities. Preserve real-world horizon/bfloat16 settings alongside Table 11’s two-step/float32 configuration. training train-config diffusion encoders architecture real-config

Reader analysis

Reader-proposed checks should match target capacity and seeds for current-versus-future supervision, then test whether inference actions depend on dream latents. Query-count and shared/separated-query ablations motivate controlling capacity instead of equating joint training with a proven causal forecast mechanism. forecast-ablation query-ablation routing-ablation architecture

8.2 Proposed reproduction checks

The following checks are proposals motivated by the paper. They have not been run as part of this reading.

Check 1: Separate future supervision from extra representation learning

Reader-proposed experiment: train matched CALVIN A/B/C variants using current-state targets versus targets at the chosen future offset. Hold the backbone, nine-query groups, action decoder, teacher modalities, preprocessing, loss weights, data and validation selection fixed; document which conflicting source configuration is chosen. Repeat across at least three seeds and evaluate the same held-out-D instruction chains, reporting per-length success and uncertainty as well as a recomputed average. Log each loss’s gradient scale to test the authors’ interference explanation. A reproducible advantage for future targets after these controls would support temporal prediction itself; disappearance of the advantage after balancing loss scales would weaken that interpretation. No such experiment was run for this report. forecast-ablation query-ablation knowledge training train-config preprocessing calvin

Check 2: Test the deployed action pathway’s dependence on dream latents

Reader-proposed experiment: first write down an explicit temporal attention matrix that reconciles, or clearly chooses between, Figure 4 and Q4. For a frozen trained checkpoint, compare normal dream latents with task-mismatched shuffled latents and with blocked dream-to-action access. Keep observations, state, initial environments and diffusion noise paired; include an unchanged-latent pass as a control. Measure action changes and closed-loop CALVIN success, then retrain a matched blocked-access control to distinguish lost predictive information from an out-of-distribution intervention. If actions and success remain unchanged, the claimed inference-time pathway is unsupported for that implementation. A selective, reproducible deterioration would support dependence, but would not by itself prove that the latents encode accurate physical futures. attention architecture diffusion routing-ablation calvin

8.3 Reading coverage

Visual audit: All nine text chunks were read individually, including references and appendices. Original PDF pages 1–11 and 22–30 were rendered and visually inspected: title/byline/version; Figures 1–10; Tables 1–13; method equations, training settings, evaluation protocols and future-work boundaries. All six final crops were separately viewed. Figure 4 was cross-checked against Sections 3.3 and 4.4 Q4, and Figure 2’s training-only box against its caption and equations. Crops preserve the original graphics, with 400-DPI rendering for the narrow attention diagram/table and 200 DPI otherwise. Reference-only pages 12–21 were read as text but not rendered. Separate supplements, external videos, linked code and earlier revisions were not inspected; no experiments were reproduced.

PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 22, 23, 24, 25, 26, 27, 28, 29, 30. Appendix coverage: reviewed.

Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.

Text reading scope & known omissions

Abstract
1 Introduction
2 Related Works (2.1–2.2)
3 Methodology (3.1–3.4)
4 Experiments (4.1–4.4, Q1–Q6)
5 Limitation & Future Works
6 Conclusion
References
A Implementation Details (A.1–A.3)
B Experiments (B.1–B.5)
C Additional Related Works (C.1)
D Additional Discussions and Future Work
E Broader Impacts

Outside the original text pass

Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
Separate supplemental material availability has not been fully verified.
All nine chunks were read individually, covering all 30 PDF pages. The extraction-only visual limitation was resolved by inspecting every original figure and table.
Identity: title and fourteen-author byline match the catalog. The inspected artifact is v3, 26 August 2025 (identity); the catalog submission date is 6 July 2025. Earlier revisions were not supplied, so revision differences cannot be compared.
Separate supplements, linked code, project pages, weights and external references were not inspected. No experiments were reproduced.

The visual audit above records the subsequent illustrated pass.

8.4 Traceable evidence

identityPDF p. 1, title/byline, affiliation line and arXiv marginInspect

The catalog title and all fourteen authors appear in order. The artifact is arXiv:2507.04447v3 [cs.CV], 26 Aug 2025, labeled Preprint. Affiliations are printed as SJTU, EIT, THU, Galbot, PKU, UIUC and USTC.

Go to primary source ↓

premisePDF pp. 1–3, Abstract, Introduction and Figure 1Inspect

The authors motivate dynamic, spatial and semantic forecasting as a more task-relevant intermediate representation than full future-image reconstruction; the human-reasoning analogy is a motivation.

Go to primary source ↓

architecturePDF pp. 4–5, Figure 2/caption, Sections 3.1–3.2, Eqs. (1)–(3)Inspect

Language, observation and proprioception enter one backbone with dream and action queries. Frozen text/visual encoders and trainable state encoder are marked. Prediction decoders are training-only; action embeddings condition diffusion during inference. The dashed training-only box encloses the world-embedding label, but the caption explicitly retains the embedding and skips only prediction heads at inference.

Go to primary source ↓

encodersPDF p. 22, Appendix A.1; p. 23, Table 10; p. 5, Section 3.2 output headsInspect

CLIP ViT-B/32, MAE-pretrained ViT-B, Perceiver Resampler, state MLP and GPT-2 Medium are specified. State includes end-effector position/Euler rotation and gripper state. Appendix decoders are lightweight ViTs; Section 3.2 describes shallow convolutions. The appendix says 64 dream tokens while also specifying nine queries for each of three modalities.

Go to primary source ↓

targetsPDF pp. 5–6, Section 3.3, Figure 3 and Eqs. (4)–(8)Inspect

Dynamic regions guide reconstruction instead of dense optical-flow prediction. Depth uses scale-normalized MSE; semantic InfoNCE uses corresponding future features as positives and spatially shifted features as negatives. The text before Eq. (5) states that a reconstruction distribution is zero, leaving its ELBO simplification unclear.

Go to primary source ↓

preprocessingPDF pp. 23–24, Appendix A.2Inspect

CoTracker tracks keypoints spaced eight pixels apart. Displacement magnitude is thresholded, with one pixel/frame given as an example, followed by one-pixel eight-neighbor dilation. The mask weights predicted and target patch embeddings. Depth-Anything v2 supplies absent depth labels; DINOv2 final-layer and SAM image-encoder features provide semantic targets.

Go to primary source ↓

attentionPDF p. 6, Figure 4 and Section 3.3; p. 10, Section 4.4 Q4Inspect

The diagram separates cross-modality dream blocks but shades each modality diagonal and both entire action-query rows, including action-to-action cells. Section 3.3 describes past-restricted causal query attention, whereas Q4 says all query-to-query edges are removed and action queries never consult siblings. These descriptions do not uniquely specify the pictured mask.

Go to primary source ↓

diffusionPDF p. 7, Section 3.4, Eq. (9); p. 23, Appendix A.1 and Table 11Inspect

DiT-B denoises Gaussian-corrupted action trajectories conditioned on an action embedding. Appendix actions have seven dimensions and two future steps with zero past-action context. Table 11 lists eight training and ten inference diffusion timesteps, float32 and linear noise; Eq. (9) prose specifies cosine noise.

Go to primary source ↓

trainingPDF p. 7, Section 4.1Inspect

Pretraining uses language-free CALVIN and DROID, with LIBERO-90 before LIBERO finetuning. Whole-frame prediction precedes comprehensive forecasting. Setup: eight A800 GPUs, AdamW, learning rate 0.001, batch 64, 20 epochs, best validation-success checkpoint, cosine LR schedule, 5% warm-up and weight decay 0.0001.

Go to primary source ↓

train-configPDF p. 24, Appendix A.3, Eq. (10) and Table 12Inspect

Loss weights are dynamic 0.1, depth 0.001, semantic 0.1, action 1. Teacher features are precomputed, trading storage for training memory/time. Table 12 agrees on eight A800 GPUs, batch 64, learning rate 0.001 and 20 epochs, but lists constant LR and weight decay 0.01, with one warm-up epoch.

Go to primary source ↓

calvinPDF p. 7, Table 1, caption and DreamVLA/VPP/Seer/Pi0 rows; p. 25, Appendix B.1–B.2Inspect

CALVIN trains on A/B/C and tests D, using fixed/wrist RGB images resized to 224×224 and robot state. Table 1 reports 1000 rollouts and five-instruction chains. DreamVLA: 98.2, 94.6, 89.5, 83.4, 78.1 percent, average length 4.44. VPP: 75.0 percent at five tasks, length 4.29; Seer length 4.28. Pi0 lists 93.8, 85.0, 76.7, 68.1 and 59.9 percent with average 3.92. Reader arithmetic: these entries sum to 383.5 percent; dividing by 100 gives 3.835 tasks (about 3.84), inconsistent with the printed 3.92. This is a consistency check, not a replacement measured result.

Go to primary source ↓

liberoPDF p. 8, Table 2 and Section 4.2; p. 7, Section 4.1Inspect

LIBERO has Spatial, Object, Goal and Long suites, each described as ten tasks with fifty demonstrations. DreamVLA reports 97.5, 94.0, 89.5, 89.5 percent and mean 92.6. SpatialVLA mean is 78.1. CoT-VLA reports 81.1, 87.5, 91.6, 87.6 with an inconsistent printed mean of 69.0.

Go to primary source ↓

real-protocolPDF p. 8, Section 4.3 and Figure 5Inspect

A Franka Panda uses two RealSense D415 RGB cameras, fixed and wrist-mounted, and 100 demonstrations per task for finetuning each compared method. Trials allow at most twenty consecutive attempts. Placement requires grasp and placement; drawer success requires displacement over ten centimeters. The independent evaluation-trial count is not unambiguously stated.

Go to primary source ↓

real-resultsPDF p. 9, Table 3, all rows and task headersInspect

DreamVLA achieves bottle/doll pick 85.0/80.0, banana/chili place 80.0/80.0, drawer open/close 70.0/65.0, category means 82.5/80.0/67.5 and overall 76.7 percent. Overall baselines: Diffusion Policy 50.8, Octo-Base 45.0 and OpenVLA 35.0 percent.

Go to primary source ↓

knowledgePDF p. 9, Section 4.4 Q1 and Table 4; p. 10, Figure 6Inspect

Figure 6 shows Dynamic 4.32, Depth 2.98, SAM 3.25, DINO 3.13, All 4.27, All-Dynamic 3.74, All-Depth 4.39, All-SAM 4.40, All-DINO 4.44. The text retains SAM for subsequent ablations. Table 4 lists cumulative dynamic/depth/semantics 4.32/4.40/4.44 above a 3.64 baseline sourced from reference [56]; its caption incorrectly names an optical-flow comparison. Figure 6 says five models although four knowledge types are named.

Go to primary source ↓

forecast-ablationPDF p. 9, Section 4.4 Q2; p. 10, Tables 5–6 and Q3Inspect

Table 5 contrasts current reconstruction with future prediction: printed lengths 4.14/4.44, five-task success 74.2/78.1 percent. Table 6 compares optical flow plus subgoal-image prediction with dynamic-region supervision: lengths 4.23/4.44, five-task success 75.4/78.1. Reader arithmetic gives 4.293 for the Auxiliary cumulative row and 4.339 for Optical, inconsistent with printed lengths.

Go to primary source ↓

routing-ablationPDF p. 10, Section 4.4 Q4–Q5; p. 11, Tables 7–8Inspect

Table 7 reports causal/structured lengths 3.75/4.44 and five-task success 62.7/78.1 percent. Reader arithmetic gives 3.931 for the causal cumulative row, inconsistent with 3.75. Table 8 reports shared/separated lengths 4.17/4.44. Q4 describes changing only the mask.

Go to primary source ↓

query-ablationPDF p. 10, Section 4.4 Q6 and Table 9Inspect

Four, nine and sixteen queries per modality yield lengths 4.32, 4.44 and 4.33. The authors attribute the middle setting to adequate capacity without redundant tokens; no uncertainty estimates are supplied.

Go to primary source ↓

real-configPDF p. 27, Appendix B.4Inspect

Real-world settings specify observation history seven, prediction of the next three visual/action steps, MAE ViT-B initialization and bfloat16 inference. DROID pretraining and 100 task-specific demonstrations collected with SoFar are stated. This horizon differs from Table 11.

Go to primary source ↓

latencyPDF pp. 27–28, Appendix B.5 and Table 13Inspect

Two-image inference on RTX 4090 averages 91 ms over five tests: encoders 12 ms, observation pass 19 ms, ten-step action head 60 ms. Without dream queries, observation pass is 16 ms and total 88 ms. The text states 11 Hz; pruning and asynchronous action generation are proposed optimizations.

Go to primary source ↓

scopePDF p. 11, Section 5; pp. 29–30, Appendix D–EInspect

Limitations include parallel-gripper manipulation, RGB-centric inputs and restricted geometry/material diversity. Single-view ablation, scaling, broader tasks, navigation and humanoid deployment are future directions.

Go to primary source ↓

qualitativePDF pp. 25–29, Appendix B.3 and Figures 7–10Inspect

The PDF supplies CALVIN sequences, dynamic-region predictions, depth predictions and real-robot examples. Depth reconstructions are acknowledged to be coarse. No quantitative forecast-error metric or calibrated depth-error bound accompanies them.

Go to primary source ↓

8.5 Primary sources

DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledge ↗

PDF · 14,882 extracted words

Source fingerprint

916d49a313c0246bfcd60bffba2e685e6432ae912f4b969fdbd3008437c3a44d