DUET-DINO: Simultaneous Cross-View World Modeling for Latent Planning in Robot Manipulation
1. Paper overview
In one sentence: Cross-attention lets side- and wrist-view predictors inform one another before latent goal matching, improving orientation-sensitive planning at a substantial CEM inference cost. e-probleme-architecturee-planninge-anglede-attentione-hardwaree-runtimee-latent-change
| At a glance | What to know |
|---|---|
| Research problem | Source description A global camera provides workspace context but may poorly resolve gripper-relative rotation and contact geometry. The authors seek action-conditioned representations that support translation, rotation and gripper changes, using a complementary wrist camera without requiring pixel reconstruction. e-problem |
| Core mechanism | Source description Two view-specific predictors exchange latent information before predicting the consequences of a shared action; their cross-attention blocks and predictor heads are jointly trained. e-architecture |
| A key reported result | Angled reach in simulation: 72.5%; 1.9±1.2 cm; 22.7±32.2° Success rate; FPE; final angular error (FAE). Four orientation-intensive tasks; 40 runs; 5 cm and 12° tolerances within 100 steps. DINOv3 wrist: 62.5%, 8.3±7.2 cm, 47.4±57.5°; independent: 55.0%, 3.2±2.5 cm, 48.8±54.4°. Both terminal errors improve over independent prediction; position alone does not establish orientation success. e-anglede-evaluation |
| Reading caution | Source description CEM costs 15–17 seconds per DUET-DINO planning step in the reported setup, limiting real-time control. Broader hardware tasks and policy-based action proposals are future work. e-runtimee-conclusion |
Core contributions
- Source description
Two view-specific predictors exchange latent information before predicting the consequences of a shared action; their cross-attention blocks and predictor heads are jointly trained. e-architecture
- Source description
Paired latent goal matching supports full 7-DoF CEM planning. Orientation-intensive and sequential manipulation evaluations test whether representation improvements transfer to control. e-planninge-anglede-lift
Figure 2. Exchange visual context before predicting each camera's future. Original paper, p. 3 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read upward from the paired image clips. Snowflakes mark frozen DINOv3 encoders; flames identify learned cross-attention blocks and predictor heads. The crossed purple and pink connections carry the other view's keys and values. Equation (2) clarifies the direction: each target view supplies its own queries. Both heads then receive the same action and end-effector state, but predict different camera latents. The dashed red path depicts the training loss, not an executed action or an extra image decoder. Its caption and Equation (4) specify both teacher-forced one-step targets and an autoregressive rollout loss. e-architecturee-training-losse-attentione-angled
What it supports. The coupling occurs before the separate predictor heads, allowing camera-specific dynamics to use information from the other view. Removing cross-attention reduces simulated angled-reach success from 72.5% to 42.5%. That ablation supports the information-exchange mechanism beyond merely supplying two goal images to a planner.
Where the evidence stops. Only the attention blocks and predictors are trained from scratch; the visual encoder is pretrained and frozen. The diagram establishes coupled future prediction, not a learned action-output head or a unified world-action generator.
2. Motivation
2.1 The problem and the proposed response
A global camera provides workspace context but may poorly resolve gripper-relative rotation and contact geometry. The authors seek action-conditioned representations that support translation, rotation and gripper changes, using a complementary wrist camera without requiring pixel reconstruction. e-problem
2.2 What this reading follows
A wrist camera sees the geometry near a gripper, while an external camera sees where that gripper sits in the workspace. DUET-DINO asks whether these complementary observations should interact inside a predictive model, before a planner decides which action to execute. The paper freezes DINOv3 features, learns two coupled action-conditioned predictors, and searches for actions that move both predicted views toward paired goal images. The strongest controlled evidence comes from simulated reaching and the cross-attention ablation. Hardware results are more modest and use a smaller search budget. This reading follows the information flow, separates the evaluation protocols, and examines the representation diagnostics with their visible labeling ambiguity. e-probleme-architecturee-planninge-anglede-attentione-hardwaree-runtimee-latent-change
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Not assigned |
| Architecture | Not assigned |
| Prediction paradigm | Not assigned |
| Quadrant | Not assigned |
This table preserves the labels recorded at reading time. The current major category is WAMs. View the current classification.
3.1 Evidence-based assessment
Insufficient evidence to decide
The snapshot is unassigned, so there is no assigned quadrant to confirm. Architecture supports an action-conditioned latent world model with two interacting view predictors and external CEM optimization. Predicting both camera futures is not joint future/action generation or inverse dynamics. Joint training alone does not justify a One Model world-action classification. e-architecturee-planninge-evaluation
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Why the second camera belongs inside the predictor
The side and wrist views answer different geometric questions. DUET-DINO preserves their separate patch representations rather than collapsing both into a single predicted image. First, the frozen encoder supplies features for each view. Next, each view queries the other through cross-attention. Finally, its own predictor combines that updated representation with the shared end-effector action and state. The loss supervises the output against features of actual future observations from that same camera. Teacher forcing trains local transitions using observed context, while the short autoregressive term exposes the predictor to its own preceding outputs. This distinction explains what is learned from scratch: cross-view conditioning and dynamics prediction, while the pretrained visual representation remains fixed. Neither training branch learns an inverse-dynamics action decoder. e-architecturee-training-loss
5.2 How predicted features become an executed action
At deployment, paired goal images specify where the robot should end up in both camera views. CEM samples candidate action sequences and asks the learned model to imagine their consequences. Equations (6)–(7) compare terminal features with the goals, add both view errors, and divide by the cost of a zero-action rollout. This denominator is shared across candidates at a planning state; reader analysis therefore suggests it preserves candidate ranking when positive, but can alter the sharpness of the negative-cost softmax weights. It does not independently balance the two cameras' error scales. The weighted elites update the action proposal, and inverse kinematics converts chosen end-effector actions into joint commands. New observations close the loop. Planning horizon H in the equations is distinct from the training rollout horizon K. e-planninge-evaluation
5.3 Separate mechanism evidence from an appealing diagnostic
The most direct mechanism check is the trained variant with cross-attention removed: angled-reach success falls from 72.5% to 42.5%, while ordinary reach falls from 92% to 81%. That pattern is consistent with a larger benefit when orientation matters, although parameter-matched and repeated-seed controls remain missing. Figure 6 then offers a possible representation-level explanation through a selected patch correspondence example. Figure 5 should receive less evidential weight until its color mapping is resolved. Reader analysis also separates the data ablation from the architecture ablation: DROID-only training changes both dataset composition and optimization length. Finally, the hardware table tests practical execution under restricted search and safety handling. It strengthens the relevance of the model but does not demonstrate reliable general manipulation. e-reache-anglede-attentione-patche-latent-changee-data-ablatione-hardware
5.4 Training and inference
During training
Only cross-attention and predictor parameters are trained from scratch. Per-view L1 losses combine teacher-forced predictions across the clip and a two-step autoregressive rollout beginning at its first frame. e-training-losse-architecture
Filtering retains 62,877 DROID and 5,856 RoboArena trajectories. Batches contain synchronized eight-frame clips at 4 FPS. Images are resized from 1280×720 to 224×224 for DINOv3; V-JEPA 2 baselines use 256×256. e-training-setup
DINOv3 uses ViT-H+/16 (840M parameters); V-JEPA 2 uses ViT-G/16 (1B). AdamW runs 120k steps, batch 256, learning rate 4.25×10^-4, weight decay 0.04, with 4.5k-step warmup from 7.5×10^-5 and final 30k-step annealing. Reported DUET-DINO training cost is 5×10^20 FLOPs across four GPUs. e-training-setup
During inference
Simulation uses 800 candidates, 50 weighted elites and two CEM iterations, with run-specific seeds shared across models. Right-side imagery is used for training/evaluation; the official V-JEPA 2-AC* checkpoint instead trained on left-side DROID data for 94.5k steps. e-evaluation
Hardware uses 500 candidates, one CEM iteration and 15 planning steps. Collision or singularity proposals are rejected; three consecutive unsafe steps terminate a run. e-hardware
5.5 Implementation flow
- Encode and exchange
A shared frozen DINOv3 encoder independently produces patch features. Cross-attention uses the target view as queries and the other view as keys and values; separate prediction heads retain view-specific dynamics. e-architecture
- Predict consequences
Each head receives its cross-conditioned features, the same action and end-effector state. The model predicts future latents; it does not directly generate actions or decode future images. e-architecture
- Search and execute
Encode paired goals, roll candidate action sequences through the model, and score their terminal latent distance. Weighted top-k CEM refines a Gaussian proposal. Receding-horizon planning uses updated observations; inverse kinematics supplies joint commands. e-planninge-evaluation
6. Experiments & results
DUET-DINO couples side- and wrist-camera latent predictors through cross-attention, then searches seven-dimensional robot actions against paired goal images. Its clearest evidence is improved simulated spatial and orientation planning; hardware success remains limited under a smaller planning budget and safety terminations (e-architecture, e-planning, e-angled, e-hardware).
6.1 Read the original evidence
Table I. Wrist imagery helps spatial reach; coupling both views improves it further. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the Side and Wrist checkmarks, then compare rows within the DINOv3 block so that the encoder family stays fixed. Wrist-only planning reaches 79% success, independent dual-view planning reaches 78%, and DUET-DINO reaches 92%. The rightmost column measures final position error in centimeters; smaller is better. The experiment contains 100 runs across ten objects in the simulated breakfast-table scene. Success requires reaching within 5 cm during a 100-step allowance. Final error is measured at the last planning step regardless of whether the run previously met the success threshold. e-reache-evaluation
What it supports. DUET-DINO improves success by 14 percentage points over independent DINOv3 views and reduces reported FPE from 9.8±13.8 to 5.4±7.4 cm. The near-identical wrist-only and independent success rates show that adding a second uncoupled predictor does not automatically improve this task.
Where the evidence stops. The starred official V-JEPA 2 checkpoint differs in training data, duration and camera side, so its comparison is not a clean architecture ablation. The paper does not define the ± statistic or supply success-rate confidence intervals.
Table II. Being near the target does not guarantee the right orientation. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the success and angular-error columns together. These 40 simulated runs span four tasks with clockwise or counterclockwise orientation changes. Success requires both 5 cm positional accuracy and 12° angular accuracy within 100 steps. The official V-JEPA 2-AC* row illustrates the distinction: 3.6 cm reported FPE accompanies only 2.5% success and 72.6° reported FAE. Within DINOv3, independent views improve position error over wrist-only prediction but slightly worsen angular error. DUET-DINO improves both. As in the reach table, the final-error columns aggregate outcomes without conditioning on success. e-anglede-evaluatione-lift
What it supports. DUET-DINO reaches 72.5% success versus 55.0% for independent DINOv3 views, with FAE falling from 48.8±54.4° to 22.7±32.2°. This is the clearest table-level evidence that cross-view prediction helps orientation-sensitive planning, rather than only getting the end-effector into the target's neighborhood.
Where the evidence stops. This table uses stricter reaching tolerances than the sequential lift experiment, which allows 15 cm and 20°. Its 72.5% cannot be pooled with Table V's 75.0% angled-reach stage rate. Uncertainty of the success-rate difference is unreported.
Table IV. The hardware advantage remains, but absolute success is limited. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. This table covers 30 real-robot runs across three challenging angled-reach poses near table corners. Compare all three metrics: DUET-DINO has the highest success rate and lowest angular error, while independent dual-view prediction has the smallest reported position error. The accompanying setup uses a Franka Research 3 with a Robotiq gripper and ZED side/wrist cameras. The continuation on page 7 is essential: hardware planning uses only 500 candidate actions, one CEM iteration and 15 planning steps. Proposals involving collisions or singularities are rejected, and three consecutive unsafe planning steps terminate a run. e-hardwaree-evaluatione-angled
What it supports. Success rises from 16.7% for independent views to 26.7% for DUET-DINO, while FAE falls from 40.0±42.0° to 28.0±17.7°. However, FPE is 14.7±17.1 cm versus 14.3±17.2 cm for independent views. The evidence supports a qualified hardware benefit, not uniformly superior metrics.
Where the evidence stops. The simulation and hardware settings change tasks, search budget and safety handling. Their success-rate gap is not a controlled estimate of transfer loss. The small hardware sample and missing confidence intervals also limit precision.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Spatial reach in simulation RoboLab breakfast_table/HomeOfficeBackground; 100 runs over ten objects; 5 cm tolerance within 100 steps. | 92%; 5.4±7.4 cm Success rate; final position error (FPE) | DINOv3 wrist: 79%, 13.1±20.3 cm; independent dual-view: 78%, 9.8±13.8 cm. A 14-percentage-point advantage over independent views. FPE includes successful and failed runs. e-reache-evaluation |
| Angled reach in simulation Four orientation-intensive tasks; 40 runs; 5 cm and 12° tolerances within 100 steps. | 72.5%; 1.9±1.2 cm; 22.7±32.2° Success rate; FPE; final angular error (FAE) | DINOv3 wrist: 62.5%, 8.3±7.2 cm, 47.4±57.5°; independent: 55.0%, 3.2±2.5 cm, 48.8±54.4°. Both terminal errors improve over independent prediction; position alone does not establish orientation success. e-anglede-evaluation |
| Angled reach under visual distribution shifts 100 simulated runs: five backgrounds × five object-count conditions × four angled-reach tasks. | 63%; 2.6±2.1 cm; 30.9±36.5° Success rate; FPE; FAE | DINOv3 wrist: 25%; independent: 45%. Supports robustness to these background/distractor variations, not unrestricted visual generalization. e-shiftse-angled |
| Angled reach on hardware 30 runs over three corner poses; Franka Research 3, Robotiq 2F-85, ZED 2i side camera and ZED Mini wrist camera; reduced planning budget. | 26.7%; 14.7±17.1 cm; 28.0±17.7° Success rate; FPE; FAE | DINOv3 wrist: 3.3%; independent: 16.7%, 14.3±17.2 cm, 40.0±42.0°. Highest success and lowest FAE, but independent views have slightly lower FPE. This differs from the simulation protocol. e-hardware |
| Sequential angled grasp and lift-to-home 40 simulated runs over four tasks; 60 reach steps (15 cm/20°), ten grasp steps, 60 lift steps (within 20 cm of home while holding the object). | 75.0% angled-reach; 60.0% lift Stage-wise angled-reach and lift success rates | Independent: 60.0%/57.5%; wrist-only: 57.5%/20.0%. Lift improves only 2.5 percentage points over independent views; reaching tolerances are looser than Table II. e-lift |
| Cross-attention removal A trained DUET-DINO ablation under the Table I and II task protocols. | Without attention: 81% reach; 42.5% angled reach Reach and angled-reach success rates | Full model: 92% and 72.5%, respectively. The larger orientation drop supports cross-view conditioning; parameter-count and training-seed controls are not supplied. e-attentione-reache-angled |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 5. A diagnostic of action sensitivity, with an unresolved color mapping. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Rows separate the static side camera from the wrist camera; columns show encoder changes, DINOv3 predictor changes and V-JEPA 2 predictor changes. The horizontal axis is episode transition step and the vertical axis is latent L1 change, with different scales across panels. Dashed lines denote predictions. Inspect the global legend carefully: solid blue is labeled V-JEPA 2 and solid red DINOv3, but the predictor columns use blue references under DINOv3 and red references under V-JEPA 2. Section IV-G describes these as comparisons to encoded dynamics; this mismatch prevents an unambiguous assignment of the intended within-encoder references. e-latent-change
What it supports. The authors interpret the diagnostic as DINOv3 preserving temporal changes with overestimated magnitude, while V-JEPA 2 underestimates wrist-view changes. Visible dashed/solid separations motivate investigating prediction sensitivity, but the labeling ambiguity means this crop alone cannot securely establish that encoder-specific explanation or precise magnitude comparisons.
Where the evidence stops. The text's side-view encoder ranking also conflicts with a literal global-legend reading. The original is preserved unchanged, and no label correction is assumed. This is latent-space evidence, not an executed-action success measurement or a cross-attention ablation.
Figure 6. Inspect whether the predicted latent keeps the target object's spatial structure. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow each row from the two input frames to encoded features at time t, encoded features at t+1, and the predicted next features. The magenta square identifies the reference patch on the target object. Every heatmap compares that patch's feature with other patch features using L1 distance; lower values indicate greater similarity. DINOv3 uses a 14×14 grid and V-JEPA 2 a 16×16 grid, so the spatial resolutions differ. The relevant qualitative comparison is whether object-aligned low-distance regions in the next encoded frame remain organized in the predicted frame, not whether either row reconstructs the photograph. e-patch
What it supports. In this example, the DINOv3 maps retain a more coherent object-shaped region, while V-JEPA 2 shows noisier correspondences and a comparatively diffuse predicted map. This supports the authors' qualitative account of feature preservation and motivates measuring correspondence quality over many examples, alongside downstream planning performance.
Where the evidence stops. One selected reference patch is not a correspondence benchmark. The encoders also differ in feature dimension and scale; raw colors are not a calibrated measure of semantic quality across models, nor proof of accurate contact dynamics.
7. Analysis & limitations
7.1 What the evidence leaves open
CEM costs 15–17 seconds per DUET-DINO planning step in the reported setup, limiting real-time control. Broader hardware tasks and policy-based action proposals are future work. e-runtimee-conclusion
Error tables use ± without defining its statistic; success-rate uncertainty is absent. Hardware safety terminations and different budgets prevent attributing the simulation-to-hardware gap solely to generalization. e-reache-anglede-hardware
DROID-only training yields 78% reach and unchanged 72.5% angled reach, but also reduces training to 94.5k steps, confounding data and compute. Under camera perturbations DUET-DINO ties independent DINOv3 views at 66.7%; that test does not isolate cross-attention benefits. e-data-ablatione-camera
Figure 5 labels solid blue as V-JEPA 2 and red as DINOv3, yet its DINOv3 predictor panels use blue references and V-JEPA 2 panels use red. Section IV-G also describes larger V-JEPA 2 side-view encoder changes. The mapping remains unresolved; no precise encoder-specific magnitude is inferred. e-latent-change
7.2 Questions for discussion
- Would cross-view gains survive equal-parameter, repeated-seed ablations?
- How much hardware failure comes from latency, unsafe proposals or prediction error?
8. Reproducibility audit
8.1 Requirements and known gaps
Reproduction needs paired camera/action data, frozen encoders, predictor/attention definitions, RoboLab scenes and paired goals. The abstract promises future code/checkpoint release; it does not establish current availability. e-identitye-architecturee-training-setupe-evaluation
The supplied paper leaves filtering rules, train/validation allocation, attention depth, numerical planning horizon, action bounds, proposal initialization, softmax temperature and GPU model unspecified. State propagation during imagined rollouts is not detailed; these choices must be recovered for faithful reproduction. e-architecturee-training-setupe-planning
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Test whether useful cross-view information causes the gain
Reader-proposed check, not performed: train the full model, a no-cross-attention model, and a parameter-matched control whose conditioning receives only same-view tokens. Keep DINOv3 frozen, data, optimization steps, goal images and CEM seeds fixed, and repeat training seeds. Evaluate the original four angled-reach tasks, reporting success, FPE, FAE and held-out per-view prediction loss. Add a diagnostic inference pass with mismatched side/wrist observations. The mechanism predicts a repeatable full-model advantage that weakens when complementary information is removed or mismatched. If matched controls erase the advantage, the existing result may reflect capacity or optimization rather than cross-view information. e-architecturee-training-losse-training-setupe-evaluatione-anglede-attention
Check 2: Separate RoboArena data from additional optimization
Reader-proposed check, not performed: run a 2×2 comparison of DROID-only versus DROID-plus-RoboArena data, each trained for 94.5k and 120k steps. Keep encoder, model, batch size, sampling rule and evaluation seeds fixed; explicitly document a common learning-rate schedule so schedule changes do not become another hidden factor. Evaluate the original reach and angled-reach tasks and report variability across training seeds. Extra-data benefits should persist within a fixed step budget. If the reach gain appears only when steps increase, the paper's 78%-to-92% comparison cannot by itself support a data-diversity explanation; unchanged orientation performance is also a falsifiable outcome. e-training-setupe-data-ablatione-evaluatione-reache-angled
8.3 Reading coverage
Visual audit: All nine PDF pages were rendered at 200 DPI and actually inspected, including title/byline/affiliations, Eqs. (1)–(7), training and compute details, simulation and hardware protocols, all six figures, Tables I–V, prose ablations, conclusion and references. Six final original crops were separately inspected; narrow tables and diagnostics use fresh 350–400 DPI PDF renders. Pages 4–8 provide supporting method, numerical and reproduction details even when not cropped. Figure 5's unresolved legend/text mapping is disclosed in the report and visual explanation. No appendix is present; separate supplements and project media were not supplied or inspected.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9. Appendix coverage: not present.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- PDF p. 1: title, authors, affiliations, abstract, I. Introduction
- PDF p. 2: contributions and II. Related Work
- PDF pp. 2–4: III. Methodology, A–C, Eqs. (1)–(7)
- PDF pp. 4–8: IV. Experiments, A–G, including training, simulation, hardware, ablations and embeddings analysis
- PDF p. 8: V. Conclusion and VI. Acknowledgements
- PDF pp. 8–9: References
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- Identity: the title page states arXiv:2609.10506v1 [cs.RO], 9 Sep 2026. Title and all eight authors match the catalog. No revision or edition difference was observed; no other version was supplied.
- The acquisition warning that text extraction does not reconstruct figure images was resolved by inspecting all nine PDF pages and every final crop.
- Separate supplemental material availability has not been fully verified; no separate supplement was supplied.
- Code, checkpoints, project media and cited works were not inspected. No experiments were reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e-identityPDF p. 1, title/byline, affiliation footnotes, arXiv margin stamp and Abstract
Title and eight-author byline match the catalog; stamp specifies arXiv:2609.10506v1, cs.RO, 9 Sep 2026. Affiliations name UTN, TUM, KIT, NVIDIA and Robotics Institute Germany. Abstract promises code and checkpoints will be open-sourced.
Go to primary source ↓e-problemPDF p. 1, I. Introduction, right column; PDF p. 2, contributions 1–4
The paper motivates paired global and gripper-centric views for fine translation, rotation and gripper-state prediction and full 7-DoF planning.
Go to primary source ↓e-architecturePDF p. 3, Figure 2 and caption; III-B, Eqs. (1)–(3)
Frozen DINOv3 yields patch features. Each view queries the other as keys/values, then a view-specific predictor conditions on action and end-effector state. Cross-attention and predictor heads are trainable.
Go to primary source ↓e-training-lossPDF p. 3, III-B Training and Eq. (4)
Joint dual-view L1 loss combines teacher forcing over the clip and autoregression from its first frame with K=2; encoder parameters remain frozen.
Go to primary source ↓e-planningPDF p. 3, III-C; PDF p. 4, Figure 3, Eqs. (5)–(7) and CEM update paragraph
Receding-horizon optimization samples Gaussian action sequences, predicts terminal features, sums per-view L1 goal costs and divides by the zero-action dual-view rollout cost. Weighted top-k CEM uses softmax over negative costs. Figure 3 uses terminal index T while Eq. (5) defines horizon H.
Go to primary source ↓e-training-setupPDF p. 4, IV-A Implementation and Training Details
62,877 DROID and 5,856 RoboArena trajectories; eight-frame clips at 4 FPS; encoder input resolutions/backbones; AdamW batch and learning-rate schedule, 120k steps, and 5×10^20 FLOPs over four GPUs for DUET-DINO.
Go to primary source ↓e-evaluationPDF p. 5, IV-B Experimental Setup and IV-C Reach Task
Single, independent dual and jointly trained DUET variants; official V-JEPA 2-AC* left-view 94.5k-step checkpoint. Simulation uses 800 samples, 50 weighted elites, two CEM iterations and matched run-specific seeds. Other training/evaluation uses right-side images, with inverse kinematics for joint commands.
Go to primary source ↓e-reachPDF p. 5, Table I, DINOv3 and V-JEPA 2 rows; IV-C success/FPE definitions
100 runs over ten objects. DUET-DINO: 92%, 5.4±7.4 cm; DINOv3 wrist: 79%, 13.1±20.3; independent: 78%, 9.8±13.8; DINOv3 side: 18%; official V-JEPA 2-AC*: 55%. Reach requires 5 cm within 100 steps; FPE is final-step error regardless of success. The ± statistic is not defined.
Go to primary source ↓e-angledPDF p. 6, Table II and IV-D Fixed Background/success definitions
40 runs/four tasks, requiring 5 cm and 12° within 100 steps. DUET-DINO: 72.5%, 1.9±1.2 cm, 22.7±32.2°; DINOv3 wrist: 62.5%, 8.3±7.2, 47.4±57.5; independent: 55.0%, 3.2±2.5, 48.8±54.4. Official checkpoint: 2.5%, 3.6±1.5 cm, 72.6±32.2°. FAE is final-step error regardless of success.
Go to primary source ↓e-shiftsPDF p. 6, Table III and IV-D Diverse Backgrounds and Distractor Objects
Five backgrounds, one to five objects, and four tasks produce 100 runs per model. Success rates: wrist 25%, independent 45%, DUET-DINO 63%; DUET-DINO FPE/FAE: 2.6±2.1 cm and 30.9±36.5°.
Go to primary source ↓e-hardwarePDF p. 6, Table IV and IV-D Hardware Runs; PDF p. 7, continuation before IV-E
30 hardware runs/three poses. DUET-DINO: 26.7%, 14.7±17.1 cm, 28.0±17.7°; independent: 16.7%, 14.3±17.2, 40.0±42.0; wrist: 3.3%. FR3/Robotiq/ZED hardware, Robot Control Stack and VLAgents are named. Budget: 500 candidates, one CEM iteration, 15 steps; collision/singularity rejection and termination after three unsafe steps.
Go to primary source ↓e-liftPDF p. 7, Table V and IV-E Angled Grasp and Angled Lift-to-Home
40 runs/four tasks; reach/grasp/lift budgets 60/10/60 steps. Reach uses 15 cm and 20°, lift requires retaining grasp within 20 cm of home. Reach/lift rates: wrist 57.5/20.0%, independent 60.0/57.5%, DUET-DINO 75.0/60.0%.
Go to primary source ↓e-attentionPDF p. 7, IV-F Cross-view Cross Attention
Removing attention yields reach 81%, FPE 10.7±15.4 cm; angled reach 42.5%, FPE 4.3±3.1 cm, FAE 55.5±52.8°. Parameter-matched and repeated-training-seed controls are not described.
Go to primary source ↓e-cameraPDF p. 7, IV-F Camera Perturbations
ReachBananaTask, 30 runs, matched seeds; side-camera shifts ±20 cm x/y, ±10 cm z, ±11.5° roll/pitch/yaw. DUET-DINO and independent DINOv3 both reach 66.7%; both single side-view predictors reach 0%.
Go to primary source ↓e-data-ablationPDF p. 7, IV-F Training Data, right column
DROID-only DUET-DINO is trained for 94.5k steps and achieves 78% reach and 72.5% angled reach, compared with the main 120k-step mixed-data configuration.
Go to primary source ↓e-latent-changePDF p. 7, Figure 5, global legend/panel headings; IV-G on pp. 7–8
Caption reports mean latent changes over 30 runs. Solid blue/red are globally labeled V-JEPA 2/DINOv3, while predictor columns use blue/red reference curves under DINOv3/V-JEPA 2 headings. IV-G attributes overestimated dynamics to DINOv3 predictions and underestimated wrist dynamics to V-JEPA 2. The side-view encoder ranking in text also differs from a literal legend reading.
Go to primary source ↓e-patchPDF p. 7, Figure 6 and caption; PDF p. 8, IV-G patch correspondence paragraph
Reference-patch L1 distances are shown for encoded t, encoded t+1 and predicted t+1 maps. Lower means more similar features. DINOv3 uses 14×14 patches of 1280 dimensions, V-JEPA 2 uses 16×16 of 1408. Authors describe more coherent object correspondences and better preservation by DINOv3 in this example.
Go to primary source ↓e-runtimePDF p. 6, continuation of IV-C, left column below Table II
DUET-DINO requires 15–17 seconds per planning step. Dual-view models are roughly twice as slow as single-view models; cited prior-work timing uses a different CEM budget.
Go to primary source ↓e-conclusionPDF p. 8, V. Conclusion
Expensive CEM world-model evaluation limits real-time control. Broader real-world tasks and generalist-policy action proposals are future work.
Go to primary source ↓8.5 Primary sources
DUET-DINO: Simultaneous Cross-View World Modeling for Latent Planning in Robot Manipulation ↗
PDF · 6,496 extracted words
Source fingerprint
830fc4749ec9be12f2721134c0a9e95e6f63cea434e9fcf3e63f41665f5c7182