Visuo-Tactile World Models
1. Paper overview
In one sentence: Predicting touch alongside vision improves contact-sensitive imagination and goal-image planning, but execution remains computationally expensive and evidence is confined to a narrow robot setting. e-probleme-architecturee-planninge-robote-adaptatione-limitations
| At a glance | What to know |
|---|---|
| Research problem | Source description A hand hovering above cloth can look like a hand pressing it, while an occluded grasp hides whether an object is still held. A vision-only predictor can therefore imagine motion without contact or lose objects. VT-WM adds local touch observations to global camera context to resolve these ambiguities. e-problem |
| Core mechanism | Author claim A multi-task latent forward model integrates four fingertip sensors with exocentric vision and action conditioning. The claimed first-of-its-kind status is the authors’ priority claim, not independently established here. e-probleme-architecture |
| A key reported result | Reach & push: VT-WM 93%. Reported success rate (%).. Real-robot, two-subgoal CEM planning with open-loop execution; five trials per task are stated, using distinct initial conditions in the same scene. V-WM 69%, from Figure 7’s rounded bars. The printed difference is 24 percentage points. The figure annotates a 35.1% relative gain; exact aggregation and unrounded rates are not supplied. This is executed manipulation, not merely plausible video. e-robote-execution-examplese-limitations |
| Reading caution | Source description Evaluation is limited to Digit 360 sensing, unseen trajectories within familiar tasks, and planning with the same scene and objects. New object properties and other sensor modalities remain untested. CEM computation restricts feedback to chunk boundaries. e-limitations |
Core contributions
- Author claim
A multi-task latent forward model integrates four fingertip sensors with exocentric vision and action conditioning. The claimed first-of-its-kind status is the authors’ priority claim, not independently established here. e-probleme-architecture
- Source description
The evaluation connects contact-sensitive rollout diagnostics to executed CEM plans and separately explores adaptation with a small demonstration set. e-permanencee-causale-robote-adaptation
Figure 3. A shared sensory predictor combines scene history, fingertip contact and candidate actions. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow the graphic from the camera and fingertip images on the left to the predicted latents on the right. Blue tokens represent visual states s_k; purple tokens represent tactile states t_k. The padlock on Cosmos agrees with Appendix B’s frozen visual encoder, while Sparsh-X is fine-tuned. The pink action arrow enters the transition predictor: actions condition predictions rather than emerge as its outputs. Section 3.2.1 explains the internal computation omitted from the block diagram: spatial and temporal self-attention, followed by cross-attention to action tokens. Separate output projectors recover the two sensory modalities. The displayed stream rates should be read alongside the training window and action-chunk details. e-architecturee-inputse-traininge-planninge-notation-gaps
What it supports. The architecture makes touch available when predicting the next visual scene as well as the next tactile state. Its practical purpose is to distinguish visually similar situations with different contact conditions. This is a shared forward sensory model; choosing commands still requires the separate CEM procedure described in the text.
Where the evidence stops. The original graphic repeats D360_2 on two tactile latent stacks, although the sensor images and text identify four fingertips. Preserve this labeling discrepancy: it does not establish a duplicated sensor input or resolve the exact token-to-finger mapping.
2. Motivation
2.1 The problem and the proposed response
A hand hovering above cloth can look like a hand pressing it, while an occluded grasp hides whether an object is still held. A vision-only predictor can therefore imagine motion without contact or lose objects. VT-WM adds local touch observations to global camera context to resolve these ambiguities. e-problem
2.2 What this reading follows
A robot hand can look correctly placed above a cloth while failing to press it. A camera-only world model may then imagine successful wiping even when the real cloth stays still. VT-WM tackles this ambiguity by predicting future visual and fingertip-touch representations from shared context and candidate actions. Its planner still scores only the predicted visual endpoint against a goal image; touch changes the dynamics behind that score. The paper evaluates both imagined trajectories and real robot execution, then explores adaptation to a plate-placement task. Read the plots with their protocol boundaries intact: better contact-sensitive rollouts, successful commands, and learning a new task are distinct pieces of evidence. e-probleme-architecturee-planninge-robote-adaptatione-limitations
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | WAMs |
| Architecture | Dual-system |
| Prediction paradigm | Other mechanisms |
| Quadrant | Outside quadrants |
3.1 Evidence-based assessment
Supports the recorded classification
The recorded Dual-system/Other mechanisms/Outside quadrants labels fit a learned forward world model coupled to a separate CEM action optimizer. Joint visual/tactile prediction does not imply joint future/action prediction or inverse dynamics. A shared sensory predictor is therefore insufficient evidence for a One Model controller. The latent-prediction and tactile tags are supported; this paper does not evaluate audio. e-architecturee-planninge-ceme-problem
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 1. Supply the contact state that the camera cannot determine
Consider two nearly identical camera views of a hand around an object: one has contact and the other does not. The paper’s central modeling choice is to encode both the scene and recent fingertip observations before asking what an action will do. Cosmos supplies visual latents, while Sparsh-X supplies touch latents. Their projected tokens share a predictor, allowing spatial attention to combine modalities within a timestep and temporal attention to connect their histories. Actions then enter through cross-attention. The predictor returns future sensory latents for both modalities. Figure 3’s separate output heads therefore indicate two prediction targets, not a world-model branch and a policy branch. Appendix B further clarifies that the visual tokenizer is frozen while the tactile encoder adapts to sensor-specific variation. e-probleme-architecturee-training
5.2 2. Train for the context the model will create for itself
Teacher forcing teaches the predictor with real preceding states and real future latent targets. That gives a dense next-step learning signal, but deployment requires repeatedly feeding predictions back into the model. The paper addresses this mismatch with a second loss evaluated using autoregressively sampled context. Those context states are generated without gradients, typically over three to five steps, while the subsequent visual and tactile predictions are compared with ground-truth latents using L1 error. The total objective weights the teacher and sampling losses equally. This supplies a concrete reason to inspect long rollouts rather than only one-step reconstructions. The source presents combined-system results against V-WM, however, so it does not separately establish how much the sampled-context objective contributes to the reported contact improvements. e-losse-visual-scope
5.3 3. Turn better imagined contact into a candidate action ranking
At planning time, the goal remains an image. CEM proposes action sequences, the world model predicts each sequence’s visual and tactile future, and a terminal visual-latent distance ranks the proposals. Touch influences this ranking through the predicted dynamics and the initial contact state; it is not an additional goal or force penalty. The reader’s causal interpretation is that a more faithful model should make impossible no-contact pushes look less promising to the optimizer. Appendix examples illustrate this connection, but do not prove that this is the sole cause of the success-rate gain. Commands execute open-loop within trajectory chunks, with updated sensing between planning calls. This feedback schedule and the incomplete CEM pseudocode must be preserved when interpreting or attempting to reproduce the robot results. e-planninge-ceme-execution-examplese-robote-limitations
5.4 Training and inference
During training
RGB clips contain nine frames at 6 fps, 320×192 pixels, described as 1.5 seconds. Touch uses two frames per sensor from the latest 0.16 seconds. Thirty-Hz action deltas are grouped in fives. The model’s maximum context is nine frames. e-inputs
Both models use 124 training demonstrations (112k datapoints; mean duration 40 seconds) and 26 validation demonstrations (17k datapoints), spanning eight tasks and including failures. Data come from a Franka Panda, Allegro Hand, four Digit 360 sensors and an exocentric camera. e-data
Cosmos remains frozen while Sparsh-X is fine-tuned. Of 173M parameters, 96M are trained. AdamW uses betas 0.9/0.95 and weight decay 0.01; 10k warmup updates reach 3e-4, followed by cosine decay to 3e-7 over 80k total updates. Effective batch size is 64 across 32 A100 GPUs. e-training
During inference
CEM rolls out candidate actions and minimizes final visual-latent L2 distance to the goal image. There is no tactile goal: touch informs dynamics and initial contact state. Commands execute open-loop within chunks; Appendix D refreshes sensor context between CEM calls. e-planninge-cem
Algorithm 1 declares a 2-second horizon at 6 Hz, 36 particles, 10 search iterations, five elites and up to three CEM calls. These are reported settings, subject to the pseudocode inconsistencies discussed below. e-cem
5.5 Implementation flow
- Encode complementary observations
Cosmos encodes the global scene; Sparsh-X encodes fingertip contact. Projected visual and tactile tokens receive positional information and are concatenated spatially. e-architecture
- Predict action-conditioned sensory futures
A 12-layer transformer alternates factorized spatial/temporal self-attention with cross-attention to actions. RoPE supplies relative positions; separate output projectors recover visual and tactile latents. Actions are inputs, not jointly predicted outputs. e-architecture
- Inspect imagined contact
Pretrained decoders visualize predicted latents. Appendix examples show commanded motion and plausible contact occurrence, while acknowledging imperfect per-finger contact localization; these illustrations are not contact-force measurements. e-action-diagnosticse-tactile-rollout
6. Experiments & results
VT-WM predicts future visual and fingertip-touch latents under candidate robot actions, then supplies those predictions to a separate goal-image CEM planner. Contact sensing improves rollout consistency and reported real-robot planning on familiar contact-rich tasks. The evidence supports this restricted setting; it does not establish novel-object generalization or high-frequency feedback control.
The supplied paper has quantitative bar plots but no quantitative results table. It also contains no dedicated component-isolating ablation: Figure 5 is a qualitative contact/no-contact diagnostic, used here in the ablation section without relabeling it as a controlled ablation. Figures 4, 6 and 7 supply the original quantitative evidence. No table or missing experiment has been reconstructed. e-visual-scopee-diagnostice-permanencee-causale-robot
6.1 Read the original evidence
Figure 4. Moving-object trajectory error falls with tactile conditioning, though significance varies by task. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each task as a paired comparison: teal is V-WM and purple is VT-WM. The vertical axis is normalized Fréchet distance, so lower bars indicate closer agreement between imagined and real object trajectories. These are rollouts under the corresponding real action sequences, not independently optimized robot plans. Section 4.1 explains that CoTracker supplies object tracks, expressed relative to initial position and normalized by the ground-truth trajectory length. The caption identifies the whiskers as 95% confidence intervals. Compare the within-task bars first; do not read the height as a success percentage. The accompanying paired tests, rather than bar overlap alone, determine the reported statistical conclusions. e-permanencee-limitations
What it supports. The authors report relative reductions of 18–47% across these five tasks, averaging approximately 33%. Placing fruits, pushing fruits and stacking cubes have significant paired improvements. Wiping and scribbling trend downward without reaching the stated significance threshold, so the aggregate improvement should not imply equally strong evidence for every task.
Where the evidence stops. This is an image-track proxy for object permanence. It evaluates unseen trajectories within the training tasks and does not establish generalization to new objects, exact contact forces, or robot task success.
Figure 6. Touch reduces spurious motion on average, with a visible exception for scribbling. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. This plot uses the same model colors and lower-is-better distance as Figure 4, but changes which objects are tracked. Here the targets are passive objects that should remain stationary rather than objects intentionally moved by the robot. The question is whether an imagined hand motion creates unsupported object motion or deformation. Start with wiping, where the separation is visually large, and then examine scribbling, where the purple VT-WM bar is higher. The caption specifies 95% confidence intervals. Section 4.1 supplies paired-test results: placing fruits, pushing fruits and wiping improve significantly, while stacking and scribbling do not. The reverse direction in the last group must remain part of the interpretation. e-causale-permanence
What it supports. The source reports an approximately 29% mean relative reduction in this diagnostic. For placing fruits, pushing fruits and wiping, the stated reductions are 43.6%, 16.4% and 66.1%, respectively. The scribbling result goes in the opposite direction, so contact sensing does not uniformly improve every measure of passive-object stability.
Where the evidence stops. The paper does not explain how the reused trajectory-length normalization handles near-stationary ground-truth tracks. This image-space diagnostic neither measures forces directly nor establishes compliance with every physical law.
Figure 7. Executed plans improve most visibly on reach-and-push; adaptation results retain reporting ambiguities. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. The left five groups compare real-robot execution after CEM planning; the images below specify the first and, where applicable, second goal. Read the percentages inside the bars before reading the blue gain annotations above them. For reach-and-push, the displayed rates are 69% for V-WM and 93% for VT-WM: a 24-percentage-point difference between rounded values. The blue 35.1% label describes a relative gain, not a 35.1-point increase. The right-hand group is a different experiment: continued training with 20 plate-task demonstrations followed by ten trials. This demonstrates planning after task-specific adaptation; the paper’s zero-shot transfer language concerns deploying generated plans. e-robote-adaptatione-planninge-execution-examplese-limitations
What it supports. The strongest familiar-task comparison is reach-and-push, which requires first establishing contact and then moving the fruit. Wiping also improves, from 70% to 92%, while reaching is tied at 100%. These results support the usefulness of contact-aware predictions for selecting executed action sequences in the tested scene, subject to the reporting limits.
Where the evidence stops. Five trials per familiar task are stated, but aggregation and raw counts are unspecified. For plate adaptation, text reports 77% while the bar reads 78%; the 250.0% annotation lacks a defined comparator. Rounded bars do not recover all blue gain labels exactly. No missing denominator or baseline is inferred.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Object permanence in imagined rollouts Unseen trajectories within training tasks: place fruits, push fruits, wipe cloth, stack cubes and scribble with marker; models receive corresponding real action sequences. | VT-WM reports 18–47% relative reduction across tasks, approximately 33% on average. Normalized Fréchet distance of CoTracker moving-object trajectories; lower is better. | Multi-task V-WM; Figure 4 shows 95% confidence intervals. Paired improvements are significant for placing, pushing and stacking, but not wiping or scribbling. This is a trajectory-consistency proxy, not a physical-law guarantee. e-permanencee-limitations |
| Causal compliance in imagined rollouts Same five-task contact-perception protocol, tracking objects expected to remain stationary. | Approximately 29% mean relative reduction; 43.6% for placing fruits, 16.4% for pushing fruits and 66.1% for wiping. Normalized Fréchet distance; lower means less spurious motion. | V-WM; Figure 6 supplies 95% confidence intervals. Only these three improvements are significant; stacking is not, and scribbling worsens without significance. Stationary-object normalization requires implementation clarification. e-causal |
| Reach & push Real-robot, two-subgoal CEM planning with open-loop execution; five trials per task are stated, using distinct initial conditions in the same scene. | VT-WM 93%. Reported success rate (%). | V-WM 69%, from Figure 7’s rounded bars. The printed difference is 24 percentage points. The figure annotates a 35.1% relative gain; exact aggregation and unrounded rates are not supplied. This is executed manipulation, not merely plausible video. e-robote-execution-examplese-limitations |
| Other real-robot planning tasks Same five-trial-per-task planning protocol; reach button and push fruits have one goal, wiping and stacking have two. | Reach button 100%; push fruits 92%; wipe cloth 92%; stack cubes 83%. Reported success rate (%), VT-WM versus V-WM. | V-WM: 100%, 83%, 70% and 75%, respectively. Figure 7 reports rounded bars without raw denominators or uncertainty; do not infer integer successful-trial counts from the stated five trials. e-robote-execution-examples |
| Place plate in the dish rack after adaptation Continue multi-task training with 20 added demonstrations; ten real-robot trials randomize the initial plate-in-grasp pose, with alignment and insertion subgoals. | 77% in Section 4.3; 78% in Figure 7. Reported success rate (%). | No numerical baseline is identified for the plot’s unexplained 250.0% gain annotation. The discrepancy remains unresolved. This tests planning after task-specific fine-tuning, not zero-shot learning of an unseen task; raw counts and aggregation are unspecified. e-adaptation |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 5 (qualitative diagnostic). No-contact examples expose hallucinated object motion and disappearance. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read time from left to right within each example, then compare the three rows vertically. The top row is the observed trajectory; the middle and bottom rows are VT-WM and V-WM predictions under the corresponding actions. On the left, colored keypoints help reveal whether the cloth moves even though the hand remains above it. On the right, the fingers close without successfully taking the marker. The correct outcome is therefore that the marker stays in place. Pay attention to the original annotations: VT-WM is still marked with slight cloth deformation, while V-WM shows object motion or disappearance. These are selected diagnostic examples of contact ambiguity. e-diagnostice-permanencee-visual-scope
What it supports. The visual makes the proposed mechanism concrete: hand motion alone need not cause object motion. In these examples, adding tactile context produces imagined outcomes closer to the observed non-contact behavior. The retained slight-deformation annotation also shows why improved contact grounding should be understood as a reduction in artifacts, not their elimination.
Where the evidence stops. This qualitative comparison is not a component-isolating ablation or an estimate of failure frequency. It does not separate tactile training effects from initial tactile context, and the illustrated predictions remain imperfect.
7. Analysis & limitations
7.1 What the evidence leaves open
Evaluation is limited to Digit 360 sensing, unseen trajectories within familiar tasks, and planning with the same scene and objects. New object properties and other sensor modalities remain untested. CEM computation restricts feedback to chunk boundaries. e-limitations
No dedicated ablation separates tactile training supervision, initial tactile context, encoder adaptation and sampled-context training. The V-WM comparison supports a combined system benefit, but cannot assign each component’s contribution. e-visual-scope
7.2 Questions for discussion
- How much benefit comes from tactile training targets versus correct initial tactile context?
- Would contact-aware plan ranking persist for new objects when vision changes little but friction or compliance changes?
8. Reproducibility audit
8.1 Requirements and known gaps
Reproduction requires the synchronized demonstrations and validation split, pretrained encoders, sensor calibration and the described robot. The datapoint unit, detailed synchronization/windowing, stationary-track normalization, success aggregation, runtime and software versions are insufficiently specified for exact replication. e-datae-traininge-inputse-causale-robot
Resolve Algorithm 1 before implementation: it declares P=36 particles but samples with N=10, writes (mu+sigma)*rand and omits updating best-cost. Training uses quaternion deltas while planning lists roll/pitch/yaw without a conversion. Figure 3 also duplicates D360_2 latent labels; the text specifies four sensors. None of these ambiguities is silently repaired here. e-ceme-notation-gaps
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Separate initial contact information from tactile training
Reader-proposed check, not performed: use held-out reach-and-push and no-contact wiping trajectories with identical RGB history and action sequences. Compare the full trained VT-WM under correct tactile context, time-shuffled context and a no-contact reference, alongside the original V-WM. Hold rollout horizon and visual scoring fixed; report moving-object error, passive-object error and contact/no-contact outcome separately. Shuffling is an input-corruption diagnostic, so also retrain a matched-capacity control without informative tactile targets to test training effects. A selective loss of VT-WM’s advantage when contact information is removed would support the proposed disambiguation mechanism. If correct and corrupted touch produce indistinguishable predictions, that mechanism would need stronger evidence. e-probleme-diagnostice-permanencee-causale-planninge-visual-scope
Check 2: Test whether visual goal scores predict executed contact success
Reader-proposed check, not performed: first resolve the CEM particle, sampling, orientation and best-cost ambiguities in a documented implementation. For matched initial poses, give V-WM and VT-WM the same candidate actions, planning budget, goal images and execution-chunk schedule on reach-and-push. Retain each predicted terminal cost, then independently label actual contact establishment and full two-subgoal success after execution. Publish per-trial outcomes, attempt counts, aggregation and confidence intervals. The falsifiable prediction is that VT-WM’s low-cost plans yield fewer no-contact failures and stronger cost-to-success agreement. An improved rollout metric without better executed ranking would weaken the claimed link from contact grounding to planning. e-ceme-notation-gapse-planninge-robote-execution-examplese-limitations
8.3 Reading coverage
Visual audit: Visually inspected the title, author affiliations and version marks; every original figure (1–17); method and loss pages; quantitative plots and diagnostic examples; Appendix B training details; Algorithm 1; and Appendix E limitations. Cross-checked Figure 3’s action direction and Cosmos lock against Sections 3.2 and B.1, preserving duplicated sensor labels. Read Figure 7’s printed rates against Sections 4.2–4.3 and retained the plate-rate and gain-annotation discrepancies. All five final crops were individually viewed with labels and legends intact; the two narrow quantitative plots were cropped from 400-dpi PDF renders. Reference pages 9–11 were read as text, not visually rendered. Separate supplemental material was not verified; no external code or experiments were inspected.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 12, 13, 14, 15, 16, 17, 18, 19. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract; Sections 1–2 (Introduction and Related Works)
- Sections 3, 3.1 and 3.2.1–3.2.3 (contact sensing, architecture, training and planning)
- Sections 4.1–4.3 (contact perception, robot planning and adaptation); Section 5
- References, PDF pp. 9–11
- Appendices A–F, including B.0.1, B.1, C, D.1 and Algorithm 1
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Version scope: the inspected artifact is arXiv:2602.06001v1, dated 5 February 2026 in the margin, with a 6 February 2026 preprint footer. Title and all six authors match the catalog; no other revision was inspected.
- The supplied extraction does not reconstruct figure images; this was addressed by inspecting the retained PDF pages and all final crops.
- Separate supplemental material availability has not been fully verified.
- Code, external resources and experiments were not inspected or executed. References were read as text; their cited works were not opened.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e-identityPDF p. 1, title block, affiliations, arXiv margin and preprint footer
Exact title and six authors match the catalog. The margin identifies arXiv:2602.06001v1, 5 February 2026; the footer says Preprint, February 6, 2026. Affiliations are the Paul G. Allen School of Computer Science, University of Washington, and FAIR, Meta.
Go to primary source ↓e-problemPDF pp. 1–3, Abstract, Sections 1 and 3.1, Figures 1–2
Touch is proposed to disambiguate occlusion and visually similar contact/no-contact states; the platform uses Digit 360 fingertips. The authors claim a first multi-task visuo-tactile world model.
Go to primary source ↓e-architecturePDF pp. 3–4, Section 3.2.1 and Figure 3
Cosmos visual and Sparsh-X tactile latents are spatially concatenated and processed by a 12-layer predictor with factorized spatial/temporal self-attention, action cross-attention, RoPE and modality-specific output projectors. Actions condition future sensory predictions.
Go to primary source ↓e-inputsPDF p. 4, Section 3.2.2
Training uses nine RGB frames at 6 fps and 320×192 resolution; tactile input has two frames per sensor over the latest 0.16 seconds, four sensors total. Actions include translation/quaternion deltas and binary hand state; 30 Hz actions are grouped in fives. Maximum context is nine frames.
Go to primary source ↓e-lossPDF pp. 4–5, Section 3.2.2, Teacher Forcing Loss and Sampling Loss equations
Both losses sum visual and tactile latent L1 errors. Teacher forcing uses ground-truth context; sampled context is generated autoregressively without gradients, typically for 3–5 steps. The two losses have equal weight.
Go to primary source ↓e-dataPDF p. 12, Appendix B.0.1 and Figure 8
Franka Panda, Allegro Hand, four Digit 360 sensors and an exocentric camera collect successful and failed teleoperation sequences across eight tasks. Training: 124 demonstrations, 112k datapoints, mean 40 seconds; validation: 26 demonstrations, 17k datapoints. Streams are timestamp-synchronized and downsampled to 6 fps.
Go to primary source ↓e-trainingPDF p. 12, Appendix B.1
AdamW uses beta1=0.9, beta2=0.95, weight decay 0.01; 10,000 warmup updates to 3e-4 and cosine decay to 3e-7 over 80,000 total updates. Effective batch size 64 spans 32 A100 GPUs. Cosmos is frozen, Sparsh-X is fine-tuned; 173M total parameters, 96M trained.
Go to primary source ↓e-planningPDF pp. 5 and 7, Sections 3.2.3 and 4.2
CEM optimizes action sequences against final visual-latent L2 distance to a goal image. Touch supplies initial context and predicted dynamics, but no tactile goal. Selected sequences execute open-loop. Planning searches wrist translation, three orientation coordinates and binary hand configuration.
Go to primary source ↓e-cemPDF pp. 14–15, Appendix D.1 and Algorithm 1
Algorithm 1 declares 6 Hz, a 2-second horizon, 36 particles, 10 iterations, seven action dimensions, five elites and up to three CEM calls with refreshed sensor context. Its particle-generation expression instead uses N (iterations), writes (mu+sigma)*rand, and never updates best-cost after initialization. These details prevent treating the pseudocode as an unambiguous implementation.
Go to primary source ↓e-permanencePDF pp. 5–6, Section 4.1 Object Permanence, Figure 4 and paired-test paragraph
CoTracker trajectories are relative to initial object position and normalized by ground-truth trajectory length. VT-WM reduces normalized Fréchet distance by 18–47% across five tasks, about 33% on average. Paired tests are significant for place fruits, push fruits and cube stacking, not wiping or scribbling; Figure 4 shows 95% confidence intervals.
Go to primary source ↓e-diagnosticPDF pp. 6–7, Figure 5 and discussion of contact/no-contact rollouts
Ground truth, VT-WM and V-WM are compared for a hand moving above a cloth and a failed marker grasp. VT-WM retains the stationary object more faithfully, but the cloth example still labels slight deformation; V-WM hallucinates motion or disappearance.
Go to primary source ↓e-causalPDF pp. 6–7, Section 4.1 Causal Compliance and Figure 6
The metric tracks nominally static objects using normalized Fréchet distance. Relative reductions are 43.6%, 16.4% and 66.1% for place fruits, push fruits and wipe cloth; the reported task average is about 29%. These three paired tests are significant; cube stacking is not (p=0.09), and scribbling degrades without significance (p=0.23). Figure 6 shows 95% confidence intervals.
Go to primary source ↓e-robotPDF pp. 7–8, Section 4.2 and Figure 7 left, bar labels
The paper reports five trials per task with distinct initial conditions. Printed V-WM/VT-WM success rates are reach button 100/100%, push fruits 83/92%, reach & push 69/93%, wipe cloth 70/92%, and stack cubes 75/83%. Blue gain annotations are 0.0%, 10.0%, 35.1%, 31.0% and 11.1%; their exact values are not all recoverable from the rounded bars. Raw counts and aggregation rules are absent.
Go to primary source ↓e-adaptationPDF pp. 7–8, Section 4.3 and Figure 7 right
Twenty plate-in-rack demonstrations are added to the multi-task dataset before continued training. Ten trials randomize initial plate-in-grasp pose and involve alignment and insertion. Text reports 77% success, while the bar reads 78%. A blue 250.0% gain annotation has no displayed comparator or explanation in the accompanying text.
Go to primary source ↓e-action-diagnosticsPDF pp. 12–14, Appendix C and Figures 9–11
Qualitative rollouts follow axis translations and hand commands. Held-out button and scribbling sequences condition prediction on real action deltas; pretrained decoders visualize latent predictions. The source acknowledges differences in per-finger contact location. Figure 11 shows a two-second table-leg grasp rollout.
Go to primary source ↓e-tactile-rolloutPDF p. 16, Figure 12
Cube-stacking snapshots compare real and predicted vision with real tactile signatures, predicted signatures and their difference from a no-contact state; these are qualitative contact illustrations.
Go to primary source ↓e-execution-examplesPDF pp. 16–19, Appendix D task descriptions and Figures 13–17
Examples distinguish plans in imagination from real execution: reach button, push fruits, reach & push, wipe cloth and stack cubes. The reach & push and wiping examples attribute vision-only failures to insufficient contact; stacking examples illustrate loss of the held cube in imagination.
Go to primary source ↓e-limitationsPDF pp. 17–18, Appendix E
Tactile experiments use Digit 360 only. Contact perception uses unseen trajectories within training tasks; planning randomizes initial robot states but holds scenes and objects fixed. CEM computation leads to open-loop trajectory chunks rather than high-frequency closed-loop policies.
Go to primary source ↓e-visual-scopePDF pp. 5–8, Section 4 and Figures 4–7; pp. 12–19, Appendices B–F and Figures 8–17
The empirical material comprises quantitative bar plots and qualitative diagnostics, with no quantitative results table or dedicated component-ablation study. The V-WM/VT-WM comparison does not separately remove tactile training supervision, initial tactile context or sampled-context loss.
Go to primary source ↓e-notation-gapsPDF p. 4, Figure 3 and Section 3.2.2; p. 7, Section 4.2; p. 15, Algorithm 1
Figure 3 labels two tactile latent stacks D360_2 while the sensor image labels D360_0 through D360_3. Training specifies quaternion rotation deltas, whereas planning lists roll/pitch/yaw. The conversion is not described.
Go to primary source ↓8.5 Primary sources
Visuo-Tactile World Models ↗
PDF · 9,162 extracted words
Source fingerprint
6a2e3ed505b4b687180522dff00618572c52766dcf03a9fe8e068156fcfa0adb