Disentangling Visuo-Tactile Foresight: Oracle-Guided Interface Discovery for World Action Models
1. Paper overview
In one sentence: With successful reference futures fixed, selective phase-local visual-to-tactile routing improves simulated action success, while leaving learned-future reliability and deployability untested. e-probleme-oraclee-routinge-controlse-main
| At a glance | What to know |
|---|---|
| Research problem | Source description End-to-end visuo-tactile control mixes physically invalid visual predictions, unreliable future branches, mismatched tactile forecasts and poor consumption of otherwise useful futures. OVTF fixes a paired, chronological, simulation-verified successful provider matched to the current initialization, asking which representation lets an actor exploit that information. e-probleme-oracle |
| Core mechanism | Source description OVTF makes future-to-action interfaces independently studyable while excluding future actions and state metadata from oracle inputs. e-oracle |
| A key reported result | Seven-task UniVTAC manipulation: AFM Oracle: 112/350, 32.0%. Successes / rollouts and average success rate. Simulation, evaluation episodes 50–99 per task, 50 rollouts each; vendor success predicate. Put Shelf trains on 0–39 with development 40–49; other tasks train on 0–49. IFM: 83/350, 23.7%; UniVTAC-ACT*: 52/350, 14.9%. The reported Oracle–IFM difference is 8.3 percentage points under matched future source, budget and training. The asterisk identifies UniVTAC's implementation rather than original ACT. e-maine-splitse-routinge-baseline |
| Reading caution |
Core contributions
- Source description
OVTF makes future-to-action interfaces independently studyable while excluding future actions and state metadata from oracle inputs. e-oracle
- Source description
AFM performs asymmetric fusion before ACT: tactile slots can read vision, visual slots read only vision, and opposite tactile streams cannot access each other at this interface. e-routinge-architecture
- Author claim
The authors separate routing quality, test-time future dependence and the policy path without futures through IFM, Forced-Zero and Zero-Train controls. e-controlse-ablation
Figure 1. A privileged future is compressed before it meets current evidence inside ACT. Original paper, p. 2 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow the three horizontal paths into ACT. Current images use frozen encoders, state becomes a proprioception token, and the successful future suffix passes through trainable encoders and AFM. The inset matrices use destination memories as rows and source streams as columns: H denotes vision, L left touch and R right touch. The additional blue cells in AFM let L and R read H; they do not let H read touch. This agrees with Eqs. (9)–(10). Below, four anchors per modality illustrate the soft phase preference. The drawing summarizes twelve slots, while Eq. (11) specifies their actual integration into ACT memory. e-architecturee-routinge-actore-encoding
What it supports. The distinguishing operation occurs before the shared action transformer. Each tactile slot jointly reads its own tactile stream and visual context; it does not perform two sequential reads. The resulting slots can interact again inside ACT, so AFM versus IFM tests explicit organization at the interface.
Where the evidence stops. The phase curves are schematic, not measured attention distributions. Equation (6) imposes a soft Gaussian preference rather than a hard window. Cross-tactile blocking applies to AFM's reader, not to all later ACT computation.
2. Motivation
2.1 The problem and the proposed response
End-to-end visuo-tactile control mixes physically invalid visual predictions, unreliable future branches, mismatched tactile forecasts and poor consumption of otherwise useful futures. OVTF fixes a paired, chronological, simulation-verified successful provider matched to the current initialization, asking which representation lets an actor exploit that information. e-probleme-oracle
2.2 What this reading follows
A successful future is useful only if the action policy can read it. OVTF studies this interface by supplying paired visual and tactile observations from successful simulation trajectories matched to the current initialization. AFM then compresses the remaining trajectory into phase-local memory, letting each tactile memory consult vision while keeping the two tactile streams separate at that stage. The key comparison holds the oracle and training setup fixed and removes those visual-to-tactile edges. Read the architecture alongside the task table and no-future controls: together they show an interface advantage, substantial task variation, and a policy that also benefits from changes beyond access to future content. e-probleme-oraclee-routinge-controlse-main
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Foundational work |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
This table preserves the labels recorded at reading time. The current major category is WAMs. View the current classification.
3.1 Evidence-based assessment
Supports the recorded classification
The foundational evaluation/action-policy classification is supported. OVTF fixes an external oracle and trains an action interface; it neither learns a joint future/action predictor nor extracts actions by inverse dynamics. A shared ACT consumer does not establish a unified predictive world-action architecture. The architecture, prediction-paradigm and quadrant axes are therefore not applicable, rather than unverified. e-probleme-oraclee-actor
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Fix the provider before judging the reader
OVTF begins with a successful demonstration matched to the current initialization. At a training action-window index, the provider samples the remaining demonstration, including the reference frame, and supplies only RGB and tactile observations. During rollout, a policy-call rule advances that reference index; the controller still receives fresh current observations. This design removes the need to predict a physically valid multimodal future while studying how future information reaches actions. It also defines the interpretation boundary: the future belongs to a successful reference trajectory, rather than a learned continuation conditioned on every deviation of the live policy. The experiment therefore asks how well a consumer uses privileged foresight. It does not measure whether a world model can generate that foresight or keep it aligned during deployment. e-probleme-oraclee-actor
Figure 2. Paired reference observations preserve the visual scene and both tactile streams through task progression. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the four columns from left to right. Each column pairs a large head-camera image with two smaller tactile images labelled L and R. The blue bottle changes position relative to the pink shelf, while the tactile views retain their marker pattern and contact-dependent appearance. The top labels locate samples at the centers of four equal trajectory segments. These samples explain what a multimodal reference contains; they are not the complete future sequence consumed by AFM. Section III.A instead samples up to 100 remaining frames at each reference index and encodes them as three sequences. e-referencee-oraclee-splits
What it supports. The oracle supplies synchronized observations from a successful reference, rather than future actions or privileged simulator state. This makes visual and tactile evidence available together while leaving the learned actor responsible for action generation. The figure establishes the observation format and chronological task example.
Where the evidence stops. Although the caption calls this a real reference trajectory, the experiments are in simulation. These whole-trajectory t/T labels are illustrative; they should not be equated with attention weights or the remaining-suffix phase defined in Eq. (1).
5.2 Use phase to organize joint visual–tactile attention
The future encoders preserve sequences instead of collapsing each modality to one token. AFM assigns four learned slots to each of vision, left touch and right touch, with fixed anchors spanning the remaining-horizon phase. Its attention score combines content similarity with a Gaussian penalty for phase distance. A left-touch slot jointly reads left tactile features and visual features in one attention operation; the right-touch slot follows the corresponding rule. Vision slots read only vision. IFM retains the slots, anchors and parameterization but removes the visual sources from tactile reading. All resulting tokens can later mix inside ACT. This makes the comparison about where explicit fusion is introduced, while the unchanged phase prior means the comparison does not independently validate the chosen anchor locations. e-encodinge-routinge-actor
5.3 Separate an interface gain from dependence on oracle inputs
Reader interpretation: three comparisons support different conclusions. Oracle versus IFM is the closest test of routing because the future provider, budget and optimization are held fixed. Oracle versus Forced-Zero asks whether the trained policy uses its future slots during evaluation. Zero-Train versus the released baseline shows that the adapted policy can improve even without future data, so the full Oracle-to-baseline gain is not a pure estimate of foresight value. The task rows also matter. Pull Out Key drops sharply when an Oracle-trained policy loses future inputs, whereas Insert Tube has equal Oracle and Forced-Zero totals. Those patterns motivate testing future dependence task by task and across seeds. They do not establish that any given deployment will benefit from a learned future provider. e-controlse-routinge-ablatione-maine-task-variatione-training
5.4 Training and inference
During training
The CVAE/action encoder is disabled and latent z remains zero. Future encoders, temporal filters, slot reader, ACT transformer and action head are trainable. Masked action L1 is the sole optimization loss; detached slot decorrelation does not backpropagate. e-actor
Task-specific UniVTAC-ACT* checkpoints initialize all AFM-family arms. Training lasts 2,000 steps on two RTX 4090 D GPUs, with micro-batch 1 per GPU and accumulation 8, yielding effective batch 16. AdamW ordinary/gate learning rates are 10⁻⁵/10⁻³, with weight decay 10⁻⁴/0. Cosine annealing ends at 10⁻⁶; clipping is 1.0. Results use step 2,000. e-training
During inference
Fresh observations and scheduled reference frames enter a transformer with four encoder layers, seven decoder layers and 50 action queries. Overlapping chunks use exp(−0.01 × age) weighting. The reference index follows policy calls; the paper explicitly supplies no physical-frequency interpretation. e-oraclee-actore-training
Oracle and IFM read futures during training and evaluation. Forced-Zero uses the Oracle-trained checkpoint but zeros future-memory slots at evaluation. Zero-Train never reads future HDF data and injects literal 12 × 512 zeros before future modules throughout training and evaluation. e-controls
5.5 Implementation flow
- Select the successful suffix
From training reference index τ_ref, uniformly sample up to 100 frames through trajectory end, inclusively. At rollout, advance the reference by Eq. (2) and resample the remaining suffix. These reference observations are not contiguous predictions from the live state. No future action, qpos, object pose, simulator state or success label is supplied. e-oracle
- Encode temporal streams
Trainable future ResNet-18 encoders copy released current-encoder weights. Figure 1 depicts separate vision and shared tactile encoders. Pooling and projection produce H, L and R sequences of width 512; two-camera tasks fuse vision with a two-layer MLP. Each stream adds a zero-gated depthwise temporal residual, with kernel 5, padding 2 and 512 groups. e-encodinge-architecture
- Read phase-local memory
Twelve learned 512-D slots comprise four per modality, anchored at phases 0.125, 0.375, 0.625 and 0.875. Each tactile slot jointly attends over its own tactile sequence concatenated with vision. Gaussian phase penalties favor nearby phases without excluding distant valid positions. IFM removes only visual-to-tactile reading edges, retaining parameterization and memory budget. e-routing
- Integrate with the actor
Frozen current encoders and state projection supply current memory. Zero-initialized tanh gates scale future slots. ACT memory places the first gated slot on the zero-latent base, followed by proprioception, remaining slots and current spatial tokens. The ACT encoder permits subsequent modality interaction; interface isolation is not an actor-wide mask. e-actore-architecture
6. Experiments & results
Oracle Visuo-Tactile Foresight (OVTF) studies how an action policy consumes successful future RGB and tactile observations. Privileged reference trajectories replace a learned future predictor. Asymmetric Phase-Local Future Memory (AFM) compresses these streams into slots with selective visual-to-tactile attention. Across seven UniVTAC simulation tasks, AFM records 32.0% success versus 23.7% with isolated modality routing. This diagnoses an interface under oracle access; it does not establish performance with predicted futures.
6.1 Read the original evidence
Table I. The pooled AFM advantage accompanies large differences in task-level future dependence. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the rightmost Oracle column, then compare IFM to test routing under the same oracle provider. Compare Forced-Zero to Oracle to remove future slots from the same trained checkpoint. Zero-Train is a separately trained no-future arm, and UniVTAC-ACT* is the released benchmark policy. Each cell gives successes out of 50 and a percentage. Bold marks the highest row score, including ties; underlining marks the second-highest distinct score. The Average row pools 350 rollouts across equally sized tasks. Look especially at Pull Out Key, the Oracle/Forced-Zero tie on Insert Tube, and the IFM maximum on Insert Hole. e-maine-task-variatione-controlse-splitse-routinge-baseline
What it supports. Oracle records 112/350 successes, or 32.0%, versus IFM's 83/350, or 23.7%. The reported average advantage is 8.3 percentage points. Yet Insert Tube gives both Oracle and Forced-Zero 42%, and Insert Hole favors IFM at 8% versus Oracle at 4%, preventing a uniform-benefit interpretation.
Where the evidence stops. These are simulated execution successes under privileged oracle access. Each task has only 50 rollouts, and no uncertainty intervals or repeated-training-seed summaries accompany the table. The explicit fairness contract covers the AFM-family arms.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Seven-task UniVTAC manipulation Simulation, evaluation episodes 50–99 per task, 50 rollouts each; vendor success predicate. Put Shelf trains on 0–39 with development 40–49; other tasks train on 0–49. | AFM Oracle: 112/350, 32.0%. Successes / rollouts and average success rate | IFM: 83/350, 23.7%; UniVTAC-ACT*: 52/350, 14.9%. The reported Oracle–IFM difference is 8.3 percentage points under matched future source, budget and training. The asterisk identifies UniVTAC's implementation rather than original ACT. e-maine-splitse-routinge-baseline |
| Future-availability ablation across seven tasks Same 350 evaluation rollouts; Forced-Zero reuses the Oracle checkpoint, whereas Zero-Train is separately trained without future data. | Forced-Zero: 68/350, 19.4%; Zero-Train: 70/350, 20.0%. Successes / rollouts and average success rate | Oracle: 112/350, 32.0%; benchmark baseline: 52/350, 14.9%. Slot removal lowers Oracle performance by 12.6 percentage points. No-future arms also exceed the baseline, so the entire baseline-to-Oracle gain cannot be attributed to future information. e-controlse-ablatione-main |
| Pull Out Key Head RGB and both tactile streams; train episodes 0–49, evaluate 50–99 in simulation. | Oracle: 27/50, 54%. Successful rollouts | IFM: 12/50, 24%; Forced-Zero: 1/50, 2%; Zero-Train: 15/50, 30%; baseline: 11/50, 22%. Strong oracle benefit coexists with severe loss when the trained policy loses expected future inputs. e-task-variatione-splitse-controls |
| Insert Tube Head/wrist RGB and both tactile streams; train episodes 0–49, evaluate 50–99 in simulation. | Oracle and Forced-Zero: each 21/50, 42%. Successful rollouts | IFM: 20/50, 40%; Zero-Train: 17/50, 34%; baseline: 7/50, 14%. Equal reported success totals do not establish an added test-time future benefit on this task. e-task-variatione-splits |
| Insert Hole Head RGB and both tactile streams; train episodes 0–49, evaluate 50–99 in simulation. | Oracle: 2/50, 4%. Successful rollouts | IFM: 4/50, 8%; Forced-Zero: 3/50, 6%; Zero-Train and baseline: each 1/50, 2%. IFM leads at low absolute success, limiting claims of uniform AFM superiority. e-task-variatione-splits |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 3. Routing and future availability answer different experimental questions. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the vertical axis as average success percentage across seven tasks, not a prediction-quality score. The dashed horizontal line is the 14.9% benchmark baseline. Oracle versus IFM changes the permitted reading edges while keeping oracle content and the stated training budget fixed. Oracle versus Forced-Zero holds the trained checkpoint fixed and removes future slots during evaluation. Zero-Train instead learns with no future HDF access. These comparisons should be followed in pairs rather than interpreted as four stages of one training run. Table II supplies the underlying totals of 70, 83, 68 and 112 successes out of 350. e-ablatione-controlse-main
What it supports. Removing future slots from the Oracle checkpoint reduces average success from 32.0% to 19.4%, a 12.6-percentage-point difference computed from the reported rates. Zero-Train still reaches 20.0%. Thus future content contributes, but the gain over the released baseline also includes an effective policy path without futures.
Where the evidence stops. The chart repeats the same evaluation summarized in Table I; it is not independent confirmation. Missing error bars prevent a robustness claim about the small Zero-Train/Forced-Zero difference, and slot removal introduces an evaluation-time input change.
7. Analysis & limitations
7.1 What the evidence leaves open
Successful, initialization-matched reference futures are privileged. The experiments test neither learned predictors nor forecast-error robustness or physical-robot deployment; simulation success cannot establish those capabilities. e-oraclee-splitse-main
Tables report 50 rollouts per task without confidence intervals or repeated-training-seed variability. The routing comparison retains the phase prior, so it does not separately establish the value of anchors, bandwidth or blocking cross-tactile access. e-routinge-maine-ablation
Reference progression follows policy-call indexing rather than measured task progress. Whether asynchronous behavior or phase mismatch erodes AFM's advantage remains untested. e-oraclee-routinge-ablation
7.2 Questions for discussion
- Does AFM retain its routing advantage with imperfect learned futures?
- Would phase misalignment selectively erase the advantage over IFM, or does visual access help without accurate phase correspondence?
8. Reproducibility audit
8.1 Requirements and known gaps
Reproduction requires task checkpoints, paired successful HDF trajectories and the episode/reset mapping, normalization, camera contract and vendor evaluator. The paper holds these fixed across AFM-family arms but does not enumerate numerical rollout horizons, early-stop settings or random seeds. e-baselinee-oraclee-splitse-training
The reported stack includes Ubuntu 20.04.6, two RTX 4090 D 24 GB GPUs, Python 3.10.12, PyTorch 2.7.0+cu128, torchvision 0.22.0+cu128, Isaac Sim 4.5.0 and Isaac Lab 0.41.3. Models use width 512, eight heads, dropout 0.1 and FP32 without AMP; future-encoding chunk size 8 is a memory optimization. e-traininge-software
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Repeat the routing comparison under matched training seeds
Reader-proposed, not run: initialize AFM Oracle and IFM from the same released task checkpoint for each of several predeclared seeds. Fix successful reference frames, phase anchors, memory dimensions, optimizer settings and the 2,000-step endpoint; vary only the visual-to-tactile reading mask. Evaluate both on the same 50 episode/reset rows per task with the vendor success predicate. Report per-task counts, paired episode outcomes and across-seed uncertainty alongside the pooled rate. The central claim predicts a positive average Oracle–IFM difference that survives seed variation; a sign-changing or negligible gap would weaken confidence in the single reported comparison. e-baselinee-oraclee-routinge-splitse-traininge-main
Check 2: Test whether cross-modal phase agreement drives the gain
Reader-proposed, not run: evaluate fixed AFM and IFM checkpoints using identical oracle image content, while shifting only the tactile phase tags relative to visual tags by predeclared offsets of ±0.125, clipped to [0,1]. Include unshifted controls and keep current observations, sample indices, reset mapping and action aggregation unchanged. Compare each model's success loss and the change in the AFM–IFM gap. Selective loss of AFM's advantage would support reliance on phase-aligned cross-modal reading. Similar degradation in both models would instead suggest general sensitivity to phase encoding; an unchanged gap would weaken the claim that accurate phase correspondence explains the routing benefit. e-oraclee-routinge-actore-splitse-ablation
8.3 Reading coverage
Visual audit: All six PDF pages were visually inspected: p. 1 identity/version and affiliations; p. 2 Figure 1 and its routing inset; p. 3 all method equations and training/inference definitions; p. 4 Figure 2, every Table I row, splits, controls and implementation/hardware/software details; p. 5 Figure 3, Table II, ablation interpretation and conclusion; p. 6 remaining references. All four final crops were inspected. Figure 1's reading directions agree with Eqs. (9)–(10); its phase curves are schematic soft support. Figure 2's whole-trajectory labels were distinguished from Eq. (1)'s remaining-horizon phase. No appendix is present; separate supplements were not supplied.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6. Appendix coverage: not present.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- PDF p. 1: title, author affiliations, version stamp, Abstract and I. Introduction
- PDF pp. 2–3: II. Related Work, A–B
- PDF p. 3: III. Method, A–C; Eqs. (1)–(12)
- PDF pp. 4–5: IV. Experiments, A–G, including training, evaluation conditions, main results and ablations
- PDF p. 5: V. Conclusion
- PDF pp. 5–6: References
Outside the original text pass
- Identity verified on PDF p. 1: title and all three authors match the supplied record; the observed edition is arXiv:2608.00547v1 [cs.RO], 1 August 2026. No title/author discrepancy was found; other revisions were not supplied or compared.
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- The extraction limitation was addressed by visually inspecting all six PDF pages, all three figures and both tables, plus four final original crops.
- No appendix or dedicated limitations section is present in the six-page PDF. Separate supplements were not supplied.
- Code and external resources were not inspected; no experiments were reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e-identityPDF p. 1, title/author block and left-margin arXiv stamp
Title matches the supplied observedTitle. Authors: Zihang Yao, Chaoyue Ding, Yingying Yu. Yao: Brigham Young University, Provo, Utah, USA; Ding and Yu: Beijing Academy of Science and Technology, Beijing, China. Yu is corresponding author. Stamp: arXiv:2608.00547v1 [cs.RO], 1 Aug 2026.
Go to primary source ↓e-problemPDF p. 1, Abstract and Section I, four confounds and OVTF proposal
The authors separate invalid visual futures, unreliable future use, mismatched tactile predictions and poor interfaces by fixing successful paired reference futures.
Go to primary source ↓e-baselinePDF p. 2, Section II.A, final paragraph
UniVTAC-ACT* denotes the implementation and task checkpoints distributed with UniVTAC, distinct from original ACT.
Go to primary source ↓e-architecturePDF p. 2, Figure 1 and caption
Frozen current vision/shared-tactile encoders, state projection and trainable future encoders feed ACT. Future memory has 4H, 4L and 4R slots. Routing matrices agree with Eqs. (9)–(10); modalities can interact again inside ACT.
Go to primary source ↓e-oraclePDF p. 3, Section III.A, first four paragraphs and Eqs. (1)–(2)
Inputs include task-dependent cameras, both tactile images and 8-D qpos. Up to 100 frames are sampled from a successful initialization-matched suffix. Rollout uses min(floor(p/2), T−1), without a physical-frequency interpretation. Oracle inputs exclude actions, qpos, object pose, simulator state and success labels.
Go to primary source ↓e-encodingPDF p. 3, Section III.A, encoder paragraphs and Eqs. (3)–(4)
Copied trainable ResNet-18 weights produce three N × 512 streams; two-camera vision uses a two-layer MLP. Depthwise temporal convolution has kernel 5, padding 2, 512 groups and initially zero tanh gates.
Go to primary source ↓e-routingPDF p. 3, Section III.B, Eqs. (5)–(10)
Twelve learned 512-D slots use anchors 0.125, 0.375, 0.625, 0.875 and Gaussian scale 0.18. Tactile AFM slots attend jointly to their own tactile plus visual sequences; cross-tactile reading is blocked. IFM removes only visual-to-tactile edges, retaining other listed design/training factors.
Go to primary source ↓e-actorPDF p. 3, Section III.C, Eqs. (11)–(12) and surrounding text
ACT memory is [z0+s1, q, s2, …, s12, M_current], with frozen current encoders/state projection and gated slots. Fifty queries produce 50 × 8 actions; aggregation uses exp(−0.01 × age). CVAE is disabled, latent zero, and only masked action L1 backpropagates.
Go to primary source ↓e-referencePDF p. 4, Figure 2 and caption
Four chronological columns at equal-segment centers show head RGB and left/right tactile RGB from one successful Put Shelf reference trajectory. The caption limits the figure to task progression and observation format.
Go to primary source ↓e-splitsPDF p. 4, Sections IV.A–B
All tasks evaluate episodes 50–99. Put Shelf trains on 0–39 and develops on 40–49; others train on 0–49. Insert Tube/Lift Can use head and wrist RGB. AFM-family arms share resets, evaluator, preprocessing, normalization, aggregation, horizon, early stopping, cameras and vendor success predicate.
Go to primary source ↓e-controlsPDF p. 4, Section IV.C
Oracle trains/tests with oracle futures and AFM; IFM changes routing. Forced-Zero zeros Oracle-checkpoint slots at evaluation. Zero-Train reads no future HDF and injects literal 12 × 512 zeros before future modules during training and evaluation.
Go to primary source ↓e-trainingPDF p. 4, Section IV.D, first paragraph
Training: task checkpoints, 2,000 steps, two RTX 4090 D GPUs, per-GPU micro-batch 1, accumulation 8, effective batch 16. AdamW ordinary/gate rates 10⁻⁵/10⁻³, decay 10⁻⁴/0; cosine endpoint 10⁻⁶, clip 1.0. Architecture: 4 encoder/7 decoder layers, width 512, 8 heads, dropout 0.1; FP32 without AMP, future-encoding chunks of 8.
Go to primary source ↓e-softwarePDF p. 4, Section IV.D, second paragraph
Ubuntu 20.04.6, Intel i9-14900K, 128 GB RAM, two RTX 4090 D 24 GB GPUs, driver 570.133.20, Python 3.10.12, PyTorch 2.7.0+cu128, torchvision 0.22.0+cu128, Isaac Sim 4.5.0, Isaac Lab 0.41.3.
Go to primary source ↓e-mainPDF p. 4, Table I, complete table and Average row; Section IV.E
Average totals: baseline 52/350 (14.9%), Zero-Train 70/350 (20.0%), IFM 83/350 (23.7%), Forced-Zero 68/350 (19.4%), Oracle 112/350 (32.0%). Bold marks row maxima; underlining marks second-highest distinct scores. No uncertainty intervals are displayed.
Go to primary source ↓e-task-variationPDF p. 4, Table I, Put Shelf, Pull Out Key, Insert Tube and Insert Hole rows; PDF p. 5, Section IV.G
Put Shelf Oracle: 27/50 (54%). Pull Out Key baseline/Zero-Train/IFM/Forced-Zero/Oracle counts: 11/15/12/1/27 of 50. Insert Tube: 7/17/20/21/21. Insert Hole: 1/1/4/3/2. The text discusses these task-specific differences.
Go to primary source ↓e-ablationPDF p. 5, Figure 3, Table II and Sections IV.F–G
AFM-family averages: 20.0%, 23.7%, 19.4%, 32.0%; dashed baseline: 14.9%. The text reports an 8.3-point AFM–IFM gap and a decline under Forced-Zero. Displayed experiments do not vary phase anchors, bandwidth or prediction corruption.
Go to primary source ↓8.5 Primary sources
Disentangling Visuo-Tactile Foresight: Oracle-Guided Interface Discovery for World Action Models ↗
PDF · 3,946 extracted words
Source fingerprint
a52a962354879fb0d325ff2aa0d93ec1caea5b49ea29aa3f8cdf0ebee2e6fbca