TacPAC: Tactile Prediction and Real-Time Action Correction in World-Action Models for Contact-Rich Manipulation
1. Paper overview
In one sentence: TacPAC turns a cached prediction of expected contact into a reference for fast tactile action correction, while keeping the expensive prediction fixed until the next chunk. e-probleme-basee-cachee-correctore-ablation
| At a glance | What to know |
|---|---|
| Research problem | Source description A plan made before contact cannot incorporate tactile evidence that arrives during execution. Vision also misses local deformation and insertion misalignment. TacPAC asks whether the anticipated contact can become a reference for fast feedback, instead of remaining a prediction that only influences the initial chunk. e-probleme-base |
| Core mechanism | Source description A three-expert Mixture-of-Transformers couples future observation prediction and action generation to a separately trained, single-pass tactile residual corrector. e-basee-correctore-training |
| A key reported result | Five-task contact-rich manipulation: TacPAC: plug 80%, fruit 65%, chip 90%, bottle 40%, card 45%; average 64%. Episode success rate (%) and unweighted five-task average. Flexiv Rizon 4, two InTac S1 sensors, wrist RealSense D405 and external D435i. Each method receives 20 real trials per task with common resets and all attempts counted. Success requires a seated plug/card, all three fruits transferred without visible damage, an intact chip transferred, or a bottle stably upright after release. T-Rex averages 48%, Dream-Tac 45%, π0.5 36%, LingBot-VA 28%, VITaL 22%, ACT 0%. The best baseline per task reaches 65%, 50%, 85%, 30%, and 35%, respectively. TacPAC leads every task against external baselines and gains 16 percentage points over the strongest average. These are executed-robot outcomes under the supplied protocol. e-setupe-protocole-maine-tasks |
| Reading caution | Reader analysis Twenty trials per task produce coarse success estimates; no success confidence intervals or repeated-training variability are reported. Baselines retain different observation modalities and recommended hyperparameters, so the external ranking does not isolate architecture. Generalization beyond the single robot and five tasks is untested. e-protocole-maine-setup |
Core contributions
- Source description
A three-expert Mixture-of-Transformers couples future observation prediction and action generation to a separately trained, single-pass tactile residual corrector. e-basee-correctore-training
- Author claim
The authors argue that matching current touch to the plan's expected contact explains the benefit of cached tactile attention states. The cache ablation supports their mechanism within this task suite. e-ablation
Figure 2. A finished predictive plan supplies the attention states used for later tactile correction. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start at the bottom left: current camera and tactile views enter the video branch, while noise enters action generation. The upper left shows the predicted future views; the center shows the planned action. The cache across the stage boundary stores tactile and action keys/values after an additional clean pass. On the right, a newly arriving tactile image drives the tactile expert, and the leftward Correct arrow indicates a delta added to the planned actions. Stage 1 trains the video/action experts; Stage 2 freezes them and trains the corrector. Section 3.1 specifies asymmetric attention: action queries can read video tokens, while video queries cannot read action tokens. e-basee-cachee-correctore-training
What it supports. The architectural contribution is a reusable connection between prediction and feedback. The corrector receives the expected tactile contact and the plan that depended on it, while avoiding a fresh video-generation pass for every tactile frame. The delta is learned through attention, without explicitly subtracting a predicted tactile image from an observed one.
Where the evidence stops. The shared-attention strip is schematic, not a symmetric attention mask. The right-to-left correction arrow revises the action suffix; it does not refresh the cached prediction or retrain the frozen base during execution.
2. Motivation
2.1 The problem and the proposed response
A plan made before contact cannot incorporate tactile evidence that arrives during execution. Vision also misses local deformation and insertion misalignment. TacPAC asks whether the anticipated contact can become a reference for fast feedback, instead of remaining a prediction that only influences the initial chunk. e-probleme-base
2.2 What this reading follows
A robot can predict how grasping should feel and still fail when actual contact departs from that expectation. TacPAC connects these two moments. Its base model predicts future camera and tactile views while generating a plan; a second-stage tactile expert then reads incoming touch against cached states from that plan and corrects actions still awaiting execution. The reported gain is substantial on five physical tasks, but the most informative comparison is internal: blocking direct access to predicted tactile states lowers average success from 64% to 47%. This reading follows the prediction, the cache, the timing constraint and the experiments that test their connection. e-probleme-basee-cachee-correctore-ablation
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Not assigned |
| Architecture | Not assigned |
| Prediction paradigm | Not assigned |
| Quadrant | Not assigned |
This table preserves the labels recorded at reading time. The current major category is WAMs. View the current classification.
3.1 Evidence-based assessment
Insufficient evidence to decide
The snapshot is unassigned. Architecture evidence establishes a coupled video/action MoT plus a frozen-cache tactile expert, rather than inferring integration from joint training alone. Future observations and actions are generated jointly with asymmetric attention; correction is learned residual prediction, not inverse dynamics. Future prediction remains active at inference. No recorded One Model label or quadrant can be confirmed. e-basee-cachee-correctore-training
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Keep flow time separate from execution time
TacPAC uses two different notions of progress. Flow time tau describes how a noisy latent or action sample is converted into a clean prediction; it does not describe how far the robot has moved. During base generation, the first visual–tactile observation stays clean while future observations and actions are generated together. The asymmetric mask allows action queries to read the predicted observations without feeding noisy action tokens back into video prediction. Physical execution begins after the planned chunk is returned. The model then makes an extra clean pass at tau=0 to obtain the cache. Later tactile frames are indexed by execution offset m, not by a diffusion or flow timestep. This distinction explains how the corrector can run once per incoming frame while the costly generative schedule runs only once per chunk. e-basee-cachee-inference
Figure 3. Feedback is indexed when it is observed, but its residual is applied only where execution has not overtaken it. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the axis as action positions, not flow-matching time. At m, a tactile image starts a correction computation. Blue steps have already executed. Orange steps in the interval from m to m+d continue while that computation runs; they keep their latest committed values. Yellow steps at and after m+d receive the new residual, through the end of the horizon H. Algorithm 1 confirms the writeback inequality i≥m+d. The expert still conditions on the original observation offset m, because that is the pairing used during training. Each new residual modifies the original planned suffix and replaces its previous correction. e-inferencee-correctore-config
What it supports. This design lets the robot continue executing while a new correction is computed. It also prevents a late result from rewriting commands already sent to the arm. The paper's implementation uses 48 actions at 30 Hz, making each complete chunk cover 1.6 seconds of commanded motion.
Where the evidence stops. The colored lengths illustrate the rule, not measured delay frequencies. The source leaves the recent latency-window length and integer conversion details unspecified, so the figure alone cannot establish deadline behavior on another inference device.
5.2 Train a residual for the plan the robot will actually receive
Stage 2 constructs the object the corrector will encounter at deployment: a plan generated by the frozen base and its clean per-layer cache. It does not substitute a perfect demonstrated plan. At a sampled offset, however, the executed prefix is filled with demonstrated actions, because those actions produced the recorded tactile image. The remaining entries keep their predicted values. The tactile expert learns the residual needed to bring that suffix toward demonstrated actions, and four offsets reuse the same plan and prefill. Initializing from the trained action expert provides compatible attention geometry; zero-initializing the output head initially leaves the plan unchanged. At runtime, each new residual again refers to the original plan, while the prefix records actual commands. This replacement rule avoids accumulating previous residuals on the unfinished suffix. e-traininge-confige-corrector
5.3 Separate mechanism evidence from the story told by the diagnostic
The activity plot and the ablation answer different questions. Figure 5 shows when tactile pixels change under the paper's width-based phase definitions. It does not measure whether the video predictor misrepresents those changes or whether they receive too little training gradient. Table 1 instead tests the controller's access to expected contact. The strongest contrast keeps the tactile-predictive base and current tactile input while withholding only direct cached tactile keys. Its 47%-to-64% average change is evidence that this access matters. My interpretation is that the two results form a plausible, incomplete causal argument: activity concentration motivates timely feedback, and the cache ablation supports conditioning that feedback on predictions. Neither establishes calibrated contact prediction, universal benefit on every task, or robustness to badly mismatched caches. e-diagnostice-ablatione-base
5.4 Training and inference
During training
Stage 1 optimizes video and action experts with weighted flow losses; the action term compares clean-action estimates in action space. Stage 2 freezes those experts, VAE, text and proprioceptive encoders. The tactile expert starts from the trained action expert with a zero-initialized output head. e-traininge-config
The frozen base generates its own plan and clean cache. Four sampled offsets share them; each uses a demonstrated prefix and the tactile image it produced, with residual supervision only on the remaining suffix. Stage 2 uses learning rate 10^{-4} and cosine decay. e-traininge-config
During inference
A 48-step chunk contains seven joint-command dimensions relative to the initial joint configuration plus one gripper dimension. Execution runs at 30 Hz for 1.6 seconds. Prefill overlaps the first actions. A frame at offset m triggers correction, but writeback starts at m+d, where d is the recent maximum observed delay in action steps. Replanning waits until the horizon is exhausted. e-inferencee-config
5.5 Implementation flow
- Plan with predictive observations
Camera and tactile views are tiled into one video stream. The observed first frame stays clean; future latents and actions undergo conditional flow matching. Action queries attend to all video and action tokens, while video queries attend only to video tokens. Thus predicted observations condition actions, but noisy actions do not condition video prediction. e-base
- Cache a clean expectation
After ten generation steps, an additional pass at flow time zero stores each layer's tactile and action keys/values. Visual keys are discarded. The cache is fixed for the whole chunk; it represents both expected contact and the plan conditioned on that contact. e-cachee-config
- Read feedback against the plan
The tactile expert encodes current touch with the frozen video tokenizer and tactile patch embedding. Its queries contain the actual executed prefix, original planned suffix and learned execution-status embeddings. They attend to the cache and the expert's own tokens. Instruction conditioning reaches this expert through cached keys; correction is learned without explicit image subtraction. e-corrector
- Execute a residual
Every update replaces the previous residual on eligible steps, adding to the original plan rather than accumulating corrections. Already sent commands remain the input prefix. The arm executes joint/gripper commands, not generated images. e-correctore-inferencee-config
6. Experiments & results
TacPAC predicts future visual and tactile observations while planning an action chunk, then uses incoming touch to revise its unfinished actions against a fixed prediction-and-plan cache. On five physical manipulation tasks, it reports 64% average success versus 48% for T-Rex and 22% for its vision-only ablation. The central evidence is the controlled cache-access ablation, not visual prediction quality alone.
6.1 Read the original evidence
Figure 4. TacPAC leads the external baselines on each task, with performance still uneven across the suite. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use the top legend to track each method across task groups; the shaded rightmost group is the five-task average. TacPAC is the blue bar, reaching 80% for plug insertion, 65% for fruit transfer, 90% for chip transfer, 40% for bottle uprighting and 45% for card insertion. Each task uses 20 trials per method. Appendix D defines success more strictly than simply making contact: all three fruits must be transferred without visible damage, the chip must remain intact, and the released bottle must stay upright. The insertion tasks require the plug or card to be seated in its target interface. e-maine-protocole-tasks
What it supports. The average is 64%, compared with 48% for T-Rex and 45% for Dream-Tac. Gains over the strongest baseline for each task range from five percentage points on chip transfer to fifteen on plug and fruit. Bottle and card remain below half of trials despite leading their external comparisons.
Where the evidence stops. No success confidence intervals or repeated-training variation are shown. All methods share the task protocol, but their observation modalities and recommended training hyperparameters differ; these bars therefore compare systems rather than isolating the tactile-cache mechanism.
Table 2. Cached single-pass correction has a much smaller latency than regenerating a chunk. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the stage labels as separate computational jobs. Chunk generation performs ten flow steps and takes 628.6 ms. Prefill adds one clean pass at 62.6 ms, paid once for a new chunk. Tactile correction reuses that cache and takes 30.4 ms per pass. The adjacent rates correspond to these average latencies; 32.90 Hz belongs to correction, not complete planning. Section 3.3 says execution starts before prefill finishes, and Appendix B sets the command rate at 30 Hz. These details explain why the controller needs both a fast correction path and an explicit rule for commands already in flight. e-efficiencye-inferencee-config
What it supports. The reported 20.7-fold latency reduction compares correction with chunk generation. Execution traces add a more directly relevant observation: only 1.6% of correction calls allow the robot to advance by at least one control step during computation. The system can therefore revise most of the unfinished chunk in the reported setup.
Where the evidence stops. Average latency does not provide tail latency, sensor-to-command latency or end-to-end task throughput. Training uses eight H100 GPUs, but the source does not explicitly identify the inference hardware. The claim of no correction-induced jerks lacks a numerical jerk measure.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Five-task contact-rich manipulation Flexiv Rizon 4, two InTac S1 sensors, wrist RealSense D405 and external D435i. Each method receives 20 real trials per task with common resets and all attempts counted. Success requires a seated plug/card, all three fruits transferred without visible damage, an intact chip transferred, or a bottle stably upright after release. | TacPAC: plug 80%, fruit 65%, chip 90%, bottle 40%, card 45%; average 64%. Episode success rate (%) and unweighted five-task average | T-Rex averages 48%, Dream-Tac 45%, π0.5 36%, LingBot-VA 28%, VITaL 22%, ACT 0%. The best baseline per task reaches 65%, 50%, 85%, 30%, and 35%, respectively. TacPAC leads every task against external baselines and gains 16 percentage points over the strongest average. These are executed-robot outcomes under the supplied protocol. e-setupe-protocole-maine-tasks |
| Tactile-pathway component ablation Same five-task evaluation. The paper reports matched data, backbone, action interface, optimization steps and resets for its prediction comparison. | Full system 64%; without tactile cache 47%; prediction only 37%; correction without tactile prediction 33%; vision only 22%. Average episode success rate (%) | Direct tactile-cache access adds 17 percentage points with base model, corrector, current tactile input and parameter count unchanged. Prediction alone gains 15 of the full 42-point improvement. This isolates direct access to predicted tactile states more cleanly than the no-prediction variant, which also removes tactile input from the base. Bottle success ties at 40% with and without tactile cache. e-ablation |
| Inference-stage efficiency Reported average stage wall-clock times; ten flow steps for generation and one pass each for prefill/correction. | Generation 628.6 ms / 1.59 Hz; prefill 62.6 ms / 15.97 Hz; correction 30.4 ms / 32.90 Hz. Latency (ms) and corresponding rate (Hz) | Correction latency is reported as 20.7 times lower than chunk regeneration. In execution traces, 1.6% of correction calls span at least one robot control step. The speedup concerns one correction, excluding generation and prefill. The authors report no correction-induced jerks, without a quantitative jerk metric. e-efficiency |
| Tactile activity across execution phases Diagnostic subset of 296 training episodes; 284,948 frames remain after codec-related exclusions. Changes are normalized within each episode. | Contact transitions occupy 4.9% of frames but 18.6% of tactile change; their relative activity is 4.1 ± 0.1 (mean ± standard error across episodes). Share of frames/change and activity relative to episode mean | A five-frame-lag check gives 5.0% of frames, 18.5% of change and relative activity 4.00 ± 0.18. Temporal concentration motivates feedback at contact transitions; pixel change does not directly measure prediction error or causal task importance. e-diagnostic |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 1. Direct access to cached tactile predictions accounts for the cleanest internal performance contrast. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each row's pathway flags before comparing its success rate. A dash means there is no corrector; the Cache flag concerns direct tactile-cache access, not removal of every cached action state. Prediction only preserves the tactile-predictive base but removes execution-time correction. Without tactile prediction, the base loses tactile input and targets, while the expert still receives current touch. Without tactile cache keeps tactile-conditioned planning and correction but blocks direct attention to predicted tactile keys. The final two rows are therefore the key comparison. Blue shading identifies best cells and green second-best cells; notice the shared best bottle result. e-ablatione-overview-notation
What it supports. Restoring direct tactile-cache access changes average success from 47% to 64%, with the base model, current input, corrector and parameter count held constant. Prediction alone reaches 37%, versus the vision-only 22%, so its 15-point improvement is about a third of the full 42-point gain. Prediction and feedback are complementary in this experiment.
Where the evidence stops. Removing prediction also changes base inputs and targets, so that contrast is less selective. Bottle success ties at 40% with and without tactile cache. Figure 1 labels the full gain '42%'; Table 1 establishes 42 percentage points, not a 42% relative increase.
Figure 5. Brief contact transitions carry a disproportionate share of tactile-image change. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Compare the orange contact-transition bars between panels. Panel (a) normalizes tactile-image change by each episode's mean, shown as the dashed line at one. Panel (b) reports what fraction of frames belongs to each phase. Thus a tall orange bar above and a short one below identify concentrated activity, not high task success. Appendix E computes change from mean absolute pixel differences across the two tactile views, removes differences into and out of codec keyframes, and normalizes within episodes before averaging. Phase labels come from measured gripper width rather than tactile images. The appendix uses episodes as the unit for means and standard errors. e-diagnostic
What it supports. The contact-transition phase has the highest relative tactile activity on every task while occupying little of each episode. The accompanying text reports 4.9% of frames containing 18.6% of total change. A five-frame-lag check retains a similar concentration, supporting the motivation to react during execution rather than waiting for the next plan.
Where the evidence stops. Pixel change is not prediction error, task relevance or measured force. Compression attenuates small changes. Width-derived phases miss object–environment contact and include some early contact in approach. The figure motivates the correction mechanism but does not establish why a future-prediction objective underweights useful events.
7. Analysis & limitations
7.1 What the evidence leaves open
Twenty trials per task produce coarse success estimates; no success confidence intervals or repeated-training variability are reported. Baselines retain different observation modalities and recommended hyperparameters, so the external ranking does not isolate architecture. Generalization beyond the single robot and five tasks is untested. e-protocole-maine-setup
Compression suppresses small tactile changes. Width-derived phases miss object–environment contact and label some early contact as approach. The diagnostic therefore supports temporal activity concentration, not a complete contact annotation. Appendix rollout images and activity curves come from different episodes. e-diagnostice-tasks
The tactile expert trains with demonstrated prefixes, whereas deployment prefixes include model commands. This leaves an exposure mismatch despite training on the base model's own planned suffix. Effects of stale caches and long correction delays are not isolated. e-traininge-inferencee-ablation
7.2 Questions for discussion
- Does correctly paired predicted contact outperform equally sized but mismatched tactile caches when action states and compute are held fixed?
- How does correction benefit change when tactile latency approaches several control steps?
8. Reproducibility audit
8.1 Requirements and known gaps
The paper specifies eight NVIDIA H100 GPUs on one node, global batch 64 and 10 epochs for both stages. It names Python ≥3.10, transformers 4.57.0, accelerate 1.12.0 and DeepSpeed 0.16.9; these are source-reported requirements, not verified installation results. e-config
Rebuilding requires synchronized visual/tactile demonstrations, relative-joint action normalization, the exact frozen base and a command queue enforcing suffix writeback. Missing details include total training-set size/split, Stage-1 loss weights and initialization provenance, numerical Dream-Tac gating adjustments, inference hardware and delay-window length. The diagnostic sample is not a stated full training-set size. e-traininge-confige-protocole-diagnostice-inference
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Test whether the cached expectation must match the current plan
Reader-proposed, not performed: reproduce the full and blocked-tactile-cache variants with the same frozen base, demonstrations, training budget, tactile images and action-cache access. Add a controlled evaluation condition that supplies tactile cache states from a different episode of the same task while leaving the active plan and action states unchanged. Use identical reset distributions on plug insertion and fragile-chip transfer, multiple training seeds, and report success intervals and failure types. Correctly paired tactile states should outperform both blocking and mismatching if plan-specific anticipated contact is essential. Equal performance would weaken that interpretation; a mismatch-induced drop alone would also need the blocked-cache control to distinguish lost information from disruptive inputs. e-cachee-correctore-ablatione-protocole-tasks
Check 2: Measure when correction latency erases the benefit
Reader-proposed, not performed: hold trained weights, action horizon, sensors and 30 Hz command rate fixed, then inject controlled correction delays ranging from zero to several action steps. Keep observation offset m tied to capture time and enforce writeback only from m+d; compare with the prediction-only controller under the same task resets. Log capture, completion and command timestamps, fraction of the suffix still editable, latency tails, and task success, including between-chunk planning time. The falsifiable expectation is that correction gains shrink as fewer relevant actions remain editable. Any write to an already executed command would falsify the intended queue implementation rather than the learned prediction mechanism. e-inferencee-efficiencye-confige-ablatione-protocol
8.3 Reading coverage
Visual audit: Inspected the title/author/version block, Figures 1–11, Tables 1–3, Algorithm 1, method and training equations, implementation details, evaluation accounting, task criteria and diagnostic protocol on the declared PDF pages. All six final original crops were individually viewed; labels, axes, legends and table headers remain readable. Figure 2's correction arrow was checked against Eqs. (2)–(3), and Figure 3's boundaries against Algorithm 1 and the inference text. The shared-attention schematic does not display the asymmetric mask described on p. 3. Figure 1's '42%' annotation is interpreted as 42 percentage points using Table 1. Appendix rollout frames and activity traces are explicitly from separate episodes. References on pp. 8–9 were read in the complete text but were not rendered. No external videos, code, datasets or separate supplements were inspected.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 10, 11, 12, 13, 14, 15. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Title, author block and Abstract (p. 1)
- 1 Introduction (pp. 1–2)
- 2 Related work (pp. 2–3)
- 3 Method: 3.1 Problem formulation, 3.2 Prediction-grounded tactile correction, 3.3 Training and inference (pp. 3–5)
- 4 Experiments: 4.1 Experimental setup, 4.2 Results and analysis, 4.3 Ablation study (pp. 5–7)
- 5 Conclusion (p. 7)
- References (pp. 8–9)
- Appendix A Notation and B Implementation details (pp. 10–11)
- Appendix C Evaluation details and D Task descriptions (pp. 11–14)
- Appendix E Diagnostic for Q2 (p. 15)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Only the supplied arXiv:2609.05266v1, dated 4 September 2026, was reviewed. Its title and all five authors match the catalog; no other revision or edition was supplied or compared.
- Text extraction does not reconstruct figure images; this limitation was addressed by inspecting the retained PDF's figures, equations and table layouts.
- Separate supplemental material availability has not been fully verified.
- The linked code, datasets and external references were not opened; no implementation or experiments were reproduced.
- Reference pages 8–9 were read as text; visual inspection covered pages 1–7 and 10–15.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e-identityPDF p. 1, title/author/affiliation block and arXiv margin
Observed title matches the supplied title. Authors are Zipei Ma, Xiaofei Wei, Junzhe Jiang, Shunlin Lu and Li Zhang; affiliations are School of Data Science, Fudan University; Shanghai Innovation Institute; NeoteAI. Margin identifies arXiv:2609.05266v1 [cs.RO], 4 Sep 2026.
Go to primary source ↓e-problemPDF p. 1, Abstract and Section 1
The paper motivates contact-sensitive feedback by visual ambiguity and the mismatch between pre-execution prediction and tactile evidence arriving during execution.
Go to primary source ↓e-basePDF p. 3, Section 3.1, Eq. (1) and asymmetric-attention paragraph
Visual/tactile views share a latent stream. Conditional flow uses a clean first frame; action queries attend video/actions, but video queries attend only video. Flow time is distinct from physical time.
Go to primary source ↓e-cachePDF p. 3, Section 3.2, 'Caching the prediction and the plan'
A separate clean pass at tau=0 constructs per-layer tactile/action keys and values; visual keys are excluded, and the cache is reused unchanged for the chunk.
Go to primary source ↓e-correctorPDF p. 3, Section 3.2, 'Tactile expert', Eq. (2); p. 4, Figure 2 and 'Technical advantage'
The expert reads current touch, status-tagged executed/planned actions and the cache. Instruction enters through cached keys. Residuals replace earlier suffix corrections relative to the original plan; no explicit predicted-minus-observed image comparison is computed.
Go to primary source ↓e-trainingPDF pp. 4–5, Section 3.3, Algorithm 1 training lines 1–5 and Eq. (3)
Stage 1 trains video/action flow objectives. Stage 2 freezes the base, generates its own plans and prefills, inserts demonstrated prefixes at sampled offsets, and supervises corrected unexecuted actions.
Go to primary source ↓e-inferencePDF p. 4, Algorithm 1 inference lines 1–5; p. 5, Figure 3 and Section 3.3 'Inference'
Robot execution begins before prefill finishes. Corrections conditioned at m are committed only for i≥m+d, with d determined from the largest recent delay. The base replans after exhausting the chunk.
Go to primary source ↓e-setupPDF p. 5, Section 4.1 'Robot and observation–action interface'; p. 10, Figure 6
Experiments use a Flexiv Rizon 4, two InTac S1 sensors, wrist D405 and third-person D435i; visual baselines omit tactile views.
Go to primary source ↓e-protocolPDF p. 5, Section 4.1 'Baselines' and 'Evaluation protocol'; p. 11, Appendix C
Twenty attempts per method/task share resets and binary criteria; failures are neither excluded nor repeated based on outcome. Baselines retain official modalities and recommended hyperparameters; Dream-Tac gating is adjusted without numerical values.
Go to primary source ↓e-mainPDF p. 6, Figure 4, all task groups and Average; Section 4.2
TacPAC's plug/fruit/chip/bottle/card rates are 80/65/90/40/45%, averaging 64%. Baseline averages are T-Rex 48%, Dream-Tac 45%, π0.5 36%, LingBot-VA 28%, VITaL 22%, ACT 0%; no success uncertainty is plotted.
Go to primary source ↓e-ablationPDF p. 6, Table 1, all rows; p. 7, Section 4.3 Q2–Q3
Average success is 22/37/33/47/64% for vision-only/prediction-only/no-prediction/no-tactile-cache/full. The no-cache variant blocks direct tactile keys while preserving the base, current touch, expert and parameter count. Both no-cache and full attain 40% on bottle.
Go to primary source ↓e-efficiencyPDF p. 7, Table 2, all rows; Section 4.3 Q4
Generation/prefill/correction take 628.6/62.6/30.4 ms with corresponding rates 1.59/15.97/32.90 Hz. Authors report 20.7-fold lower correction latency, 1.6% of calls spanning a control step, and no correction-induced jerks.
Go to primary source ↓e-configPDF p. 10, Appendix B, all four implementation paragraphs; p. 11, Table 3
Experts have 30 layers and 24 heads of dimension 128; hidden/feed-forward widths are 3072/14336 for video and 1024/4096 for action/tactile. A 3072-to-1024 bridge and zero-initialized tactile head are specified. H=48 at 30 Hz; ten flow steps; VAE and future-frame strides 4 each; three future latent frames; visual/tactile views 256×256/128×128. K=4, Stage-2 learning rate 10^-4 with cosine schedule; both stages use eight H100s, batch 64, ten epochs. Software requirements are listed, but inference hardware and Stage-1 initialization provenance are not.
Go to primary source ↓e-diagnosticPDF p. 6, Section 4.3 Q2; p. 7, Figure 5; p. 15, Appendix E and Eq. (4)
Mean absolute inter-frame tactile change is measured on 296 episodes, retaining 284,948 frames. Transitions occupy 4.9% of frames and 18.6% of change; activity is 4.1±0.1 times episode mean. Five-frame lag gives 5.0%, 18.5%, 4.00±0.18. Codec boundary differences are excluded; normalization and standard errors use episodes. Width phases use 10 mm/s movement, 3 mm minimum travel, 15-frame minimum hold and ±8-frame boundary windows.
Go to primary source ↓e-tasksPDF p. 11, Appendix D introduction; pp. 12–14, Figures 7–11 and captions
Captions give starting states and success criteria for plug seating, three-fruit damage-free transfer, intact-chip transfer, stable released bottle and card seating. Each figure's visual/tactile rollout is separate from the median-profile training episode used for the activity curve; their frames are not aligned.
Go to primary source ↓e-overview-notationPDF p. 2, Figure 1 right-hand success chart; p. 6, Table 1, Vision only and TacPAC rows
Figure 1 annotates the 22%-to-64% success difference as '42%'. Table 1 establishes a 42-percentage-point absolute increase; the annotation should not be read as a relative percent improvement.
Go to primary source ↓8.5 Primary sources
TacPAC: Tactile Prediction and Real-Time Action Correction in World-Action Models for Contact-Rich Manipulation ↗
PDF · 7,291 extracted words
Source fingerprint
bd5bba44569f4706ff03c33c0be4cdde42d4e7e8d86fee07c0aefbefac10e17b