VTAM: Video-Tactile-Action Models for Complex Physical Interaction Beyond VLAs
1. Paper overview
In one sentence: VTAM makes tactile deformation part of predictive video modeling and action supervision, with strong task success but unresolved implementation and mechanism claims. e02e03e04e05e07e10e11
| At a glance | What to know |
|---|---|
| Research problem | Author claim A gripper can appear correctly positioned while missing, slipping on, or crushing a chip. The authors argue that semantic visual features underrepresent these contact transitions, and that merely adding tactile inputs can leave a policy dominated by vision. VTAM therefore learns temporal tactile dynamics and supplies contact-sensitive supervision during action learning. e02e08 |
| Core mechanism | Source description Treat tactile images as a third predictive video stream, alongside two camera views, using a shared video VAE and alternating attention. e03 |
| A key reported result | Potato chip pick-and-place: 90% Task success rate. Real xArm6; 20 trials with randomized chip position; 1 Hz inference. Success requires undamaged transport into the plate. Genie Envisioner 0%; π0.5 vision 10%; π0.5 + tactile 5%. The gain over π0.5 vision is 80 percentage points. The abstract calls it 80%; this report uses the unambiguous difference between table values. e01e06e07e12 |
| Reading caution | Reader analysis The experiments cover one robot and three task families, without reported seed variation, confidence intervals or held-out object/environment splits. Table 1 combines wiping conditions. The authors’ generalist framing exceeds what these task-specific results alone establish. e01e06e07e10e11e12 |
Core contributions
- Source description
Treat tactile images as a third predictive video stream, alongside two camera views, using a shared video VAE and alternating attention. e03
Figure 2. Tactile information enters the predictive backbone before conditioning the action expert. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start at the three latent groups on the left: third-person camera, first-person camera and tactile input. Follow them through the video block, then follow the rightward arrow into the action branch’s multi-view cross-attention. The action branch emits future actions, states and forces. The ×28 loops denote depth, not twenty-eight executed robot steps. Cross-check the schematic against Equations 2–3: those equations specify separate intra-view attention followed by attention over concatenated views, while the drawing uses generic self-attention labels around the in-view blocks. Appendix A further specifies that each action layer attends to the corresponding video layer. e03e05e11
What it supports. The architecture places tactile processing upstream of the action expert, making contact features available through the video representation. Appendix A then freezes that representation during Stage II. This supports a predictive-backbone-plus-expert interpretation, while leaving the effect of force supervision on the expert’s use of tactile features to be tested.
Where the evidence stops. The figure says “Past Action + Past State,” but Section 3.3 conditions on current state with zero action/force slots. This input discrepancy is unresolved. The drawing also does not establish a common joint video/action denoising target.
2. Motivation
2.1 The problem and the proposed response
A gripper can appear correctly positioned while missing, slipping on, or crushing a chip. The authors argue that semantic visual features underrepresent these contact transitions, and that merely adding tactile inputs can leave a policy dominated by vision. VTAM therefore learns temporal tactile dynamics and supplies contact-sensitive supervision during action learning. e02e08
2.2 What this reading follows
Picking up a potato chip requires knowing when the fingers have actually made contact and when further closure risks damage. VTAM approaches that problem by adapting a video backbone to forecast two camera streams and a tactile stream, then conditioning an action expert on those representations. A deformation-derived virtual-force target adds supervision during policy training. The paper reports large gains on chips, peeling and wiping, but the evidence deserves two separate readings: what the robot trials establish, and what the architecture explains. Follow the original diagrams, task table and ablation below, keeping uncalibrated force proxies and the documented frozen backbone in view. e02e03e04e05e07e10e11
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Not assigned |
| Architecture | Not assigned |
| Prediction paradigm | Not assigned |
| Quadrant | Not assigned |
This table preserves the labels recorded at reading time. The current major category is WAMs. View the current classification.
3.1 Evidence-based assessment
Insufficient evidence to decide
The recorded taxonomy is entirely unassigned. Architecture evidence supports a predictive visuo-tactile backbone coupled to a distinct conditional action expert, frozen-backbone training in Stage II, and joint action/state/force prediction. It does not establish a single jointly denoised video/action model or an inverse-dynamics decoder operating on a selected future image. World-model features participate in policy generation, while virtual force is auxiliary supervision. Any One Model or quadrant assignment requires editorial definition and resolution of inference details; the catalog snapshot remains unchanged. e03e05e11e13
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Learn a contact representation before learning control
VTAM separates modality adaptation from action supervision. First, a pretrained video VAE represents both camera images and GelSight images, and the video backbone learns future observations through flow matching. Only future frames contribute to the Stage I loss; conditioning images are excluded. The authors’ rationale is that learning tactile dynamics and a control policy simultaneously causes unstable representation shifts. Appendix A instantiates this as GE-base adaptation followed by training a separate action expert against a frozen backbone. The practical distinction matters: calling the first stage “video-only” does not mean tactile data is absent, because Section 3.3 explicitly includes tactile observations in its latent prediction target. The backbone receives task data before freezing; it is not an unchanged generic visual encoder. e03e05e11
Figure 9. Predicted tactile frames qualitatively preserve the changing deformation pattern. Original paper, p. 19 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the columns as the selected temporal sequence and compare the upper and lower rows using the ground-truth/prediction convention of the paper’s forecast visualizations. Inspect both the marker grid and the original yellow arrows: the middle column contains a visibly stronger directional pattern than its neighbors. Appendix C says the arrows visualize estimated force magnitude and direction for interpretation only. The model receives raw tactile tokens, not these arrows as explicit force inputs. Figures 7–8 on the same page provide the accompanying camera forecasts; Figure 5 earlier shows the same paired comparison across all three streams. e04e09e13
What it supports. The selected peeling example shows similar broad deformation patterns in the paired rows, making the tactile prediction target concrete. This is qualitative evidence that the representation can preserve contact-related image structure. It motivates, but does not itself establish, a causal connection between forecast quality and the robot’s successful peeling motions.
Where the evidence stops. The arrows have no calibrated force scale or numerical error summary. Their appearance cannot establish Newton-level accuracy, continuous-time dynamics or success on unseen contacts. The paper supplies no quantitative forecast metric linking this example to the task table.
5.2 Use virtual force as supervision for contact dependence
The virtual-force target compresses a tactile deformation field into three numbers. Spatially averaged flow components represent shear, and average divergence approximates compression relative to a no-contact reference. The action expert predicts this proxy together with future action and state, so reducing its denoising loss requires information associated with contact. That is the intended mechanism; the chip ablation supplies outcome evidence for the objective, not a direct measurement of that mechanism. Reader analysis: under Appendix A’s frozen-backbone recipe, the loss can train the expert to use existing tactile-sensitive features, but cannot update the frozen video weights. This distinction motivates measuring the expert’s response to altered tactile input, rather than taking the authors’ general gradient-balancing language as established. e04e05e10e11
5.3 Connect predictions to the actual robot protocol
The action branch consumes the video representation through cross-attention and jointly denoises action, force and state targets. The physical robot executes actions; a generated tactile frame or predicted force proxy is not itself a control command. The paper reports 1 Hz inference with 30 Hz action execution, so success concerns control under relatively infrequent policy updates. Appendix A gives a 54-action chunk, but does not explain exactly how that chunk is executed or refreshed. There is also an unresolved action-space mismatch: Section 3.3 describes a seven-dimensional end-effector-pose/gripper command, whereas the implementation appendix specifies absolute joint positions. Reader analysis: those missing connections are necessary to reproduce the reported behavior, even though the task success values and their stated criteria can be read unambiguously. e03e05e06e07e11e12
5.4 Training and inference
During training
Stage I adapts GE-base, an LTX-Video backbone, for 50,000 steps. Its flow-matching loss covers future visual and tactile frames, excluding initial conditioning frames. Appendix A calls this video-only pre-training; Section 3.3 explicitly includes the tactile stream. Stage II freezes that backbone and trains the action expert for 20,000 steps, jointly matching action, state and force velocities. e05e11
Appendix A reports four A100 GPUs with 40 GB each, bf16, DeepSpeed ZeRO Stage 2, AdamW, per-GPU batch 16, and learning rates 3e-4 then 5e-5 with 1,000 warmup steps. Inputs are 192×256, with nine video frames and 54 actions per chunk. The dataset has 100 chip, 61 peeling and 105 wiping teleoperated trajectories. e06e11
During inference
The video representation conditions joint action/state/force denoising; the robot executes the action component. Raw tactile tokens supply contact information, while force labels are training targets. Evaluation uses 1 Hz policy inference and 30 Hz action execution. No explicit candidate-rollout search or calibrated force-feedback controller is specified. e03e05e06e07e13
Figure 2 labels the conditioning input as past action plus past state, but Section 3.3 specifies a current-state token with zero-padded action and force dimensions. The exact input reconciliation, inference denoising steps and chunk execution/replanning schedule are not documented. e03e05e11
5.5 Implementation flow
- Encode contact as a view
A pretrained video VAE maps each camera or tactile frame into continuous latents. The stated motivation is preserving local deformation and motion details. Within each of 28 backbone blocks, view-specific attention and attention across concatenated views build predictive multimodal features. Equations 2–3 specify intra-view attention followed by cross-view attention; Figure 2 uses more schematic self-attention labels. e03e11
- Construct deformation supervision
Dense optical flow between a no-contact reference and the current tactile image yields mean horizontal flow, mean vertical flow and mean divergence. These approximate tangential shear and normal compression. They are geometrical proxies without calibrated force units. e04
- Condition an action expert
A separate 28-layer Transformer branch self-attends over action/state tokens and cross-attends to corresponding video-layer hidden states, with diffusion-timestep AdaLN modulation. The formal joint target contains 7 action, 3 force and 16 state dimensions. Section 3.3 defines pose plus gripper-width actions, whereas Appendix A specifies normalized absolute joint positions; their conversion is unresolved. e03e05e11
6. Experiments & results
VTAM adapts a video world model to predict camera and tactile streams, then trains a conditional action expert with an auxiliary deformation-derived force target. It reports large gains in three real robot contact tasks. The strongest evidence is task success and a chip ablation; calibrated force accuracy, broad generalization and the claimed gradient mechanism remain unestablished. Method details below preserve inconsistencies between the formulation and implementation appendix.
6.1 Read the original evidence
Table 1. VTAM leads all three reported task columns under the paper’s real robot protocol. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each column as a distinct task success criterion. Chip success requires an undamaged grasp and placement; peeling counts strips longer than 10 cm; wiping requires complete stain removal without moving the board or supports. These definitions come from Appendix B, not from a shared geometric accuracy metric. The experiment comprises twenty chip trials, twenty peeling motions, and twenty wiping trials at each of two inclinations, for eighty evaluations per model. The Wipe column combines the two settings. All policies are evaluated at 1 Hz inference, distinct from the 30 Hz action execution rate. e06e07e11e12
What it supports. VTAM reports 90% chip, 85% peel and 95% wipe success. The strongest chip baseline is π0.5 vision at 10%, so the difference is 80 percentage points. Adding tactile images to that baseline as another view yields 5%, showing that additional sensory access alone did not improve this particular system.
Where the evidence stops. No uncertainty or per-inclination wiping breakdown is shown. Peeling uses consecutive motions at a fixed position. Appendix A gives different training configurations for π0.5 and VTAM, so these are complete-system comparisons rather than compute-matched tests.
Figure 4, top panel: Chip Pick-and-Place. A missed grasp can leave a visually plausible transport motion with no chip. Original paper, p. 10 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow time from left to right, comparing the upper baseline row with the lower VTAM row. Look at the chip’s location relative to the gripper and plate, rather than using gripper motion alone as evidence of success. The red box marks the authors’ representative baseline failure. The accompanying Section 4.4 describes policies continuing toward the plate after a missed grasp, and describes VTAM using tactile deformation to verify contact. The selected panel’s upper row is labeled only “Baseline,” so it cannot identify a particular Table 1 baseline from its label alone. e08e12
What it supports. The sequence illustrates why a contact-sensitive task can fail despite a plausible overall arm trajectory: transporting the gripper does not establish that the object was grasped. The paper’s separate quantitative chip protocol counts missed grasps, fracture and drops as failures, tying the qualitative illustration to an executed-object outcome.
Where the evidence stops. These are selected still frames, not a full trial record or force trace. Section 4.4’s reported reattempt behavior is not quantified by this crop; the images cannot establish the internal decision rule or recovery frequency.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Potato chip pick-and-place Real xArm6; 20 trials with randomized chip position; 1 Hz inference. Success requires undamaged transport into the plate. | 90% Task success rate | Genie Envisioner 0%; π0.5 vision 10%; π0.5 + tactile 5%. The gain over π0.5 vision is 80 percentage points. The abstract calls it 80%; this report uses the unambiguous difference between table values. e01e06e07e12 |
| Cucumber peeling Twenty consecutive peeling motions at a fixed cutting position; cucumber height decreases. Main comparison uses 1 Hz inference. | 85% (17/20) Trials producing peel strips longer than 10 cm | All three Table 1 baselines: 0%. Supports adaptation during repeated contact at this location. Consecutive motions on changing geometry should not be treated as independently randomized object trials. e07e12 |
| Whiteboard wiping Twenty trials each at 0° and 45° inclination; random stain; at most five wiping motions. Complete removal must leave board and supports undisturbed. | 95% Task success rate, combined Wipe column | Genie Envisioner 2.5%; both π0.5 variants 0%. The single reported wiping rate covers both settings; separate inclination-specific rates are absent. e07e12 |
| Chip component ablation Separate experiment: 10 trials per variant at 1 Hz inference. | Full VTAM 90%; no virtual-force regularization 10%. Chip pick-and-place success rate | Vision-only 0%; downstream-only late fusion 0%. Strong association with the full design in this small experiment. These are not the 20-trial baseline results, and no uncertainty or direct gradient-collapse measurement is supplied. e10 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 2. The chip ablation links success to both upstream tactile integration and force supervision. Original paper, p. 12 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read this table separately from Table 1: its caption specifies ten trials per variant, and Section 4.6 fixes inference at 1 Hz. The first row removes tactile input, the second integrates it only downstream, and the third keeps joint latent modeling while removing virtual-force regularization. The final row is the full design. The most targeted comparison is between the last two rows because it tests the added force objective within the joint-latent setting. The integration descriptions identify architectural variants; “Hierarchical World Model” is a row label, not a measured explanation of the gain. e10e11
What it supports. The reported rate rises from 10% without the force objective to 90% with the complete method. Downstream tactile integration and vision alone each score 0%. This supports testing the force target as a useful training intervention, and it makes naive late fusion an inadequate substitute within this reported chip experiment.
Where the evidence stops. Ten trials per variant leave substantial uncertainty. Success rates do not directly measure modality collapse. Appendix A’s frozen backbone limits any interpretation that the Stage II loss changes video-backbone parameters; expert-level tactile dependence remains a testable explanation.
7. Analysis & limitations
7.1 What the evidence leaves open
The experiments cover one robot and three task families, without reported seed variation, confidence intervals or held-out object/environment splits. Table 1 combines wiping conditions. The authors’ generalist framing exceeds what these task-specific results alone establish. e01e06e07e10e11e12
The regularizer is described as maintaining tactile gradients throughout the backbone, yet Appendix A freezes the video model during action training. It can affect the trainable expert’s use of frozen features; backbone adaptation from that loss is not established under the documented recipe. e03e04e11
Video and tactile forecasts are shown qualitatively, without prediction-error metrics or calibrated force measurements. Yellow arrows are visualization overlays, not explicit force inputs. Selected successful sequences do not measure contact anticipation accuracy. e09e13
7.2 Questions for discussion
- Does correctly aligned force supervision improve tactile dependence beyond a generic auxiliary denoising target?
- How much success depends on temporal tactile prediction rather than recognizing the current contact image?
- Which action representation and conditioning inputs produced the published robot results?
8. Reproducibility audit
8.1 Requirements and known gaps
Required ingredients include GE-base initialization, xArm6, GelSight Mini, two D455 cameras, synchronized demonstrations, no-contact references and normalization statistics. The optical-flow estimator, divergence discretization, exact state layout, data split and software versions are unspecified. Resolve the action-space and conditioning inconsistencies before implementation. e03e04e05e06e11
GE-Act receives the matching two-stage schedule, but π0.5 uses different image resolution, action horizon and fine-tuning duration. Comparisons therefore evaluate complete systems, not compute-matched architectures. Proposed checks should hold demonstrations, backbone, controller frequency and evaluation starts fixed while varying true versus shuffled force supervision and temporal versus static tactile input. e10e11e12
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Does aligned force supervision produce tactile dependence?
Reader-proposed, not performed: start each run from the same Stage I checkpoint and freeze the video backbone as Appendix A specifies. Train otherwise identical action experts with the original virtual-force targets, no force loss, and force targets shuffled between demonstrations while preserving their distribution and loss weight. Use the same chip data, action representation, update rates and randomized evaluation starts, and repeat across training seeds. Report chip success with uncertainty, break/drop/miss counts, and the change in predicted actions when tactile input is replaced by a no-contact frame. Verify that video weights remain unchanged. If aligned targets do not outperform shuffled targets or do not increase appropriate tactile sensitivity, the proposed contact-specific explanation is weakened even if an auxiliary objective helps optimization. e04e05e06e10e11e12
Check 2: Does forecasting tactile futures help beyond current contact recognition?
Reader-proposed, not performed: compare the original Stage I future-tactile target with a control that repeats the current tactile image as the target at every future slot. Keep current inputs, future camera targets, token count, data, initialization and training budget fixed. Then freeze each backbone and train identical action experts with the same correctly aligned force supervision. Evaluate chips and report wiping separately at 0° and 45° using the stated success criteria. Also measure held-out future-tactile reconstruction error against true future frames, including contact changes. Better forecasts accompanied by better execution would support the temporal-modeling rationale; equal control success from the static-target control would weaken the claim that anticipating tactile evolution is necessary in these tasks. e03e05e09e10e11e12e13
8.3 Reading coverage
Visual audit: Inspected the original title/authors/version and Figure 1 on p. 1; motivation and related-work framing on pp. 2–3; Figure 2 and Equations 1–10 on pp. 4–7; setup, all Table 1 cells and evaluation text on pp. 7–8; qualitative behavior text and all Figure 4 panels on pp. 9–10; Figure 5 and prediction discussion on p. 11; Table 2, ablation discussion and conclusion on p. 12; complete training and baseline recipes on p. 17; task criteria and Figure 6 on p. 18; all Appendix C camera/tactile forecasts on pp. 19–20. Cross-checked Figure 2 arrows and labels against Equations 2–3, current-state conditioning and the frozen-backbone recipe; discrepancies are disclosed. All five final original crops were individually viewed, including the higher-DPI narrow tables. Bibliography pp. 13–16 was read as text. Continuous videos, external code and separate supplements remain outside this pass.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 17, 18, 19, 20. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Title, author block, abstract and Figure 1 (p. 1)
- 1. Introduction (p. 2–3)
- 2. Related Works (p. 3–4)
- 3. Method, including 3.1–3.3 and Equations 1–10 (p. 4–7)
- 4. Experiments, including 4.1–4.6 (p. 7–12)
- 5. Conclusion (p. 12)
- References (p. 13–16)
- Appendix A. Training Details (p. 17)
- Appendix B. Experimental Details (p. 18)
- Appendix C. Video Prediction Examples (p. 18–20)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- The inspected artifact is arXiv:2603.23481v1 [cs.RO], dated 24 March 2026. Its title and all twelve authors match the supplied catalog. No other revision was supplied or compared.
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout. This acquisition limitation was addressed by inspecting the original PDF figures, tables and supporting method pages.
- Separate supplemental material availability has not been fully verified.
- All five supplied text chunks were read completely. Code, external project material and continuous videos were not inspected; no experiments were reproduced. Reference entries were read as bibliography, without opening their cited works.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title/author block, arXiv margin, Figure 1 and abstract
The exact catalog title and twelve authors appear on arXiv:2603.23481v1, 24 March 2026. Affiliations are University of Illinois Urbana-Champaign, Stanford University and Shanghai Jiao Tong University. Figure 1 includes language conditioning. The abstract describes the chip improvement as 80%.
Go to primary source ↓e02PDF p. 2, Section 1, first four paragraphs
The motivation concerns visually ambiguous contact and vision-dominated tactile fusion; predictive tactile representation and virtual-force supervision are proposed remedies.
Go to primary source ↓e03PDF p. 4, Figure 2/caption and Section 3; p. 5, Section 3.1, Equations 1–3
The video VAE encodes two camera views and one tactile stream. Equations specify intra-view followed by cross-view attention; Figure 2 links video latents to a separate action branch and labels its input past action plus past state.
Go to primary source ↓e04PDF p. 5, Section 3.2, Equation 4; p. 6, Section 3.2, Equation 5
The three-component force proxy uses spatial means of tactile flow components and divergence relative to a no-contact reference. It is explicitly uncalibrated. The force component participates in joint flow-velocity supervision, with claimed gradient-balancing effects.
Go to primary source ↓e05PDF p. 6, Section 3.3, Equations 6–7; p. 7, continuation and Equations 8–10
Stage I predicts future camera/tactile latents and excludes conditioning frames from its loss. Stage II defines 7-D pose/gripper actions, 3-D force and 16-D state, with current-state conditioning and zero action/force slots. Its loss sums component velocity-matching errors.
Go to primary source ↓e06PDF p. 7, Section 4.1; p. 8, Figure 3/caption and Section 4.2
Setup: xArm6, parallel gripper, GelSight Mini and two Intel RealSense D455 cameras; collection and execution at 30 Hz. Manual demonstrations comprise 100 chip, 105 wiping and 61 peeling trajectories with synchronized RGB, tactile and state information.
Go to primary source ↓e07PDF p. 8, Section 4.3 and Table 1, all model rows and Chip/Peel/Wipe columns
At 1 Hz inference, each model has 80 real-world trials: 20 each for chip, peeling, flat wiping and tilted wiping. VTAM reports 90/85/95%; Genie Envisioner 0/0/2.5%; π0.5 vision 10/0/0%; π0.5 + tactile 5/0/0%. Wiping is a single column; uncertainty is not reported.
Go to primary source ↓e08PDF p. 9, Section 4.4; p. 10, Figure 4 and caption
Selected sequences compare baseline failures with VTAM. The chip account describes missed grasps and contact-dependent lift/reattempt behavior; peeling and wiping accounts describe loss of contact or excessive pressing. Red boxes mark representative baseline failures, without measured force traces.
Go to primary source ↓e09PDF p. 11, Figure 5/caption and Section 4.5
Each camera/tactile pair places ground truth above predictions. The paper presents qualitative consistency and some blur, without numerical prediction-fidelity metrics.
Go to primary source ↓e10PDF p. 12, Table 2, all rows and caption; Section 4.6
Ten chip trials per variant at 1 Hz: no tactile 0%, late-fusion tactile 0%, no virtual-force regularization 10%, full VTAM 90%. The accompanying modality-collapse explanation is not accompanied by direct gradient or attention measurements.
Go to primary source ↓e11PDF p. 17, Appendix A, VTAM World Model, Optimization Details, GE-Act Baseline and π0.5 Policy
Documents 4× A100 40 GB, ZeRO Stage 2/bf16, GE-base LTX-Video (28 layers, 32 heads, width 2048), 50,000/20,000 stages, AdamW and their learning rates. Stage II freezes the video model and trains a 28-layer action branch with corresponding-layer cross-attention and timestep AdaLN. Video/action chunks are 9/54, images 192×256, actions absolute joint positions, loss weights one. GE follows the same schedule; π0.5 task-specific fine-tuning is 10,000 steps with 224×224 images and horizon 50. Software version identifiers are absent.
Go to primary source ↓e12PDF p. 18, Appendix B, Evaluation Protocol and three task paragraphs; Figure 6/caption
Chip evaluation uses 20 randomized positions and rejects missed, broken or dropped chips. Wiping uses 20 trials each at 0°/45°, random stains, up to five motions and no support disturbance. Peeling uses 20 consecutive motions at a fixed position; 17 produce strips longer than 10 cm. This specific peeling setup qualifies the general independent/randomized-trial description.
Go to primary source ↓e13PDF p. 18, Appendix C; p. 19, Figures 7–9/captions; p. 20, Figures 10–12/captions
Qualitative camera and tactile forecast comparisons cover peeling and wiping. Yellow tactile arrows visualize estimated force magnitude/direction only; the model consumes raw tactile tokens without explicit force inputs. No arrow calibration scale or quantitative forecast metric is supplied.
Go to primary source ↓8.5 Primary sources
VTAM: Video-Tactile-Action Models for Complex Physical Interaction Beyond VLAs ↗
PDF · 7,444 extracted words
Source fingerprint
954f7590a181a34fe18a6344c0798fcbf8ef3cf8e7a87cf13d947174b786ed4a