Vtla: Vision-tactile-language-action model with preference learning for insertion manipulation
1. Paper overview
In one sentence: VTLA combines wrist vision and tactile histories in a token-generating policy, then improves OOD action accuracy through preference learning whose physical-control benefit remains unisolated. e02e04e05e06e08e09e10e12e16
| At a glance | What to know |
|---|---|
| Research problem | Source description Peg insertion combines global alignment with local contact uncertainty. VTLA asks whether a pretrained vision-language model can combine wrist vision and fingertip touch to correct planar displacement and yaw. Its setting is repeated insertion of a grasped peg, trained in simulation and tested physically. e02e03e16 |
| Core mechanism | Author claim VGTE places vision after tactile inputs and encodes tactile histories as image grids. The authors motivate these choices through recency bias and difficulty understanding short contact sequences. e04 |
| A key reported result | Real-world insertion at 0.6 mm clearance: VTLA triangle: 95/3.94; pentagon: 100/1.85. Success (%) / reported mean attempt steps. Simulation-trained policies on the physical robot; 20 trials per task. VLA: 90/4.06 and 100/2.3; TLA: 30/2.00 and 40/1.88. Triangle improves over VLA by one success in 20. Pentagon success ties with fewer reported steps. Uncertainty and the step-averaging denominator are unspecified. e11e12 |
| Reading caution | Author claim The authors identify possible loss of tactile-specific features through the off-the-shelf vision encoder, insufficient tactile-language alignment and limited deep multimodal fusion. e14 |
Core contributions
- Author claim
VGTE places vision after tactile inputs and encodes tactile histories as image grids. The authors motivate these choices through recency bias and difficulty understanding short contact sequences. e04
- Author claim
Ground-truth-distance preferences supply what the authors call regression-like supervision while preserving token-based action generation. e06
Figure 3. One action decoder, first imitated and then preference-tuned. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the blue upper panel as supervised training. Tactile, vision and language blocks feed the same QwenLM decoder, whose output tokens are compared with the ground-truth correction. The lower green panel starts from this SFT policy and samples alternative actions for the same observations. Follow the curved arrows to the two candidates, then the dashed ground-truth comparison: the checked candidate is closer in each displayed coordinate than the crossed candidate. This agrees with Section 3.3’s chosen/rejected rule. The frozen reference policy used by the loss is defined in the text, although it is not separately drawn here. e04e05e06e16
What it supports. The lower stage adds a relative preference between plausible action texts instead of changing the output into a continuous regression head. Its supervision still comes from the simulated reference action. At deployment the trained policy generates corrections from observations; ground-truth comparison belongs to training.
Where the evidence stops. Equation (2) prints β on only the chosen log-ratio, with no clarification of the implemented placement or value. Figure 3 does not resolve this. Section 3.2 explicitly freezes the vision encoder and adapter despite its earlier broad fine-tuning wording.
2. Motivation
2.1 The problem and the proposed response
Peg insertion combines global alignment with local contact uncertainty. VTLA asks whether a pretrained vision-language model can combine wrist vision and fingertip touch to correct planar displacement and yaw. Its setting is repeated insertion of a grasped peg, trained in simulation and tested physically. e02e03e16
2.2 What this reading follows
Inserting a peg requires both locating the hole and interpreting what happens when the surfaces touch. VTLA brings these signals into Qwen2-VL: short fingertip histories become image grids, a wrist image supplies global context, and a language instruction requests a corrective action. Training first imitates simulated action labels, then prefers sampled responses closer to those labels. This reading follows that information flow into three distinct tests: offline action prediction, simulated insertion and physical execution. The results support useful transfer in a matched insertion setup, while the ablations and qualitative cases leave several proposed mechanisms only partly tested. e02e04e05e06e08e09e10e12e16
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | VLA |
| Architecture | One Model |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
3.1 Evidence-based assessment
Supports the recorded classification
The shared QwenLM decoder receives tactile, visual and language tokens and directly generates actions, supporting VLA, One Model and autoregressive/tactile/post-training subcategories. The frozen DPO reference is a training counterpart, not another inference planner. No future-state/action joint prediction or inverse dynamics is specified, supporting the recorded not-applicable prediction paradigm and quadrant. e04e05e06e16
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Turn a collision into language-model context
VTLA’s action input is a contact event, not just a scene description. The robot approaches an already selected hole, descends, and records both the wrist view and the changing fingertip signals. Each sensor contributes four frames, which are arranged into a 2×2 image representation before encoding. Section 3.2 then places the visual input after the tactile inputs, motivated by recency bias: vision should remain influential near action prediction. A pretrained visual encoder and modality adapter convert these images into tokens for a shared language decoder. The explicit training statement freezes those front-end modules. The resulting output is a textual correction in x, y and yaw; physical motion and the next contact observation come from the robot loop. e03e04e05e16
5.2 Learn a preference without changing the action interface
The first stage teaches the decoder to imitate a reference action string with next-token prediction. The authors argue that this classification objective poorly reflects distances between continuous controls. Their second stage samples different responses to the same input and labels the response nearer the ground-truth action as preferred. DPO compares trainable-policy likelihoods with those of a frozen SFT reference. The important interpretive point is that distance informs the preference labels; the deployed output remains token-generated action text. A claim of regression-like supervision therefore does not establish calibrated continuous regression. Table 4 supports an OOD prediction gain, but does not test execution with and without DPO. The printed equation’s one-sided β placement also requires clarification before faithful implementation. e05e06e07e10
5.3 Track the evidence from prediction to execution
The paper’s three evaluation layers answer different questions. Table 1 compares predicted actions with stored labels, and its GCR asks whether all correction directions are correct. Table 3 executes repeated insertions in simulation, allowing fresh observations after unsuccessful contact. Table 7 tests a physical robot after simulation-only policy training. As a reader interpretation, this separation explains why moderate offline GCR can coexist with high episode success: an episode permits repeated attempts, although the paper does not quantify that relationship. Physical pentagon success ties VLA while the reported attempt count improves; triangle differs by only one success in twenty trials. The poor-lighting snapshots add a useful case study, but need repeated, controlled testing before they establish a robust lighting advantage. e03e07e08e09e11e12e16e17
5.4 Training and inference
During training
SFT uses Qwen2-VL 7B and LlamaFactory: learning rate 5×10⁻⁴, batch 64, 10 epochs. Despite an earlier sentence saying the entire model is fine-tuned, the explicit parameter statement freezes the vision encoder and modality adapter, tuning only the language model. e05e07
DPO starts from SFT: learning rate 5×10⁻⁶, batch 32, 3 epochs; the reference model is frozen. Compute hardware, software versions, numerical β, sampling configurations and preference-distance weighting are unspecified. e06e07
During inference
The robot grasps and approaches the hole, descends, captures vision at contact and records four tactile frames per fingertip during collision. VTLA generates a correction; observation and action repeat until success or failure, with a 15-attempt cap. The robot executes actions and supplies fresh feedback. e03e16
DPO operates during training. The described execution loop neither compares actions against ground truth nor queries a reward model. There is no explicit dynamics predictor or inference-time world-model planner. e05e06e16
5.5 Implementation flow
- Generate insertion supervision
Isaac Gym and a custom visuotactile simulator produce 28,000 samples across five shapes and 0.6–2.0 mm clearances. A randomized strategy supplies observations and labels, with randomization of physical properties, task configurations and images. Exact training/validation allocation is not given. e03e08e15
- Encode contact and vision
Tactile histories become image-like grids. A pretrained vision encoder and modality adapter produce tactile and visual tokens, which join text tokens in a shared language decoder. Vision follows touch in the prompt. No future-state rollout is described. e03e04e05e16
- Prefer closer actions
The SFT model generates alternative responses to each training input. Closer responses become chosen examples, farther responses rejected examples. Trainable and frozen reference policies start from SFT. Two generation configurations produce 2,400 preference examples; the smaller ablation uses 1,000. e06e07
6. Experiments & results
VTLA converts wrist vision, fingertip contact histories and a peg-specific instruction into corrective insertion actions through Qwen2-VL. Simulation instruction tuning is followed by preferences ranking sampled actions against a reference action. Evidence supports improved OOD action prediction and physical insertion in a matched setup, with small trial counts and incomplete implementation details (e04–e12).
6.1 Read the original evidence
Table 1. Action-direction accuracy is distinct from completed insertion. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the ID and OOD column groups rather than comparing every bold number together. Each group contains GCR, where higher is better, and three L1 errors, where lower is better. Section 4.1 defines GCR as getting all three action directions correct; it is not a task-completion rate. The rows compare diffusion policy with visual and tactile observations, vision-only VLA, touch-only TLA, and VTLA. The evaluation uses 6,000 ID and 4,000 OOD samples. Read the rotational column independently from translation: the table does not state units that would justify combining their raw magnitudes. e07e08
What it supports. VTLA reaches 47.3% ID and 31.2% OOD GCR, versus VLA’s 46.1% and 29.5%. TLA is much weaker on this offline metric. However, VLA’s ID rotational L1 is lower, 0.886 versus 0.904, so the multimodal policy is not best on every reported measure.
Where the evidence stops. The paper leaves the precise correctness rule, action scaling and dataset split construction incomplete. These offline rows cannot establish insertion success, and the table supplies no uncertainty estimates for the differences.
Table 3. Shape-dependent gains appear when actions are actually executed. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each shape as its own task, pairing Suc with Step. The setup fixes clearance at 0.6 mm and reports 50 trials per shape. Square, triangle and hexagon fall under the ID header, while pentagon and round fall under OOD. Moving across a row therefore changes geometry while preserving the reported clearance. Compare success before interpreting steps: a policy with many failures can still display a small mean step value if failed episodes are handled differently. The source does not define that averaging denominator, so Step should be kept as the reported quantity rather than reconstructed as total effort. e03e07e09
What it supports. On the OOD pentagon, VTLA reports 92% success and 3.97 steps, versus VLA’s 82% and 4.41. The round shape reverses the success ordering: VLA reaches 94%, VTLA 92%. Square VTLA also takes more reported steps than VLA despite higher success. Benefits therefore depend on geometry and metric.
Where the evidence stops. Fifty trials per task and no reported intervals limit precision. The paper does not fully specify how five collected shapes map into training and held-out samples. OOD labels here should not be generalized to arbitrary novel objects.
Table 7. Physical transfer is demonstrated, with a modest advantage over vision alone. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the triangle and pentagon as separate physical tests, each with 20 trials according to Section 4.4. For triangle, VTLA’s 95% versus VLA’s 90% corresponds to one additional successful episode. For pentagon, both achieve 100%, making the reported attempt count the distinguishing quantity. TLA performs much worse in success despite small Step entries. This is why the two columns must remain paired: the source does not explain whether unsuccessful episodes enter the step average. The policies were trained on simulated data, while these results come from the UR3 setup with wrist vision and fingertip sensing. e07e11e12e16
What it supports. VTLA achieves 95%/3.94 steps on triangle and 100%/1.85 on pentagon. VLA reports 90%/4.06 and 100%/2.3, respectively. These observations establish successful physical execution and a reported pentagon efficiency improvement, but the physical advantage over vision alone is smaller than the gap to touch-only TLA.
Where the evidence stops. No uncertainty intervals are supplied, and 20 trials give coarse five-percentage-point resolution. Table 7 does not ablate DPO, so its physical gains cannot be attributed specifically to preference learning.
Figure 6. A contact-feedback example when wrist vision is degraded. Original paper, p. 15 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow the task-scene column downward from Step 1 to Step 3. The arrows indicate successive insertion attempts; the adjacent wrist images and tactile grids show the observations available during the first two attempts. The final photograph and check mark record successful insertion after three steps. Appendix C compares this sequence with Figure 7 on the same PDF page: a VLA sequence ends in failure at step 15, and its caption states matching initial peg-hole misalignment. Both captions specify 0.6 mm clearance. The comparison is an illustrated episode, not a measured curve of performance against illumination. e17
What it supports. The snapshots are consistent with the authors’ explanation that touch can help when the hole is difficult to locate visually. They document a successful VTLA episode and a matched VLA failure, giving a concrete motivation for testing complementary sensing under controlled visual degradation.
Where the evidence stops. No illumination level, aggregate trial count or randomization protocol is provided for this comparison. The image cannot establish a population success-rate advantage or reveal how much the decoder relied on tactile versus residual visual information.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Offline insertion-action prediction 6,000 ID and 4,000 OOD evaluation samples; Table 1. | VTLA GCR: 47.3% ID, 31.2% OOD; OOD L1 (x,y,rz): (0.305,0.324,1.136). GCR (%) and componentwise L1 error | VLA GCR: 46.1%/29.5%; TLA: 15.3%/14.4%; DP: 7.8%/8.5% (ID/OOD). GCR requires all three action directions to be correct, not successful insertion. VLA has lower ID rotational L1 (0.886 versus 0.904), so VTLA does not dominate every metric. e07e08 |
| Simulated insertion at 0.6 mm clearance 50 trials per shape; square/triangle/hexagon labeled ID, pentagon/round OOD. | VTLA: square 90/5.91, triangle 88/4.53, hexagon 90/4.68, pentagon 92/3.97, round 92/4.74. Success (%) / reported mean attempt steps | Pentagon: VLA 82/4.41, TLA 80/4.60, DP 26/5.61. Round VLA success is 94%, above VTLA. Gains depend on shape; neither uniform superiority nor universally fewer attempts is established. e09 |
| DPO ablation on offline action prediction No DPO, 1,000 preferences, or 2,400 preferences; reported ID/OOD evaluation. | OOD: 27.0 → 31.4 → 31.2; ID: 47.5 → 47.5 → 47.3. GCR (%) | The 1k variant gains 4.4 percentage points OOD, approximately 16.3% relatively. The gain concentrates OOD. More preference examples do not improve GCR here; this is not an executed-success DPO ablation. e07e10 |
| Real-world insertion at 0.6 mm clearance Simulation-trained policies on the physical robot; 20 trials per task. | VTLA triangle: 95/3.94; pentagon: 100/1.85. Success (%) / reported mean attempt steps | VLA: 90/4.06 and 100/2.3; TLA: 30/2.00 and 40/1.88. Triangle improves over VLA by one success in 20. Pentagon success ties with fewer reported steps. Uncertainty and the step-averaging denominator are unspecified. e11e12 |
| Real-world clearance and shape transfer VTLA only; 20 trials per task; Tables 5–6. | Square at 1.6/1.0/0.6 mm: 100/100/95%; at 0.6 mm, triangle/hexagon: 95%, pentagon/round: 100%. Success (%) | Tables 5–6 contain no baseline rows. These are observed successes in a matched setup, not a guarantee on arbitrary unseen geometries. e11 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 4. Preference tuning primarily improves the reported OOD action metric. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow the three rows downward within the OOD GCR column. Without DPO it reads 27.0; the 1k preference variant reaches 31.4; the row named DPO-2k reaches 31.2. Section 4.1 clarifies that the last name denotes 2,400 examples, not exactly 2,000. Now inspect ID GCR: 47.5, 47.5 and 47.3. The accompanying L1 columns generally improve with preferences, but the success of the central generalization claim rests mainly on the OOD comparison. Keep this table separate from the executed insertion tables: its rows measure action predictions against stored labels. e07e10
What it supports. The 1k preference condition gains 4.4 percentage points OOD, approximately 16.3% relative to 27.0%. ID GCR does not improve. Increasing to 2,400 examples offers no additional GCR gain in this comparison. The authors’ explanation that candidate diversity could matter more than count remains a hypothesis.
Where the evidence stops. This ablation does not isolate token order, tactile temporal encoding or DPO’s effect on executed insertion success. Candidate-generation settings and repeat-run variability are not reported, limiting a causal account of the plateau.
7. Analysis & limitations
7.1 What the evidence leaves open
The authors identify possible loss of tactile-specific features through the off-the-shelf vision encoder, insufficient tactile-language alignment and limited deep multimodal fusion. e14
No isolated token-order or temporal-grid ablation is reported. The poor-lighting appendix shows VTLA succeeding in three steps versus a matched VLA failure at 15, without aggregate trials or illumination measurements. e04e10e17
Figure 4 Case 2 labels VTLA success at four steps, but adjoining prose says three. The episode count remains unresolved. The prose claim of success above 95% also overstates Table 5’s 95% entry. e11e13
7.2 Questions for discussion
- Would tactile temporal ordering matter with prompt order, token budget and DPO held fixed? (e04, e10)
- Would DPO’s offline OOD gain survive matched physical trials and failure-inclusive attempt accounting? (e10, e12)
8. Reproducibility audit
8.1 Requirements and known gaps
Reproduction needs the simulator, label convention and exact split, plus UR3, Robotiq 2F-85, RealSense D405 and two GelStereo 2.0 sensors. Initial physical offsets span ±2.5 mm in x/y and ±5° yaw. ROS tactile acquisition is printed as “20 PFS,” requiring clarification. e03e07e16
Table 8 specifies friction U(0.2,0.7), Young’s modulus U(1.0e5,5.0e5) Pa and image translation U(−10,10) pixels, among other ranges. Jitter/noise/blur strengths remain unspecified. Resolve action scaling, GCR correctness, failure inclusion in mean steps, DPO β placement/value and sampling settings before interpreting replication discrepancies. e06e07e15
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Separate temporal information from input ordering
Reader-proposed check: train a factorial comparison of tactile-before-vision versus vision-before-tactile prompts, crossed with chronological versus randomly shuffled tactile grids. Keep observations, grid dimensions, token budget, SFT initialization, optimizer updates and DPO condition fixed; add repeated-single-frame grids as a control for temporal information. Evaluate the same ID/OOD samples and matched simulated initial poses, reporting GCR, componentwise L1, success and attempts with failures explicitly accounted for. A consistent chronological-grid benefit would support temporal information use; an ordering-only effect would support the prompting explanation. If neither survives repeated seeds, the separate VGTE mechanism claims would remain unsupported. No such check was run for this report. e03e04e05e07e09e10
Check 2: Test preference diversity against count and execution
Reader-proposed check: start from one SFT checkpoint and compare no DPO, 1,000 preferences, and 2,400 preferences from the paper’s two-configuration recipe with 2,400 preferences sampled more diversely. Hold optimizer-update count and source-input coverage constant where possible; document the distance rule, action scaling, β value and exact interpretation of Equation (2). Repeat seeds, then evaluate both the offline OOD split and identical 0.6 mm insertion trials. Diversity should improve over the equal-sized original recipe if the authors’ plateau explanation is correct. If GCR rises without an execution gain, that would limit the claimed control benefit of preference tuning. These are proposed experiments, not reproduced results. e06e07e09e10e12
8.3 Reading coverage
Visual audit: Visually inspected the title/authors/version on p. 1; all seven figures and all eight tables; method, objectives, training and evaluation details on pp. 3–8; limitations on p. 9; and every appendix page, including randomization, hardware, contact capture and poor-lighting cases. Six final original crops were inspected at their output resolution. Figure 3’s preference check/cross agrees with its example and Section 3.3; the printed β placement remains unresolved. Figure 4 Case 2 conflicts with its nearby prose (four versus three VTLA steps), as recorded in the base report. All five text chunks were read, including reference-only pp. 10–12, which were not visually inspected. No external video, code, separate supplement or journal edition was inspected.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 13, 14, 15. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Title, authors, abstract and keywords (PDF p. 1)
- 1 Introduction (pp. 1–2)
- 2 Related Works (p. 3)
- 3.1 Data Collection (pp. 3–4)
- 3.2 Instruction Tuning with Vision-Guided Temporally Enhanced Tokens (pp. 4–5)
- 3.3 Preference Learning (pp. 5–6)
- 4.1 Experiment Setup (p. 6)
- 4.2 Comparison with Baseline Methods (pp. 6–7)
- 4.3 Ablation Study (p. 7)
- 4.4 Real-world Robotic Insertion (pp. 7–8)
- 5 Conclusion and Limitations (pp. 8–9)
- References (pp. 9–12)
- Appendix A Domain Randomization on Simulated Dataset (PDF pp. 13–14; appendix pp. 1–2)
- Appendix B Real-World Robot Setup for Insertion Task (PDF p. 14; appendix p. 2)
- Appendix C VTLA vs. VLA under Poor Lighting Condition (PDF pp. 14–15; appendix pp. 2–3)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- The extraction limitation was addressed by visually inspecting all seven figures and all eight tables in the supplied PDF. All five supplied text chunks were read individually, including Appendix C, absent from the manifest section list.
- Identity/version note: the title and all six authors agree with the catalog apart from title capitalization. The reviewed source is arXiv:2505.09577v1, dated 14 May 2025 on PDF p. 1. The catalog identifies a 2026 Biomimetic Intelligence and Robotics article, DOI 10.1016/j.birob.2026.100333. That journal edition was not supplied; its equivalence and revisions were not verified.
- No external project pages, videos, code or separate supplements were inspected; no experiments were reproduced. Reference entries were read as part of this paper, not as independently reviewed publications.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title/author block and arXiv margin
Title: VTLA: Vision-Tactile-Language-Action Model with Preference Learning for Insertion Manipulation. Authors: Chaofan Zhang; Peng Hao; Xiaoge Cao; Xiaoshuai Hao; Shaowei Cui; Shuo Wang. Margin: arXiv:2505.09577v1, 14 May 2025. Affiliations: State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences; Samsung R&D Institute China–Beijing; Beijing Academy of Artificial Intelligence.
Go to primary source ↓e02PDF pp. 1–2, Abstract, Figure 1 and Section 1
The policy combines vision, tactile observations and language for contact-rich insertion, using simulation training and physical validation. The introduction motivates global vision and local contact information.
Go to primary source ↓e03PDF pp. 3–4, Section 3.1 and Figure 2
Isaac Gym and a custom visuotactile simulator collect 28,000 samples with left/right tactile histories, wrist image and action label, across five shapes and 0.6–2.0 mm clearances. Histories use 2×2 grids. Collision triggers retraction/retry; reaching insertion depth without collision succeeds; the cap is 15 attempts. Task/action labels follow TLA, without their full construction being restated.
Go to primary source ↓e04PDF p. 4, Section 3.2, VGTE paragraphs and dataset prompt
Vision follows tactile inputs to exploit claimed recency bias. Tactile sequences become image-like representations processed by a ViT. The prompt specifies peg type and desired corrective action.
Go to primary source ↓e05PDF pp. 4–5, Section 3.2, Figure 3 Stage 1 and Equation (1)
A pretrained vision encoder and modality adapter encode visual/tactile observations for a language model with text tokens. NTP trains action generation. The final parameter statement freezes the encoder and adapter, tuning only the language model. The loss mask is unspecified.
Go to primary source ↓e06PDF pp. 5–6, Section 3.3, Figure 3 Stage 2, Equation (2) and symbol definitions
SFT-generated actions are ranked by ground-truth proximity; closer responses are chosen. Trainable and frozen reference policies initialize from SFT. Printed β precedes only the chosen log-ratio. The text defines β>0 and sigmoid but gives no numeric β, distance weighting or clarification of placement. Figure 3’s check/cross correctly prefer the candidate closer to the displayed ground truth.
Go to primary source ↓e07PDF p. 6, Section 4.1, methods, metrics and implementation details
Two generation configurations yield 2,400 preferences; DPO variants use 1,000 or 2,400. GCR counts actions correct in all x/y/rz directions; L1 is componentwise. SFT: Qwen2-VL 7B, LlamaFactory, rate 5×10⁻⁴, batch 64, 10 epochs. DPO: rate 5×10⁻⁶, batch 32, 3 epochs. Exact correctness rule, L1 units, software versions, compute hardware and sampling settings are absent.
Go to primary source ↓e08PDF p. 6, Section 4.2, Table 1, all rows and ID/OOD columns
Evaluation uses 6k ID and 4k OOD samples. VTLA/VLA/TLA/DP GCR: 47.3/46.1/15.3/7.8% ID; 31.2/29.5/14.4/8.5% OOD. VTLA OOD L1: 0.305/0.324/1.136; ID rotational L1: 0.904 versus VLA 0.886. Exact split construction is not stated.
Go to primary source ↓e09PDF pp. 6–7, Section 4.2, Tables 2–3 and 50-trial protocol
Table 3 uses 0.6 mm clearance, with square/triangle/hexagon ID and pentagon/round OOD; each task has 50 trials. VTLA success/steps: 90/5.91, 88/4.53, 90/4.68, 92/3.97, 92/4.74. Pentagon VLA/TLA/DP: 82/4.41, 80/4.60, 26/5.61. Round VLA: 94/4.81. Table 2 compares square clearances. Neither table gives uncertainty or the precise mean-step denominator.
Go to primary source ↓e10PDF p. 7, Section 4.3 and Table 4, no-DPO/DPO-1k/DPO-2k rows
GCR ID: 47.5/47.5/47.3%; OOD: 27.0/31.4/31.2%. DPO-1k OOD L1: 0.305/0.324/1.137 versus 0.349/0.367/1.223 without DPO. The authors hypothesize diversity could matter more than preference count. This offline ablation does not isolate VGTE ordering/grid effects or DPO’s executed-success effect.
Go to primary source ↓e11PDF pp. 7–8, Section 4.4, Tables 5–6 and 20-trial protocol
Real tasks use 20 trials and simulation-trained VTLA. Square success/steps at 1.6/1.0/0.6 mm: 100/1.60, 100/1.95, 95/4.31. At 0.6 mm: square 95/4.31, triangle 95/3.94, hexagon 95/3.52, pentagon 100/1.85, round 100/5.2. Prose says above 95% despite 95% entries.
Go to primary source ↓e12PDF p. 8, Table 7, triangle/pentagon Suc and Step columns; p. 7, Section 4.4 trial count
At 0.6 mm physical clearance, VLA triangle/pentagon success and steps: 90/4.06 and 100/2.3; TLA: 30/2.00 and 40/1.88; VTLA: 95/3.94 and 100/1.85. With 20 trials, five percentage points equal one success. Intervals and explicit failure treatment in mean steps are absent.
Go to primary source ↓e13PDF p. 8, Figure 4 Case 2 and preceding Section 4.4 paragraph
Text assigns three steps to VTLA in Case 2, while the graphic labels success at four steps. TLA fails at 15. The discrepancy prevents a definitive VTLA episode count.
Go to primary source ↓e14PDF p. 9, Limitations
The visual encoder may lose tactile-specific features; the authors call for dedicated tactile-language alignment and improved vision/touch fusion in LLMs.
Go to primary source ↓e15PDF pp. 13–14 (appendix pp. 1–2), Appendix A, Table 8 and explanations
Randomization covers physics, in-hand offsets/contact depth, tactile color, lighting and visual transforms. Uniform ranges include Young’s modulus 1.0e5–5.0e5 Pa, friction 0.2–0.7 and image translation −10 to 10 pixels. Jitter/noise/blur strengths and the prose’s gripper-width range are unspecified.
Go to primary source ↓e16PDF p. 14 (appendix p. 2), Appendix B and Figure 5; PDF p. 6, robot setup
The platform uses UR3, Robotiq 2F-85, RealSense D405 and two GelStereo 2.0 sensors. Initial x/y misalignment: −2.5 to 2.5 mm; yaw: −5° to 5°. Wrist capture occurs at contact; each sensor captures four frames during collision. ROS acquisition is printed as 20 PFS. Observation/instruction generates an action and the loop repeats; latency and action scaling are unspecified.
Go to primary source ↓e17PDF pp. 14–15 (appendix pp. 2–3), Appendix C, Figures 6–7 and captions
At 0.6 mm clearance under poor lighting, Figure 6 shows VTLA success at three steps. Figure 7 shows VLA failure at 15 with matching initial misalignment stated in its caption. No aggregate trials, illumination levels or uncertainty are reported.
Go to primary source ↓8.5 Primary sources
VTLA: Vision-Tactile-Language-Action Model with Preference Learning for Insertion Manipulation ↗
PDF · 7,351 extracted words
Source fingerprint
a3f2197448da871a2d5903939ac2975e6b1419ffdcc00b28b74a2e05d4815dbc