CLAP: Contrastive Latent Action Pretraining for Learning Vision-Language-Action Models from Human Videos
1. Paper overview
In one sentence: CLAP uses a robot-grounded token vocabulary to learn from human videos, then adapts a faster continuous controller while regularizing its semantic backbone. e02e03e04e05e06e11e12
| At a glance | What to know |
|---|---|
| Research problem | Source description Human videos contain varied objects and semantics but lack robot action labels. Robot trajectories supply executable control yet cover fewer situations. The authors argue that reconstruction-only latent actions entangle manipulation with background and appearance changes. CLAP anchors video-derived latents to a vocabulary learned from physical trajectories. Human videos still have task annotations; “unlabeled” refers to missing action and kinematic supervision. e02e03e04 |
| Core mechanism | |
| A key reported result | Human-video transfer on OOD Make Bouquets: CLAP-RF: 10% without target human data → 45% with it. Task success (%). Astribot; familiar task with unseen flower combinations; three additional human-video settings, 100 episodes each; 20 evaluation attempts. UniVLA: 5% → 10%; pi0.5 teleoperation-only: 10%. A 35-percentage-point CLAP gain. Wrong-target failures fall from eight to zero, but nine placement failures remain. This supports semantic transfer within a known task. e11e12 |
| Reading caution | Author claim The authors identify new-task transfer without robot grounding, hand/gripper morphological ambiguity and multistage training complexity as unresolved. Flower demonstrators intentionally mimic parallel-jaw grippers. Results therefore do not establish transfer of arbitrary human dexterity. e21e11 |
Core contributions
Figure 3. Robot actions define the vocabulary that visual transitions must learn to use. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start at the lower left: robot action chunks pass through the action encoder, establishing physical reference latents and a decoder back to trajectories. Above, the inverse-dynamics encoder processes frame pairs and separates action-relevant from nuisance information. The matrix compares action-relevant visual latents with reference latents; its diagonal marks matched pairs. For human examples, Algorithm 2 supplies an EMA-teacher reference rather than a measured action. The right panel separates autoregressive CLAP-NTP from CLAP-RF, whose action expert receives backbone keys and values. The drawn future images are schematic: Section IV-B specifies reconstruction of DINOv3 features. The five displayed codebook entries are illustrative, not the selected vocabulary size. e03e04e05e06e15
What it supports. The architecture creates a route from video transitions to decodable robot action tokens. Human videos can therefore supervise the NTP policy without continuous action labels. The RF expert receives that knowledge through its conditioning backbone; it still learns its continuous trajectory objective from robot demonstrations.
Where the evidence stops. The overview omits the EMA teacher, stop-gradient operations and separate environment codebook shown in Algorithm 2. Follow that algorithm for update directions. Future-feature reconstruction is auxiliary training, not demonstrated inference-time rollout planning.
2. Motivation
2.1 The problem and the proposed response
Human videos contain varied objects and semantics but lack robot action labels. Robot trajectories supply executable control yet cover fewer situations. The authors argue that reconstruction-only latent actions entangle manipulation with background and appearance changes. CLAP anchors video-derived latents to a vocabulary learned from physical trajectories. Human videos still have task annotations; “unlabeled” refers to missing action and kinematic supervision. e02e03e04
2.2 What this reading follows
A human video can show which flower to select without specifying a robot’s end-effector trajectory. CLAP bridges that gap in stages: learn a decodable vocabulary from robot actions, align visual transitions to it, and train a language-conditioned token policy on both sources. A separate rectified-flow expert then learns continuous control from robot trajectories, with Knowledge Matching constraining the token backbone. This distinction explains both the promise and the boundary of the results. Human videos substantially improve unfamiliar object combinations within known tasks, but correct selection still leaves grasping and placement problems. The figures below follow that path from representation learning to executed behavior. e02e03e04e05e06e11e12
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | VLA |
| Architecture | Dual-system |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
3.1 Evidence-based assessment
Supports the recorded classification
The recorded VLA and Dual-system classification fits CLAP-RF’s explicit VLM backbone plus separate action expert. The family also contains an autoregressive token policy and latent-action pretraining. Inverse dynamics creates training labels; future-feature reconstruction is auxiliary. Neither establishes joint future-observation/action generation at deployment, so prediction paradigm and quadrant remain Not applicable. e03e04e05e06
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Learn what a token can execute before asking video to predict it
Start with the supervisory asymmetry: a robot demonstration supplies an action chunk, whereas a human video supplies only a visual transition and task annotation. Act-VAE first learns to compress robot chunks into discrete tokens that its decoder can reconstruct as trajectories. VD-VAE then learns to predict within that established vocabulary. Its action-relevant branch shares the frozen robot codebook, while the nuisance branch has a separate learnable codebook. Both branches help reconstruct future visual features, but the nuisance branch is penalized to discourage unnecessary usage. Contrastive alignment supplies the physical anchor that reconstruction alone lacks. The selected rate-distortion configuration balances reconstruction fidelity against how many tokens the downstream language model must predict; good reconstruction is a prerequisite for useful tokens, not itself a task-success measurement. e02e03e04e15
Figure 4. A frozen token-policy reference constrains adaptation while the action expert learns robot control. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the snowflake and flame symbols before following the arrows. The reference model is frozen; the policy model and action expert are trainable. Gray blocks denote observations and instructions, blue blocks the discrete subtask/action sequence, and green blocks continuous actions. The key-value connection supplies the expert with the policy backbone’s internal representations. Although the KL connector has arrowheads at both ends, Eq. (7) specifies asymmetric reverse KL from the trainable token distribution to the frozen reference. Algorithm 4 places this comparison in robot-only RF post-training, after the reference has inherited the optional adaptation on robot and human tokens. e06e08e22
What it supports. KM couples two different objectives: continuous trajectory learning for the RF expert and preservation of the token policy’s distribution. The reference is not an additional action generator to be averaged with the expert. Human-video information reaches this stage through the adapted model and reference, rather than fabricated continuous human action labels.
Where the evidence stops. The double-headed graphic must not be interpreted as symmetric KL or two trainable models. The paper leaves the KM coefficient unspecified, and the diagram alone cannot establish whether semantic preservation causes the reported OOD gains.
5.2 Trace human supervision through the two policy stages
The human-video teacher produces a target for latent alignment, not a measured robot trajectory. After alignment, VD-VAE pseudo tokens allow human demonstrations to enter the same next-token objective as robot demonstrations. CLAP-NTP learns an observation- and instruction-conditioned sequence containing subtask text and action codes. For target adaptation, the paper first trains in this token space using both data sources. It then freezes a reference, copies the adapted backbone into a trainable policy branch, and attaches the RF expert. Only robot trajectories supervise the continuous flow objective. KM compares the trainable policy’s token distributions with the frozen reference, which already contains human-video information. Thus the bridge into continuous control is the adapted representation and its regularization; the method does not manufacture continuous ground-truth robot actions for every human clip. e04e05e06e08
5.3 Separate choosing correctly from executing successfully
The most informative reading pairs Figure 8’s success gains with Table III’s failure categories. After adding human videos, OOD bouquet wrong-target failures disappear in the reported trials, yet placement remains difficult. My interpretation is that the intervention primarily strengthens object selection and instruction grounding in this experiment; it does not establish equivalent transfer of fine motor skill. That reading agrees with the authors’ gripper-mimicking collection protocol and their warning about human-hand morphology. Table VI asks a different question: how to adapt the continuous controller while retaining useful pretrained representations. Its improvement with the full feature-and-KM configuration motivates a controlled reproduction, but does not identify every contribution independently. Keep semantic selection, continuous execution and adaptation stability as separate measurable outcomes when evaluating the broader transfer claim. e11e12e16e19e21
5.4 Training and inference
During training
The mixture includes curated AgiBot (about 100,000 episodes/1,500 hours), Astribot (27,000/50 hours), DROID (about 76,000/350 hours) and 90 hours of Ego4D. DROID is right-arm data with zero-padded left-arm channels. These are reported dataset quantities, not an independently audited training split. e07
Qwen3VL-4B NTP trains for 150,000 steps, peaking at 5e-5 learning rate. Target adaptation uses three NTP epochs at 2e-5 and five RF/KM epochs, with action-expert rate 1e-4. Human videos influence RF through adapted representations and the reference, not direct continuous-action supervision. e08e06
During inference
NTP autoregressively emits tokens that the action decoder can translate to control. RF predicts continuous chunks using backbone features. VD-VAE’s future frame and forward reconstruction are training machinery. The source does not specify deployed replanning cadence, executed chunk fraction or RF integration settings. e03e05e06e09e22
5.5 Implementation flow
- Build the physical vocabulary
Act-VAE compresses action chunks through Transformer register tokens and nearest-codebook lookup. Its decoder reconstructs trajectories; codebook and commitment terms complement reconstruction. Visual alignment freezes this action encoder and vocabulary. The selected rate-distortion setting uses 16 tokens and K=256, with a configuration ambiguity noted below. e03e15e22
- Ground visual transitions
A frozen DINOv3 backbone encodes current and future frames. VD-VAE divides transition features into action-relevant and nuisance streams. Only the former uses the frozen action codebook; a learnable environment codebook handles the latter. A forward decoder reconstructs clean future features, while L1 penalizes nuisance-latent usage. e04
- Choose contrastive targets by source
Robot transitions align to stop-gradient Act-VAE encodings of actual action chunks. Human transitions align to stop-gradient outputs of an EMA teacher on clean frames; the online branch sees augmentation. SigLIP uses other batch samples as negatives. Human positives are learned references, not measured robot trajectories. e04
- Adapt an expert from a token policy
NTP predicts subtask and action-token sequences with next-token cross-entropy. Target adaptation first permits robot and human tokens, then freezes an NTP reference. Robot-only RF/KM training updates a policy copy and a DiT action expert that cross-attends to its key-value cache. e05e06
6. Experiments & results
CLAP turns human visual transitions into robot-grounded action tokens, then trains a language-conditioned policy on robot demonstrations and pseudo-labeled videos. Its optional rectified-flow controller replaces autoregressive decoding with faster continuous action chunks. The clearest transfer evidence concerns object selection within familiar tasks: human videos improve OOD bouquet success, but placement failures remain. Future-feature reconstruction supports representation training; this is not an inference-time world simulator.
6.1 Read the original evidence
Table I. Continuous control improves some manipulation stages while the token policy retains an OOD-placement advantage. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow each task’s subcolumns before reading Task Mean. Pick and Place distinguishes successful pickup from placement; Pack the Doll separates placing the doll from closing the lid. C-1 and C-2 are the two evaluated flower combinations. These are physical Astribot trials with the chassis and torso locked, using head and wrist RGB observations. The test budgets differ: 20 episodes per PnP setting, 10 each for doll packing and folding, and 10 per bouquet combination. Compare the two highlighted CLAP rows to study the controller tradeoff, then use the baseline rows as broader references. The printed Task Mean is an author-reported aggregate with unspecified weighting. e09e10
What it supports. CLAP-RF reports the highest Task Mean at 62.7%, versus 60.0% for pi0.5. Relative to NTP, RF increases doll P&P from 80% to 90% and folding from 30% to 40%, but OOD placement decreases from 85% to 70%. The controller advantage is therefore task-dependent.
Where the evidence stops. These small trial sets have no reported confidence intervals. Baselines differ in pretraining and architecture, so their gaps do not isolate contrastive alignment. Preserve the reported aggregate without assuming it is an unweighted average of all displayed stage columns.
Figure 8. Target human videos improve object generalization, especially unfamiliar bouquet combinations. Original paper, p. 10 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the two bar charts independently. The left chart is Make Bouquets (OOD), with CLAP-RF bars rising from 10 to 45; the right is PnP (OOD), rising from 70 to 85. Arrows mark fine-tuning with human video, while pi0.5 is a teleoperation-only reference. UniVLA also receives human-video training through its own tokenizer and action-head procedure. The demonstration images show the collection constraint: an egocentric viewpoint close to the robot’s head and deliberate gripper-like hand movements. Section V-A5 supplies the adaptation data: three additional bouquet settings with 100 videos each and 20 human trajectories per OOD PnP object. e11e12e21
What it supports. CLAP-RF gains 35 percentage points on OOD Bouquets and 15 on OOD PnP; UniVLA gains five points on each. Table III sharpens the interpretation: bouquet wrong-target failures drop from eight to zero across 20 attempts, while nine placement failures remain. Better semantic selection does not eliminate motor errors.
Where the evidence stops. The chart axes have different ranges, and the PnP axis starts at 40. Compare printed values rather than apparent bar heights. The tasks are familiar, the added demonstrations are curated, and these results do not demonstrate acquisition of entirely new tasks.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Human-video transfer on OOD Make Bouquets Astribot; familiar task with unseen flower combinations; three additional human-video settings, 100 episodes each; 20 evaluation attempts. | CLAP-RF: 10% without target human data → 45% with it. Task success (%) | UniVLA: 5% → 10%; pi0.5 teleoperation-only: 10%. A 35-percentage-point CLAP gain. Wrong-target failures fall from eight to zero, but nine placement failures remain. This supports semantic transfer within a known task. e11e12 |
| LIBERO generalist manipulation One policy across four suites; 30,000 tuning steps, batch 128; 500 trials per suite. | Spatial 98.6; Object 99.2; Goal 98.0; Long 93.0; reported average 97.2. Success (%) | Generalist X-VLA 98.1 average; pi0.5 and FLOWER 96.9. Competitive, not the best average. Specialist results use a different training protocol. No uncertainty accompanies these point estimates. e14 |
| Astribot original real-world task set Five task categories; PnP uses 20 trials, doll/folding 10, bouquets 10 per combination. | CLAP-RF 62.7; CLAP-NTP 58.7. Reported Task Mean (%) | pi0.5 60.0; pi0 54.0. RF improves folding from 30% to 40%, but OOD placement drops from NTP’s 85% to 70%. Aggregation weights are unspecified; printed means are retained without reinterpretation. e09e10 |
| CLAP-NTP alignment and human-data ablation Table VII’s separate ID/OOD PnP and Bouquets ablation setting. | Full 60.0; no pseudo positives 58.8; no contrastive loss 53.8; no human data 47.5. Average success (%) | OOD Bouquets: full 35%, no contrastive 20%, no human data 5%. Human data and alignment matter in this setting. Do not merge these figures with RF transfer results or Table I; per-ablation uncertainty is absent. e17 |
| Single-image policy inference latency LIBERO inputs; one 224×224 image on one NVIDIA H100. | NTP 382.0 ms/9.2 G; RF 70.1 ms/12.1 G. Latency (ms); memory (source unit G) | pi0: 45.8 ms/7.6 G. RF lowers NTP latency at higher memory cost, but is slower than pi0. This is not measured three-camera robot-loop latency. e20e09 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table VI. Feature selection and regularized adaptation both affect the continuous controller. Original paper, p. 12 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read this as a post-training study, distinct from Table IV’s comparison with other policy families. The first block changes which VLM features the RF head uses; the second compares fine-tuning strategies. The last row combines low- and middle-level features with KM. LIBERO has separate Spatial, Object, Goal and Long columns plus an average; the final columns are real-world PnP under ID and OOD objects. Section IV-C explains why feature selection matters: the action expert is shallower than the VLM and cannot consume every layer. The direct-fine-tuning row is a useful reference for whether successful adaptation also retains OOD performance. e08e16e22
What it supports. The full configuration reports 97.2% LIBERO average and 70% PnP OOD, versus 96.2% and 60% for direct VLM fine-tuning. High-level-only features score 93.4% LIBERO average, below low-level-only features at 96.1%. These results motivate examining both representation choice and the adaptation constraint.
Where the evidence stops. The table does not provide a complete feature-selection-by-KM factorial experiment or exact layer indices. The KI comparison is evidence about this implementation and setup, not a general ranking of regularization methods. Uncertainty is not reported.
Table VII. Within this NTP study, removing human data and contrastive alignment especially harms OOD bouquets. Original paper, p. 13 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. The upper block modifies a UniVLA-style baseline’s backbone and data recipe; it should not be read as an isolated alignment experiment. The lower block starts from full CLAP-NTP and removes pseudo-positive labels, the contrastive objective or human data, then tests a Prismatic-7B backbone. Read the OOD Bouquets column alongside the average: it reveals where the main semantic-generalization loss occurs. Parenthesized changes in the lower block are relative to full CLAP-NTP, while upper-block changes refer to the UniVLA baseline. This study has its own values and setting; it is separate from Table I’s deployment comparison and Figure 8’s RF target-adaptation experiment. e17e09e11
What it supports. Full CLAP-NTP averages 60.0%, falling to 53.8% without contrastive learning and 47.5% without human data. OOD Bouquets decreases from 35% to 20% and 5%, respectively. Removing pseudo positives changes the average only to 58.8%. The stronger evidence concerns alignment and human data together, not the teacher alone.
Where the evidence stops. The upper data-recipe row prints 45.8 although its four displayed scores average 45.0; the crop preserves that inconsistency. No repeated-seed uncertainty is supplied. Do not interpret these component removals as a complete account of interactions or transfer to arbitrary human motion.
7. Analysis & limitations
7.1 What the evidence leaves open
The authors identify new-task transfer without robot grounding, hand/gripper morphological ambiguity and multistage training complexity as unresolved. Flower demonstrators intentionally mimic parallel-jaw grippers. Results therefore do not establish transfer of arbitrary human dexterity. e21e11
Small real-world trial counts and absent uncertainty limit fine comparisons. Manually chosen t-SNE examples and Astribot-only trajectory projections cannot prove universal human-to-robot isomorphism. Table VII prints 45.8 for a data-recipe row whose four values average 45.0; that aggregate remains unresolved. e09e18e17
7.2 Questions for discussion
- How much transfer survives human demonstrations that do not imitate gripper morphology?
- Does KM still help with identical feature layers, data and training budget?
8. Reproducibility audit
8.1 Requirements and known gaps
Reconstruction needs curated mixtures, pretrained visual/VLM backbones, action normalization and tokenizers. Table IX specifies 32-action chunks and eight codes per arm, but its [256,128] codebook entry is not reconciled with Table V’s K=256. EMA rate, KM weight, exact feature layers and RF solver settings need clarification. e07e08e15e22
Table IX estimates Act-VAE/VD-VAE/NTP training at approximately 190/380/3,800 hours on one A100 80G; these are estimates, not observed distributed training times. A focused proposed check would fix backbone and data while removing contrastive alignment, then separately vary KM with identical RF features and training budget. e22e17e16
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Hold data and backbone fixed while testing alignment under visual distractors
Reader-proposed, not run: train full CLAP-NTP and a zero-contrastive-loss variant with identical Act-VAE vocabulary, initialization, robot/human mixture, update budget and several paired seeds. Preserve VD-VAE reconstruction and nuisance regularization in both. Evaluate the same held-out object combinations under original and changed backgrounds, logging wrong-target, pick and place failures separately. If alignment removes action-irrelevant sensitivity, its advantage should persist or grow under background perturbation, especially in wrong-target errors. If only aggregate success changes while selection and robustness do not, the proposed semantic-disentanglement explanation is weakened. Document any choices for source-omitted hyperparameters. e04e12e13e17e22
Check 2: Isolate Knowledge Matching from RF feature selection
Reader-proposed, not run: start both RF branches from the same adapted NTP checkpoint and keep low-plus-middle feature indices, robot trajectories, optimizer schedule and expert size identical. Compare zero KM weight against a validation-selected positive weight, using paired seeds and a fixed held-out ID/OOD split. Record task success, the three failure categories, and token-distribution divergence from the frozen reference. The claimed retention mechanism predicts lower divergence and better OOD selection without a material loss of ID control. If benefits disappear with feature layers held constant, Table VI’s combined configuration cannot be credited to KM alone. Report chosen layer indices and KM weight because the PDF does not supply them. e06e08e12e16e22
8.3 Reading coverage
Visual audit: Visually inspected the title/author page, all method and training pages, Figures 1–11, Tables I–IX, Algorithms 1–4 and the limitations on PDF pages 1–15. Six final original crops were individually inspected; Table VI is the post-training crop. Figure 3 was cross-checked against Algorithm 2 and feature-space reconstruction; Figure 4’s two-headed connector was checked against the frozen reference and reverse-KL direction in Eq. (7). Figure 8 task labels were verified directly against the rendered page and results prose. Table VII’s data-recipe aggregate discrepancy is retained. All eight supplied text chunks were read individually, including reference-only pages 16–17. Those two pages were not visually inspected; no scientific claim here depends on their contents. No external supplement, code or live demonstration video was inspected.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15. Appendix coverage: not present.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Title, author credits and abstract (p. 1)
- I. Introduction (pp. 1–2)
- II. Related Work, A–C (pp. 2–3)
- III. Methodology, A–E; Algorithms 1–4 and Eqs. (1)–(7) (pp. 3–7)
- IV. Model Training Details, A–C; Eq. (8) (pp. 6–8)
- V. Evaluation, A–D (pp. 8–14)
- VI. Conclusion and limitations (pp. 14–15)
- Table IX model/training configuration (p. 15)
- References (pp. 15–17)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- The supplied artifact is arXiv:2601.04061v2, dated 14 June 2026. Title and all eight authors match the catalog; earlier revisions were not supplied or compared.
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout. This acquisition limitation was addressed by visually inspecting PDF pages 1–15 and every final crop.
- Separate supplemental material availability has not been fully verified.
- No separately labeled appendix is present in the supplied 17-page PDF; Table IX is on p. 15 alongside references.
- Code and external links were not inspected; no experiments were reproduced. Reference-only pages 16–17 were read as text, not visually inspected.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title, author line, affiliation footnote and arXiv margin
The title and eight authors match the catalog: Chubin Zhang, Jianan Wang, Zifeng Gao, Yue Su, Tianru Dai, Cai Zhou, Jiwen Lu and Yansong Tang. The artifact is arXiv:2601.04061v2, 14 June 2026. Affiliations are Tsinghua University, Astribot, the University of Hong Kong and Massachusetts Institute of Technology.
Go to primary source ↓e02PDF pp. 2–3, Introduction; Section III-A, Eq. (1)
CLAP addresses visual entanglement and the gap between action-labeled robot trajectories and instruction-annotated human videos lacking kinematic labels. Each arm contributes position, Euler orientation and gripper aperture to a 14-dimensional action.
Go to primary source ↓e03PDF p. 4, Figure 3(a), Section III-C1, Algorithm 1 and Eq. (2)
Act-VAE uses Transformer encoder/decoder modules and register tokens, nearest-neighbor quantization, action reconstruction, codebook and commitment losses. The action encoder and codebook are frozen for subsequent alignment.
Go to primary source ↓e04PDF p. 5, Section III-C2, Algorithm 2 and Eqs. (3)–(4); p. 8, Section IV-B
VD-VAE splits action-relevant and nuisance latents, quantizes them with frozen action and learnable environment codebooks, and reconstructs clean future DINOv3 features. Robot positives use stop-gradient Act-VAE encodings; human positives use a clean-input EMA teacher while the online encoder sees augmentation. SigLIP contrasts positives with other batch samples. L1 penalizes nuisance latents.
Go to primary source ↓e05PDF p. 5, Algorithm 3; p. 6, Section III-D and Eq. (5)
CLAP-NTP autoregressively predicts subtask and action-token sequences from observations and instructions, trained with robot action tokens or human pseudo action tokens.
Go to primary source ↓e06PDF p. 4, Figure 3(b); p. 6, Figure 4, Section III-E and Eqs. (6)–(7); p. 7, Algorithm 4
The DiT rectified-flow expert cross-attends to the NTP key-value cache. All-data NTP adaptation precedes freezing the reference and robot-only RF/KM training of the policy branch and action expert. Reverse KL is policy-to-reference over discrete-token distributions; the reference is frozen despite the figure’s two-headed comparison arrow.
Go to primary source ↓e07PDF p. 7, Section IV-A, items 1–4 and Figure 5
Data include the curated AgiBot subset (about 100,000 episodes, 1,500 hours), Astribot S1 (27,000 episodes, about 50 hours at 30 Hz), DROID (about 76,000 trajectories, 350 hours), and 90 hours of Ego4D from the UniVLA subset. DROID occupies right-arm channels with zero-padded left-arm channels.
Go to primary source ↓e08PDF p. 8, Sections IV-B and IV-C1–2; p. 15, Table IX
Qwen3VL-4B is the VLA backbone. NTP uses 150,000 steps, peak learning rate 5e-5, 1,000-step warmup and cosine decay after 100,000 steps to 5e-6. Target adaptation uses three NTP epochs at 2e-5 then five robot-only RF/KM epochs with action-expert learning rate 1e-4. Early and middle VLM features condition the shallower expert.
Go to primary source ↓e09PDF pp. 8–9, Section V-A1–3 and Figure 7
Astribot deployment locks the chassis and torso and uses head and two wrist RGB views. Seen and unseen Pick and Place each use 20 trials. Pack the Doll and Fold T-shirt each use 200 tuning demonstrations and 10 test trials. Bouquets uses 100 demonstrations and 10 tests for each of two combinations.
Go to primary source ↓e10PDF p. 9, Table I, CLAP-NTP/CLAP-RF and baseline rows; p. 10, Section V-A4
Reported Task Mean is 62.7% for CLAP-RF, 58.7% for NTP, 60.0% for pi0.5 and 54.0% for pi0. RF/NTP are 40/30 on folding, 90/80 on doll P&P, 70/70 on doll closing and 70/85 on OOD placement. The source does not specify the Task Mean weighting formula.
Go to primary source ↓e11PDF p. 10, Figure 8; pp. 10–11, Section V-A5, Experimental Design, Comparative Analysis and Results
A GoPro viewpoint resembles the robot head and human operators mimic gripper motions. Bouquets adds three settings with 100 videos each; PnP adds 20 videos per OOD object. CLAP-RF improves 10% to 45% on OOD Bouquets and 70% to 85% on OOD PnP; UniVLA improves 5% to 10% and 50% to 55%. pi0.5 uses teleoperation only. Figure labels agree with these task assignments.
Go to primary source ↓e12PDF p. 11, Table III and Section V-A5, Failure-Case Analysis
Failures are mutually exclusive over 20 attempts per setting. With/without human data, PnP wrong-target/pick/place counts are 1/2/0 versus 4/2/0; Bouquets counts are 0/2/9 versus 8/3/7. Correct object selection does not guarantee placement.
Go to primary source ↓e13PDF p. 10, Table II and Figure 9; p. 11, Section V-A6
Perturbations change background, lighting or target/distractors. CLAP-RF Close success remains 70% under all three; pi0.5 gives 60%, 50% and 60%. Reported means are 70.0% and 56.7%, respectively.
Go to primary source ↓e14PDF p. 11, Table IV and caption; p. 12, Section V-B
A single CLAP-RF trains across four LIBERO suites, each with 10 tasks and 50 demonstrations per task, after filtering training failures. Fine-tuning uses 30,000 steps, batch 128 and KM; evaluation uses 500 trials per suite. CLAP-RF scores 98.6/99.2/98.0/93.0%, average 97.2%; X-VLA averages 98.1%, pi0.5 and FLOWER 96.9%. The table separates specialist and generalist protocols.
Go to primary source ↓e15PDF pp. 7–8, Eq. (8) and Figure 6; p. 12, Table V and Section V-C1
The chosen rate-distortion row has Nq=16, K=256, MSE 0.0004, PSNR 40.00 dB and rate 0.086. Action normalization is [-1,1]. PSNR evaluates reconstruction, not execution success.
Go to primary source ↓e16PDF p. 12, Table VI; p. 13, Section V-C4
Full low+mid-level features plus KM gives LIBERO average 97.2% and PnP ID/OOD 85/70%. Direct VLM fine-tuning gives 96.2% and 85/60%; KI gives 88.3% and 75/65%. High-level-only and low-level-only rows give 93.4% and 96.1% LIBERO averages. Exact selected layer indices and a complete factorial experiment are not specified.
Go to primary source ↓e17PDF p. 13, Table VII and Section V-C2–3
CLAP-NTP Full averages 60.0%; removing pseudo positives gives 58.8%, removing contrastive loss 53.8%, removing human data 47.5%, and Prismatic-7B 56.3%. OOD Bouquets is 35%, 35%, 20%, 5% and 25%, respectively. The data-recipe row prints 75/65/30/10 and average 45.8, which is not their simple arithmetic mean.
Go to primary source ↓e18PDF p. 1, Figure 1; p. 13, Section V-D, Action latent space; p. 14, Figure 10
Tokens are grouped into 32 semantic clusters for examples. Projected decoded trajectories appear only for Astribot because suitable camera extrinsics were unavailable for AgiBot and Ego4D. The t-SNE examples use three manually selected semantic categories, 20 pairs each.
Go to primary source ↓e19PDF pp. 13–14, Section V-D, Human data and Figure 11
A 500-episode Bouquets mixture varies human-video fraction. WiLoR palm position proxies the end effector and fingertip distance proxies gripper width. The estimated-pose condition increases failures relative to the 60% human-video condition. This is a limited test of pose-derived control.
Go to primary source ↓e20PDF p. 14, Section V-D, Inference speed and Table VIII
A single NVIDIA H100 with one 224×224 image yields CLAP-NTP/RF latencies of 382.0/70.1 ms and memory of 9.2/12.1 G. pi0 latency is 45.8 ms. This input differs from the three-view real-world setup.
Go to primary source ↓e21PDF pp. 14–15, Section VI, limitations paragraphs
The authors identify unseen-task transfer without robot grounding, human-hand versus gripper morphology, intentionally gripper-like flower demonstrations and multistage engineering complexity as limitations.
Go to primary source ↓e22PDF p. 15, Table IX; p. 5, Algorithm 2; p. 6, Eq. (7); p. 8, Section IV-C2; p. 12, Table V
Table IX specifies chunk size 32, eight codes per arm, Act-VAE codebook size [256,128], 100,000 steps for each VAE, and 150,000 for NTP. It estimates 190/380/3,800 hours on one A100 80G, rather than reporting measured distributed wall time. The relation of [256,128] to Table V’s K=256, EMA rate, KM weight, RF solver settings and selected feature-layer indices remain unspecified or unresolved.
Go to primary source ↓8.5 Primary sources
CLAP: Contrastive Latent Action Pretraining for Learning Vision-Language-Action Models from Human Videos ↗
PDF · 12,883 extracted words
Source fingerprint
ff176354753c9b525013eaf7bfd91c5432a6c360af1084ff260ae46746769345