V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning
1. Paper overview
In one sentence: Supervising visible patches at multiple encoder depths makes video features substantially more useful for dense vision, while extra supervision and scaling recover much of the recognition performance lost to local reconstruction. feature-hypothesisarchitecturerecipe-ablationlayer-diagnosticdense-resultssta-resultsmanipulation
| At a glance | What to know |
|---|---|
| Research problem | Author claim Masked-only video representation learning can recognize actions while yielding fragmented patch features. The authors hypothesize that unsupervised visible tokens become global aggregators, leaving local geometry difficult to recover with a linear probe. The objective is to retain spatial detail, temporal consistency and global semantics together. jepa-objectivefeature-hypothesis |
| Core mechanism | Source description A distance-weighted visible-token loss complements masked prediction; deep supervision at four encoder levels repairs much of the accompanying recognition penalty. weighted-contextdeep-supervisionrecipe-ablation |
| A key reported result | Short-term object interaction anticipation: 7.71 for ViT-G. Top-5 mAP All. Ego4D STA v2 validation: 47,395 examples; train: 98,276. Frozen backbone plus trained detection/attention head. V-JEPA 2 ViT-g: 6.02; same-protocol DINOv3 7B: 5.68; literature STAformer: 5.67. Requires correct noun/verb, box IoU >0.5 and contact time within 0.25 seconds; discounts four highest-scoring false positives. Input configuration conflicts remain. sta-protocolsta-resultssta-appendix |
| Reading caution | Author claim The authors attribute weaker cluttered-scene segmentation to limited exposure in VisionMix, but do not isolate that cause experimentally. They describe gripper timing and premature opening as remaining manipulation failure modes. segmentation-limitmanipulation |
Core contributions
- Source description
A distance-weighted visible-token loss complements masked prediction; deep supervision at four encoder levels repairs much of the accompanying recognition penalty. weighted-contextdeep-supervisionrecipe-ablation
- Source description
Native image/video tokenizers, VisionMix163M, scaling to ViT-G and distillation form a transferable feature family; downstream evidence spans dense probes, forecasting and separately trained planners. data-tokenizerdistillationmanipulationnavigation
Figure 4. Both visible and masked tokens receive multi-level feature targets. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read left to right along the upper branch. The input takes either the 2D image patchifier or the 3D video patchifier, receives a modality embedding, and loses the masked tokens before entering the x-encoder. Several encoder depths feed the fusion MLP. Mask tokens are then appended, and the predictor emits features at several levels. Now trace the lower branch: the y-encoder sees the complete input and supplies targets. Blue dashed connections terminate at the weighted context loss; orange connections terminate at the masked prediction loss. The stop-gradient marks belong to the target paths. The distance weighting is defined precisely by Equations (2)–(3), rather than by the diagram's shorthand. architecturejepa-objectiveweighted-contextdeep-supervision
What it supports. The added context branch asks visible-token predictions to agree with locally indexed clean-input targets. Deep supervision repeats this constraint at several representation depths. Together, these operations give a direct training signal for patch structure that the original masked-only loss did not impose on visible outputs. This is feature pretraining; no robot action is emitted by this diagram.
Where the evidence stops. The caption describes predictor outputs as masked-token predictions, but the blue branch, Equation (2) and the architecture paragraph on page 6 explicitly require visible-token predictions too. The crop preserves that wording boundary; the two-loss formulation governs this explanation.
2. Motivation
2.1 The problem and the proposed response
Masked-only video representation learning can recognize actions while yielding fragmented patch features. The authors hypothesize that unsupervised visible tokens become global aggregators, leaving local geometry difficult to recover with a linear probe. The objective is to retain spatial detail, temporal consistency and global semantics together. jepa-objectivefeature-hypothesis
2.2 What this reading follows
A video encoder can recognize an action while producing patch features that are difficult to use for locating objects or estimating depth. V-JEPA 2.1 targets that gap by changing which tokens receive a prediction loss and how deeply that loss reaches into the encoder. Follow the architecture first, then the cumulative recipe and the appendix test of final-layer information. The dense-task and anticipation tables show what becomes accessible through frozen features. Finally, the robot table tests an embodied application, with an important distinction between changing the representation and changing the planner. The strongest lesson is about representation quality; task-specific training and search still determine downstream behavior. feature-hypothesisarchitecturerecipe-ablationlayer-diagnosticdense-resultssta-resultsmanipulation
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Foundational work |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
This table preserves the labels recorded at reading time. The current major category is Components of WAMs. View the current classification.
3.1 Evidence-based assessment
Supports the recorded classification
The recorded foundational visual-encoder classification is supported. Base pretraining predicts latent visual tokens without action outputs. Manipulation and navigation add separate learned dynamics and search. Shared image/video encoding therefore does not establish a One Model world-action architecture or joint future/action prediction; the recorded architecture, paradigm and quadrant remain not applicable. architecturemanipulationnavigationfuture
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Why supervise a patch the encoder already saw?
The original objective only scores predictions at missing positions. A visible token can help solve that task by summarizing the scene, even if its own spatial location becomes difficult to decode. This is the authors' proposed explanation for fragmented feature maps, rather than a proof about every JEPA solution. The context term adds a different obligation: after passing through the predictor, each visible token must match the clean-view target at its own index. Equation (3) gives stronger weight near masked regions, where continuity between observed and hidden content matters. Reader interpretation: this redistributes training pressure toward spatially anchored features, but may compete with global aggregation. Table 1 makes that competition observable through the immediate SSv2 drop, so dense-looking PCA maps alone would be insufficient evidence of a successful recipe. jepa-objectivefeature-hypothesisweighted-contextrecipe-ablation
5.2 Why training at four depths differs from probing four depths
V-JEPA 2.1 changes the training information flow before choosing a downstream head. Four encoder levels are concatenated and compressed into the predictor input, and both losses supervise predictions against the corresponding target levels. The final model can therefore be evaluated with only its output layer for dense images. Table 13 separates that training choice from the later choice to concatenate four layers at evaluation. Deep supervision raises last-layer ADE20K performance above the unsupervised-depth model's four-layer performance, although using four layers still helps. This distinction also explains why the smaller distilled models use a different recipe: the fixed ViT-G teacher supplies final-layer targets, deep supervision is removed, and a separately averaged student becomes the released representation. Distillation transfers the teacher's learned structure without repeating its multi-level target construction. deep-supervisionlayer-diagnosticdense-protocoldistillation
5.3 From representation prediction to actions in the world
There are several different uses of prediction in this paper. Ego4D trains a detection head to forecast a human's next interaction; its boxes, classes and contact times are evaluation outputs. EK100 uses the pretrained predictor's future features in a classifier. Robot manipulation instead learns action-conditioned dynamics from DROID and searches candidate actions with CEM, then executes the selected behavior through model-predictive control. Navigation uses yet another downstream model: diffusion over latent features, evaluated through open-loop trajectory errors. Reader interpretation: improvements across these tasks support the usefulness of the representation, but they do not identify one universal action-generating architecture. Table 6 is especially instructive because representation and planning changes are visible in separate rows, and its strongest grasp result spends more time planning as well as using the newer features. sta-protocolek100manipulationnavigationarchitecture
5.4 Training and inference
During training
VisionMix163M replaces ImageNet with 142M curated images and reweights video sources. Separate modality workers aggregate gradients, with global batches of 2,304 images and 128 clips. Pretraining lasts 135,000 steps at 256-pixel resolution and 16 video frames; 12,000 cooldown steps use 512-pixel images and 64-frame, 384-pixel videos. data-tokenizerpretraining
The 24-block predictor has width 384. EMA is 0.99925, weight decay 0.04, and context coefficients are 0.5 for video and 0.7 for images. Distillation replaces the EMA target with a frozen ViT-G teacher, uses a new 12-block predictor and final-layer losses, and releases the averaged student. Low- and high-resolution teachers match the two training stages. pretrainingdistillation
During inference
Dense image tasks train linear probes on frozen final-layer features; VOS propagates initial masks through cosine-similarity neighbors. EK100 retains the frozen predictor to infer future-frame features and combines them with context features in an attentive classifier. These outputs describe future events rather than executing them. dense-protocolvos-protocolek100
Manipulation trains a separate frame-causal action-conditioned predictor on DROID with teacher-forcing and two-step rollout losses, then uses CEM model-predictive control. Navigation instead trains a conditional diffusion world model, predicts clean features with DDIM, and searches 480 candidate two-second trajectories in a three-degree-of-freedom action space. manipulationnavigation
5.5 Implementation flow
- Tokenize and mask
Use 16×16 image patches and 16×16×2 video tubelets, modality embeddings and 3D RoPE. The context encoder sees visible tokens; an EMA target encoder sees the complete input. architecturedata-tokenizer
- Fuse encoder depths
Concatenate normalized representations from three intermediate blocks and the final block, then compress them with an MLP. Append positional mask tokens and predict representations at four levels. ViT-G uses blocks 12, 24, 36 and 48. architecturedeep-supervisionpretraining
- Supervise both token sets
Apply L1 prediction error to masked tokens and weighted L1 error to visible tokens at each level. Context tokens nearer masked regions receive stronger weighting; gradients stop at the EMA targets. jepa-objectiveweighted-contextdeep-supervision
6. Experiments & results
V-JEPA 2.1 learns image/video patch features by predicting both hidden and visible representations at multiple encoder depths. This substantially improves dense vision while largely preserving global recognition; separate task heads and world models turn the features into anticipation and planning systems.
6.1 Read the original evidence
Table 8. Large dense-task gains over V-JEPA 2 coexist with stronger segmentation and tracking scores from DINOv3. Original paper, p. 15 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the two V-JEPA 2.1 rows at the bottom and the V-JEPA 2 row just above them. The first two numeric columns are depth RMSE, where smaller is better. The next three are segmentation mIoU, followed by video-mask J&F means, where larger is better. For dense images, the evaluation fits a linear head to frozen final-layer features; tracking instead propagates a supplied first-frame mask. The parameter field also records patch size. Compare DINOv3's 7B row separately: it nearly matches ViT-G on NYUv2 while remaining stronger in KITTI, all three segmentation columns and both tracking columns. dense-resultsdense-protocolvos-protocolresolution-ablation
What it supports. ViT-G cuts NYUv2 RMSE from V-JEPA 2's 0.642 to 0.307 and raises ADE20K mIoU from 24.4 to 47.9. The outcome is a much more useful general video representation for dense tasks. It is not a clean sweep: DINOv3 7B scores 55.9 on ADE20K and 74.1 on YouTube-VOS versus 47.9 and 72.7 here.
Where the evidence stops. Comparisons vary in model scale and pretraining. YouTube-VOS uses a custom 80/20 division of its training videos, not the official hidden test set. Table 14 lists 72.6 for high-resolution ViT-G YouTube-VOS, whereas this table gives 72.7; neither reports uncertainty.
Table 4. The 7.71 score measures a joint forecast of object, action and contact time. Original paper, p. 11 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the column subscripts as progressively stricter event requirements. The paper defines b as the next active object's box, N as its noun class, V as the upcoming verb, and δ as time to contact. The rightmost mAP All column requires these aspects jointly, so it is the principal score. The upper block contains published systems; the lower block evaluates frozen encoders with the authors' detection/attention protocol. Compare the final ViT-G row with V-JEPA 2 and DINOv3 within this lower block before using the literature comparison. Appendix C.4 says each encoder is reported with its best respective configuration. sta-protocolsta-resultssta-appendix
What it supports. ViT-G reaches 7.71 mAP All versus 6.02 for V-JEPA 2 and 5.68 for DINOv3 7B. Its box AP is 50.7 and box-plus-contact-time AP is 20.2. The joint improvement therefore accompanies useful localization and timing predictions, while DINOv3 remains stronger in some noun-sensitive columns. This is supervised forecasting from frozen features, not an executed interaction.
Where the evidence stops. The Top-5 metric discounts four high-scoring false positives; 7.71 is not a robot success rate. The main text specifies eight frames over 0.5 seconds, while Appendix C.4 selects 16 frames at 2 fps. That configuration conflict remains unresolved.
Table 6. Matched planning yields a 10-point grasp gain; the 20-point gain also changes planning settings. Original paper, p. 13 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the planning details before reading success rates. The first two rows use 800 CEM samples, ten optimization iterations, horizon one and three seconds of planning. That pair is the closest comparison of the learned representation: grasp rises from 60% to 70%, with reach and pick-and-place unchanged. The final row changes several settings together: 300 samples, 15 iterations, horizon eight and 14 seconds. It reaches 80% grasp. The caption places all results on a Franka Panda cup-manipulation setup with ten tasks per skill and one A100 GPU for planning. The predictor was separately trained on DROID, as described on page 12. manipulation
What it supports. This is direct closed-loop robot evidence, beyond visual plausibility or offline feature probes. The matched planner gains ten percentage points in grasp success, while the longer, slower planner gains twenty points over the original baseline. The authors associate remaining failures with gripper timing and opening, showing that better spatial features do not solve every component of manipulation.
Where the evidence stops. The caption's '20% improvement' corresponds to 60%→80%, a 20-percentage-point change under a different planner. Ten tasks per skill and absent confidence intervals limit precision; the reported long-horizon degradation of V-JEPA 2 has no numerical row here.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Dense-loss and deep-supervision ablation Cumulative ViT-L recipe; ADE20K linear and SSv2 attentive probes. | 22.2 / 72.8 → 33.8 / 62.5 with context loss → 38.6 / 72.1 with multi-level prediction. ADE20K mIoU / SSv2 top-1 accuracy | Masked-only V-JEPA 2 is the starting row. Dense gains initially cost recognition accuracy. Use Table 1 values; prose gives 33.9 and 0.473 rather than 33.8 and 0.474 for context-only ADE20K/NYUv2. recipe-ablationweighted-context |
| Short-term object interaction anticipation Ego4D STA v2 validation: 47,395 examples; train: 98,276. Frozen backbone plus trained detection/attention head. | 7.71 for ViT-G. Top-5 mAP All | V-JEPA 2 ViT-g: 6.02; same-protocol DINOv3 7B: 5.68; literature STAformer: 5.67. Requires correct noun/verb, box IoU >0.5 and contact time within 0.25 seconds; discounts four highest-scoring false positives. Input configuration conflicts remain. sta-protocolsta-resultssta-appendix |
| Action anticipation EK100 validation; 32 frames at 8 fps, 384×384; one-second anticipation. | 40.8 for 2B ViT-G. Mean-class action Recall@5 | 1B V-JEPA 2: 39.7; 1B V-JEPA 2.1: 38.4. The larger model improves by 1.1 points over V-JEPA 2; the equal-size comparison does not improve. ek100 |
| Closed-loop cup manipulation Zero-shot lab Franka Panda deployment; ten tasks per skill, CEM on one A100. | Matched planner: 70% / 100% / 80%; longer planner: 80% / 100% / 80%. Grasp / reach / pick-and-place success | V-JEPA 2: 60% / 100% / 80%. Matched setting: 800 samples, 10 iterations, horizon 1, 3 seconds; longer: 300, 15, horizon 8, 14 seconds. Grasp improves by 10 percentage points with matched planning, or 20 points with changed planning. No confidence intervals are reported. manipulation |
| Open-loop navigation planning Validation trajectories on Recon, Tartan Drive, Scand and Sacson; two-second plans. | ViT-G Tartan Drive: 5.687; average ATE: 2.990; time: 10.6 seconds. ATE / planning time | NWM: 5.831; 3.032; 103.2 seconds. Approximately 9.7× faster, calculated from reported times; gains are not universal across datasets. This evaluates trajectory error, not closed-loop navigation success. navigation |
| Dense image understanding Frozen final-layer single-image probes on NYUv2, KITTI and ADE20K; source benchmark evaluation. | NYUv2 0.307; KITTI 2.461; ADE20K 47.9. Depth RMSE (lower better) / segmentation mIoU | V-JEPA 2: 0.642 / 4.650 / 24.4; DINOv3 7B: 0.309 / 2.346 / 55.9. Strong improvement over V-JEPA 2; the small NYUv2 lead over DINOv3 has no uncertainty estimate and does not extend to KITTI or segmentation. dense-resultsdense-protocol |
| Video object segmentation DAVIS 2017 validation and a custom 20% holdout from YouTube-VOS training videos; nonparametric mask propagation. | 69.0 on DAVIS; 72.7 on YouTube-VOS (Table 8). J&F mean | V-JEPA 2: 52.5 / 53.7; DINOv3 7B: 71.1 / 74.1. Improves correspondence quality but remains behind DINOv3. Table 14 instead lists 72.6 for high-resolution ViT-G YouTube-VOS. dense-resultsvos-protocolresolution-ablation |
| Global recognition Frozen attentive probes, benchmark validation, multi-view video evaluation. | SSv2 77.7%; Diving-48 89.2%; K400 87.7%; IN1K 85.5%. Top-1 accuracy | V-JEPA 2: 77.3 / 90.2 / 87.3 / 85.1. Global capability is largely retained, with a Diving-48 regression. The SSv2 frame-count conflict prevents an unambiguous replication setting. recognitionrecognition-appendix |
| Video question answering Llama 3.1 8B alignment on filtered PerceptionLM; compare the two rows trained in this paper. | 57.9 / 83.1. Reported benchmark average / PerceptionTest validation accuracy | Retrained V-JEPA 2: 57.8 / 80.1. Mixed transfer: TemporalBench and TOMATO decline. Older starred PerceptionTest results use the unavailable test server and are not directly comparable. videoqa |
| Compact-model distillation 300M ViT-L, same downstream probe protocols. | Distilled: 76.5 / 46.7. SSv2 accuracy / ADE20K mIoU | Scratch ViT-L: 74.2 / 42.0; 2B teacher: 77.7 / 47.9. Distillation closes much of the gap; Table 11 gives teacher DAVIS 69.0, contrary to 67.0 in the distillation prose. distilled-resultsdistillation |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 1. Dense supervision introduces a measurable recognition cost before multi-level prediction recovers it. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Each plus sign adds to the preceding recipe, so read downward rather than treating the rows as independent alternatives. Higher IN1K and SSv2 accuracy and ADE20K mIoU are better; lower NYU RMSE is better. Compare the first three rows before considering scale: context loss changes ADE20K from 22.2 to 33.8 but SSv2 from 72.8 to 62.5. Multi-level prediction then reaches 38.6 and 72.1. The remaining rows add broader data, native modality tokenization, larger model capacity and cooldown. Figure 5 identifies the starting architecture as ViT-L; the model-scaling row moves to ViT-G. recipe-ablationweighted-contextdeep-supervisiondata-tokenizerpretraining
What it supports. The dense-loss idea alone does not satisfy the paper's joint local/global goal. Deep supervision recovers most of the lost SSv2 performance while further improving segmentation. The final 47.9 ADE20K and 77.7 SSv2 scores belong to the full scaled recipe, whose effect cannot be assigned solely to the context-loss term.
Where the evidence stops. These are cumulative changes, not a factorial causal test. Table 1 gives context-only ADE20K 33.8 and NYU 0.474; nearby prose gives 33.9 and 0.473. This edition retains the table values and records the discrepancy.
Table 13. Deep supervision strengthens the final layer and reduces, but does not erase, the benefit of reading intermediate layers. Original paper, p. 37 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. The cross and check describe whether deep self-supervision was used during pretraining. Within either row, the paired columns change only which encoder features reach the evaluation probe: the final layer or four concatenated layers. This is the paper's scratch-trained ViT-L diagnostic. For ADE20K, the unsupervised-depth row improves from 34.9 to 39.1 when the probe reads four layers; with deep supervision, the corresponding scores are 42.0 and 43.9. For NYU, remember that lower error is better: the pairs are 0.513/0.463 and 0.381/0.370. The rightmost columns repeat the comparison for Diving-48 recognition. layer-diagnosticdense-protocolrecognition-appendix
What it supports. Calculated from the displayed ADE20K entries, the benefit of four-layer evaluation falls from 4.2 to 1.9 mIoU points. More strikingly, the deeply supervised model's final layer already exceeds the other model's four-layer score. This supports the claim that training makes useful information more accessible at the output, rather than merely allowing a richer downstream probe to retrieve it.
Where the evidence stops. Four-layer evaluation still improves every displayed task. The result supports reduced dependence on intermediate features, not their universal redundancy. The source does not provide repeated-run uncertainty for these differences or a matched probe-parameter analysis.
7. Analysis & limitations
7.1 What the evidence leaves open
The authors attribute weaker cluttered-scene segmentation to limited exposure in VisionMix, but do not isolate that cause experimentally. They describe gripper timing and premature opening as remaining manipulation failure modes. segmentation-limitmanipulation
Sequential recipe gains combine objectives, data, capacity and resolution. Table 13 supports better final-layer features, not elimination of every multi-layer benefit. PCA displays are qualitative and their RGB permutation is selected for appearance. recipe-ablationlayer-diagnosticqualitative
Source inconsistencies remain: navigation Table 7 says finetuning while Section 3 describes frozen encoders; ATE is expanded differently in its caption and text. Table 14 also contains regressions despite the prose claiming universal resolution gains. These ambiguities are retained, not resolved by assumption. distillationnavigationresolution-ablation
7.2 Questions for discussion
- Would multi-level supervision recover recognition at fixed predictor capacity and pretraining budget?
- Does the grasp improvement persist when both encoders receive identical horizon, search budget and repeated object configurations?
8. Reproducibility audit
8.1 Requirements and known gaps
A full reproduction needs VisionMix source access, EMA/masking implementation, both training resolutions and task-specific probes. Appendix A specifies schedules but not pretraining GPU count, runtime or software versions. Its cooldown starts at 6e-4 after a 5.25e-4 plateau; Section 2.3.1 expresses context warmup in epochs without defining its conversion to steps. data-tokenizerpretrainingweighted-context
Resolve Ego4D settings before reproducing 7.71: Section 3.1 uses eight frames over 0.5 seconds and four feature scales; Appendix C.4 selects 16 frames at 2 fps and lists five scales. SSv2 uses 16×2×3 in Table 9, but Appendix C.5 prints 64×2×3 while still saying 16-frame clips. sta-protocolsta-appendixrecognitionrecognition-appendix
Reader-proposed checks: run a matched-budget factorial context-loss/deep-supervision ablation, then compare both encoders under identical short- and long-horizon CEM settings. The illustrated edition specifies controls and falsifiable outcomes; neither check was executed. recipe-ablationlayer-diagnosticmanipulation
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Separate context weighting from supervision depth
Reader-proposed experiment, not executed: train four ViT-L conditions crossing context loss off/on with final-layer-only/four-level supervision. Hold data mixture, masks, updates, predictor depth and width, learning-rate schedule and evaluation search budget fixed; report compute and repeat with independent seeds. Use the documented distance weighting and modality coefficients in the context-on runs, with an explicitly recorded warmup-to-step conversion. Evaluate frozen features on ADE20K and NYUv2 using both final-layer and four-layer probes, plus a fixed SSv2 protocol. The central prediction is that context loss improves dense scores and deep supervision reduces its recognition penalty. If the interaction disappears under matched capacity and compute, the cumulative ablation would not isolate the proposed explanation. Record the SSv2 frame-count choice because the source gives conflicting specifications. recipe-ablationweighted-contextpretraininglayer-diagnosticdense-protocolrecognitionrecognition-appendix
Check 2: Factor representation and planner budget in cup grasping
Reader-proposed experiment, not executed: evaluate both V-JEPA 2 and V-JEPA 2.1 with both complete planner settings from Table 6: (800 samples, 10 iterations, horizon 1) and (300 samples, 15 iterations, horizon 8). Train the action-conditioned predictors on the same DROID subset and losses, pair initial/goal cup configurations, randomize run order, and repeat trials beyond the source's ten-task setup. Record grasp success, planning latency, premature closure, dropped objects and confidence intervals. Add a latency-matched search comparison to separate longer prediction horizons from extra decision time. A representation-specific long-horizon benefit should appear as a reproducible interaction between encoder and horizon, not only as a gain available to either encoder when planning longer. This would also quantify the paper's un-tabulated claim that V-JEPA 2 degrades at longer horizons. manipulation
8.3 Reading coverage
Visual audit: Visually inspected the title/author block, all 15 numbered figures and all 14 numbered tables, the method and evaluation text, and Appendices A–D. Cross-checked Figure 4's two loss paths and target-side stop-gradient against Equations (1)–(3) and the architecture text. All six final original PDF crops were inspected; Table 13's crop was corrected to retain both full rows. Table 1 uses a 400-DPI original render for legibility. Other crops and declared pages use 200-DPI renders. Reference pages 25–32 were read as text; separate supplements, code and external resources were outside this supplied reading. Protocol and numerical disagreements are disclosed rather than silently reconciled.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 33, 34, 35, 36, 37. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Title, abstract and Introduction (pp. 1–3)
- 2.1–2.2 JEPA preliminaries and dense-feature analysis (pp. 3–5)
- 2.3.1–2.3.6 Dense loss, deep supervision, data, tokenizer, scaling and distillation (pp. 5–9)
- 3.1–3.4 Anticipation, manipulation and navigation (pp. 9–14)
- 3.5–3.10 Dense tasks, recognition, video QA, qualitative features and distilled models (pp. 15–23)
- 4 Related work and 5 Conclusion and Future work (pp. 20, 23–24)
- References (pp. 25–32)
- Appendix A Pretraining and B Distillation (pp. 33–34)
- Appendix C.1–C.6 Evaluation protocols (pp. 34–36)
- Appendix D.1–D.2 Additional ablations (p. 37)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- All 11 supplied text chunks were read individually. The extraction limitation was addressed by visually inspecting PDF pages 1–24 and 33–37, including all numbered figures and tables; reference pages 25–32 were read as text.
- Version scope: the supplied PDF is arXiv:2603.14482v3, stamped 11 June 2026, with a title-block date of June 12, 2026. The catalog submission date is March 15, 2026; earlier editions were not supplied or compared. The observed title and nine-author list match the catalog display. Its BibTeX uses Mahmoud Assran and Michael Rabbat where the observed credits say Mido Assran and Mike Rabbat; metadata preserves the observed credits.
- The linked code, checkpoints and external datasets were not inspected. No training, evaluation or robot experiment was reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
identityPDF p. 1, title block and arXiv margin
Exact observed title; nine authors: Lorenzo Mur-Labadia, Matthew Muckley, Amir Bar, Mido Assran, Koustuv Sinha, Mike Rabbat, Yann LeCun, Nicolas Ballas and Adrien Bardes. Affiliations: FAIR at Meta and Universidad de Zaragoza. Margin: v3, 11 Jun 2026; title date: June 12, 2026.
Go to primary source ↓jepa-objectivePDF pp. 3–4, Section 2.1 and Eq. (1)
Predictor inputs are context and mask tokens; masked-only L1 targets come from a stop-gradient EMA encoder of the clean input.
Go to primary source ↓feature-hypothesisPDF p. 4, Section 2.2 and Figure 3
PCA shows fragmented V-JEPA 2 features. Authors hypothesize that visible tokens lack local supervision and aggregate global information.
Go to primary source ↓architecturePDF pp. 5–6, Figure 4 and architecture paragraph
Image/video patch embeddings feed shared encoders; multi-level features are fused before prediction. Blue context and orange masked branches have target-side stop-gradient. The caption mentions masked predictions, while the diagram, Eq. (2) and p. 6 explicitly include context predictions.
Go to primary source ↓weighted-contextPDF pp. 5 and 7, Eqs. (2)–(3), Table 2 and Section 2.3.1
Visible-token L1 is scaled by λ divided by the square root of distance to the nearest masked token. Warmup is described over epochs 50–100. Weighted λ=0.5 with warmup gives 33.8 ADE20K and 62.5 SSv2; prose instead states 33.9 and 0.473 NYUv2.
Go to primary source ↓deep-supervisionPDF pp. 7–8, Section 2.3.2
Three intermediate outputs plus final output are concatenated and MLP-fused; predictor outputs at all four levels receive context and prediction losses. The p. 8 prose says 72.0 SSv2 whereas Table 1 gives 72.1.
Go to primary source ↓recipe-ablationPDF pp. 6–7, Figure 5 and Table 1, cumulative rows
Baseline IN1K/SSv2/NYU/ADE20K: 82.2/72.8/0.682/22.2; context: 72.6/62.5/0.474/33.8; multi-level: 80.8/72.1/0.463/38.6; data: 81.6/72.6/0.418/40.8; tokenizer: 81.6/72.6/0.415/41.4; model scale: 84.8/76.1/0.365/47.1; cooldown: 85.5/77.7/0.307/47.9. Figure 5 starts with ViT-L.
Go to primary source ↓data-tokenizerPDF pp. 8–9, Table 3 and Sections 2.3.3–2.3.5
VisionMix163M replaces 1M ImageNet images with LVD-142M; video weights emphasize SSv2 and YT-1B. LVD sampling-weight cells are blank; separate modality batches contain 2304 images and 128 videos. Native 2D 16×16 and 3D 16×16×2 patchifiers avoid image temporal duplication. Scaling reaches 2B ViT-G.
Go to primary source ↓pretrainingPDF pp. 6 and 33, implementation details, Appendix A and Table 12
135k primary steps, 12k warmup, 12k cooldown; 4-fps videos change from 16 frames at 256 to 64 at 384, images from 256 to 512. Predictor: 24 blocks, width 384; EMA 0.99925; weight decay 0.04; ViT-G taps [12,24,36,48]; λ=0.5 video/0.7 image. Learning rates: 1e-4 to 5.25e-4, then cooldown 6e-4 to 1e-6. Pretraining hardware/runtime/software versions are not specified.
Go to primary source ↓distillationPDF pp. 9 and 34, Section 2.3.6, Section 3 opening and Appendix B
Frozen teacher replaces EMA targets; new 12-block predictor matches teacher output dimensions; no deep self-supervision. Student EMA is used for final output, not the loss. Teachers match pre-/post-cooldown stages. Section 3 describes downstream use of a frozen encoder.
Go to primary source ↓sta-protocolPDF pp. 9–10, Section 3.1 and Figure 6
Ego4D STA v2 has 98,276 train and 47,395 validation examples. Outputs are box, noun, verb and contact time in last-observed-frame coordinates. Top-5 mAP All uses IoU >0.5 and 0.25-second timing tolerance with four discounted false positives. Main text: eight video frames over 0.5 seconds at 384 pixels plus a 1080-pixel image; four feature scales.
Go to primary source ↓sta-resultsPDF p. 11, Table 4, mAP All and component columns
ViT-G mAP All 7.71 versus V-JEPA 2 6.02, DINOv3 7B 5.68 and STAformer 5.67. ViT-G AP_b 50.7, AP_b+V 25.8, AP_b+δ 20.2, mAP_N+δ 12.8; nearby prose says 12.9 with inconsistent notation. Literature and common-protocol rows are separated.
Go to primary source ↓sta-appendixPDF pp. 35–36, Appendix C.4
Last-frame queries attend to video keys/values, followed by interpolation, image-feature fusion, a 2D convolution and five pyramid scales (1/4 through 1/64). RPN/RoI features predict noun, verb and positive contact time. Selected configuration: 16 frames at 2 fps, 384 pixels, learning rate 1.2e-4; multi-size image training. Other encoders use best respective configurations.
Go to primary source ↓ek100PDF pp. 12 and 36, Table 5, Section 3.2 and Appendix C.6
Validation action mean-class Recall@5: V-JEPA 2 1B 39.7, V-JEPA 2.1 1B 38.4, 2B 40.8. Context: 32 frames at 8 fps and 384×384; evaluation anticipates one second ahead. Frozen encoder/predictor outputs feed three classifier queries. Appendix describes randomized training anticipation and focal loss.
Go to primary source ↓manipulationPDF pp. 12–13, Section 3.3, Figure 8 and Table 6
Separate 24-layer approximately 300M action-conditioned predictor learns from raw DROID with teacher-forcing and two-step rollout losses. Franka Panda visual-goal MPC uses CEM and one A100, averaging ten tasks per skill. Matched planner (800 samples, 10 iterations, horizon 1, 3 s) gives grasp 60% vs 70%; changed planner (300,15,8,14 s) gives 80%. All rows give 100% reach and 80% pick-and-place. Gripper failures are described qualitatively.
Go to primary source ↓navigationPDF p. 14, Section 3.4, Figure 9 and Table 7
Conditional diffusion predicts clean latent features with DDIM; CEM uses 480 two-second candidates at 4 fps and translation/yaw actions. Eight denoising steps replace at least 128. ViT-G/NWM: Tartan ATE 5.687/5.831, mean ATE 2.990/3.032, time 10.6/103.2 s. ViT-G has worse Recon and Sacson ATE. Caption says finetuning and normalized Average Trajectory Error; prose uses Absolute Trajectory Error without a normalization formula.
Go to primary source ↓dense-resultsPDF pp. 15–16, Table 8 and Section 3.5
ViT-G NYUv2/KITTI RMSE 0.307/2.461; ADE20K/Cityscapes/VOC mIoU 47.9/73.5/85.0; DAVIS/YT-VOS J&F 69.0/72.7. V-JEPA 2: 0.642/4.650, 24.4/45.9/63.9, 52.5/53.7. DINOv3 7B: 0.309/2.346, 55.9/81.1/86.6, 71.1/74.1. Prose segmentation deltas do not all equal Table 8 subtraction.
Go to primary source ↓dense-protocolPDF pp. 15 and 34, Section 3.5 and Appendices C.1–C.2
Dense probes use frozen final-layer image-tokenizer features. Depth uses a linear head with learned BatchNorm; segmentation uses normalized features and AdamW. ADE20K/VOC use 512 pixels for patch size 16 versus 448 for size 14; Cityscapes uses 1024-pixel height. Default depth resolutions and benchmark split details are not fully enumerated.
Go to primary source ↓segmentation-limitPDF p. 16, Section 3.5 results
Authors hypothesize that fewer highly cluttered VisionMix scenes explain weaker ADE20K and Cityscapes segmentation.
Go to primary source ↓vos-protocolPDF pp. 16–18 and 34–35, Section 3.6, Figure 13 and Appendix C.3
First-frame masks propagate through local cosine-similarity neighbors without learned parameters. DAVIS has 60 train/30 validation videos; YouTube-VOS training videos are divided 80/20. Appendix selects hyperparameters on DAVIS training; Section 3.6 describes searches on each training split before applying the DAVIS configuration. Final settings: 15 context frames, circle size 12, top-5, temperature 0.2, 480-pixel short side.
Go to primary source ↓recognitionPDF pp. 18–19, Table 9 and Section 3.7
Frozen attentive probes average multi-clip logits. ViT-G SSv2/Diving-48/K400/IN1K: 77.7/89.2/87.7/85.5; V-JEPA 2: 77.3/90.2/87.3/85.1. Table 9: SSv2 16×2×3, K400 16×8×3, Diving-48 32×4×3. Literature probe architectures can differ.
Go to primary source ↓recognition-appendixPDF p. 36, Appendix C.5
Four probe blocks use 16 attention heads, with final query cross-attention. SSv2 text prints 64×2×3 but calls it a 16-frame clip. Diving-48 uses four encoder layers and 50 epochs; most tasks use 20 epochs and a best-of-20-head sweep.
Go to primary source ↓videoqaPDF pp. 19–20, Table 10 and Section 3.8
Same-filtered-data rows: average 57.8/57.9 and PerceptionTest validation 80.1/83.1 for V-JEPA 2/2.1. TemporalBench drops 32.3 to 28.5 and TOMATO 41.4 to 38.0. Alignment uses Llama 3.1 8B, removes 18% of PerceptionLM via filters/Qwen3VL judging, and adds longer-context alignment and PerceptionTest training. Filtering thresholds are not given. Older starred PerceptionTest scores are test scores; that server is inactive.
Go to primary source ↓qualitativePDF pp. 20–22, Section 3.9 and Figures 14–15
Image/video PCA feature comparisons select the RGB-component permutation for visual appeal. They visualize structure rather than measuring a calibrated semantic or physical state.
Go to primary source ↓distilled-resultsPDF pp. 20 and 23, Section 3.10 and Table 11
Scratch/distilled ViT-L: SSv2 74.2/76.5, ADE20K 42.0/46.7. Table 11 gives teacher DAVIS 69.0, distilled ViT-L 68.7, distilled ViT-B 67.0; p. 20 incorrectly compares ViT-L to teacher 67.0.
Go to primary source ↓futurePDF p. 24, Section 5
Emphasis is on representation learning; further scaling and dense world-model development for robotics are proposed.
Go to primary source ↓layer-diagnosticPDF p. 37, Table 13 and Appendix D.1
Without/with deep supervision, last-layer versus four-layer ADE20K: 34.9/39.1 versus 42.0/43.9; NYU: 0.513/0.463 versus 0.381/0.370; Diving-48: 85.8/86.9 versus 87.2/88.1. Four-layer gains shrink but remain positive.
Go to primary source ↓resolution-ablationPDF p. 37, Table 14 and Appendix D.2
Higher video cooldown resolution improves many scores but not all: ViT-G Diving-48 falls 89.5 to 89.2 and YT-VOS 72.7 to 72.6; ViT-g DAVIS falls 68.0 to 67.4. High-resolution tracking entries differ from Table 8; the prose claim of gains across all tasks is too broad.
Go to primary source ↓8.5 Primary sources
V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning ↗
PDF · 19,218 extracted words
Source fingerprint
b40580916242ade00e2ac177cb01473d511f021d5a25f7552fd17512b3a72bbb