Awesome-WAMs
GitHub
PAPER REPORTENAll readings ↗

DINOv2: Learning Robust Visual Features without Supervision

English reading report: Method, equations, original figures, experiments and reproducibility.

Authors: Maxime Oquab; Timothée Darcet; Théo Moutakanni; Huy V. Vo; Marc Szafraniec; Vasil Khalidov; Pierre Fernandez; Daniel Haziza; Francisco Massa; Alaaeldin El-Nouby; Mahmoud Assran; Nicolas Ballas; Wojciech Galuba; Russell Howes; Po-Yao Huang; Shang-Wen Li; Ishan Misra; Michael Rabbat; Vasu Sharma; Gabriel Synnaeve; Hu Xu; Hervé Jegou; Julien Mairal; Patrick Labatut; Armand Joulin; Piotr Bojanowski

Affiliations: Meta AI Research; Inria; Université Paris Saclay; ENS-PSL

Source: Transactions on Machine Learning Research · ref-e18cd2f0df3768294977 ↗ · Catalog record

Reading: 439 / 558 · 6 original figures & tables · ~19 min ·

1. Paper overview

In one sentence: DINOv2 combines curated image retrieval with global and patch-level self-supervision to make frozen visual features broadly useful, while leaving task decoding and domain bias unresolved. e-probleme-pipelinee-objectivese-loss-ablatione-imagenete-segmentatione-depthe-qualitativee-discussion

At a glanceWhat to know
Research problem
Source description

Can image-only pretraining produce features that transfer across both whole-image recognition and pixel-level tasks without adapting the backbone? The paper argues that data diversity and stable scaling are as consequential as the self-supervised objective. Downstream supervised heads remain part of many evaluations. e-probleme-probinge-segmentation

Core mechanism
Source description

An automatic visual-retrieval pipeline produces LVD-142M, totaling 142,109,386 images, using curated datasets to guide web-image selection and deduplication. e-pipelinee-data-composition

A key reported resultImageNet-1k frozen linear classification: 86.5 validation; 89.6 ReaL; 78.4 V2

Top-1 accuracy (%). ImageNet training labels; frozen ViT-g/14, 224×224; best validation-selected linear probe. ReaL/V2 are additional evaluation sets.

OpenCLIP-G: 86.2/89.4/77.2; iBOT-L validation: 82.3. Validation gains are 0.3 and 4.2 percentage points, respectively. Shared evaluation code does not equal matched architecture or pretraining data. e-imagenete-probing

Reading caution
Reader analysis

Pretraining is image-only, while many downstream heads require labels. Several evaluation domains guide retrieval, so broad transfer does not establish independence from all benchmark domains. Main cross-method tables vary architecture and data; the selected results omit confidence intervals. e-dataset-usee-probinge-imagenete-loss-ablation

Core contributions

  • Source description

    An automatic visual-retrieval pipeline produces LVD-142M, totaling 142,109,386 images, using curated datasets to guide web-image selection and deduplication. e-pipelinee-data-composition

  • Reader analysis

    The work combines existing objectives and efficiency techniques into a scalable recipe, then transfers a 1.1B-parameter teacher's features to smaller encoders. It presents engineering and data contributions rather than a new action model. e-objectivese-efficiencye-distillatione-discussion

Figure 3. Curation is a learned visual search process before representation training. Original paper, p. 4 ↗

Excerpt from the authors’ paper; cropped without altering the figure or table.

How to read it. Follow the arrows from left to right. The upper collection supplies uncurated images; the lower collection supplies curated queries. Both enter an embedding space, but the upper branch passes through duplicate removal before retrieval. The upward arrow from the query at retrieval shows how a curated example selects relevant candidates. The final cylinder combines retained source images with retrieved examples. Section 3 identifies the similarity encoder as a self-supervised ViT-H/16 trained on ImageNet-22k. This graphic describes construction of the training distribution; the student–teacher learning objectives are specified separately in Section 4. e-pipelinee-data-compositione-objectivese-data-discrepancies

What it supports. The pipeline makes data selection part of the method. Its final LVD-142M composition contains 142,109,386 images, with both directly included datasets and retrieval-based additions. Consequently, image-only representation learning still inherits choices about which curated domains deserve coverage; absence of labels in the loss does not remove those choices.

Where the evidence stops. The arrows agree with the caption and Section 3. However, the preceding related-work paragraph says no pretrained encoders are used, while Section 3 explicitly specifies one. This explanation follows the concrete method description and leaves that wording conflict unresolved.

2. Motivation

2.1 The problem and the proposed response

Source description

Can image-only pretraining produce features that transfer across both whole-image recognition and pixel-level tasks without adapting the backbone? The paper argues that data diversity and stable scaling are as consequential as the self-supervised objective. Downstream supervised heads remain part of many evaluations. e-probleme-probinge-segmentation

2.2 What this reading follows

DINOv2 asks how much useful visual structure can be learned before a downstream task is chosen. Its answer joins two decisions: build a diverse image distribution through visual retrieval, then train an encoder to agree with teacher targets at both image and patch scales. The resulting features support classification, instance retrieval, segmentation and depth estimation with limited task-specific training. Read the figures as a chain of evidence: the data pipeline defines what the model sees, the ablations separate some benefits of the losses, and the frozen-feature tables test what information remains accessible. Qualitative correspondences suggest semantic structure, but do not measure physical dynamics or control. e-probleme-pipelinee-objectivese-loss-ablatione-imagenete-segmentatione-depthe-qualitativee-discussion

3. Research context

We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.

Catalog dimensionRecorded classification
Major categoryFoundational work
ArchitectureNot applicable
Prediction paradigmNot applicable
QuadrantNot applicable

This table preserves the labels recorded at reading time. The current major category is Components of WAMs. View the current classification.

3.1 Evidence-based assessment

Supports the recorded classification

Reader analysis

The recorded foundational visual-encoder classification is supported. DINOv2 supplies reusable perception representations. Its masked-patch targets concern the same image, and video action recognition predicts class labels. No world-transition predictor, inverse-dynamics action extraction or joint future/action model is specified, so the architecture, prediction-paradigm and quadrant entries remain Not applicable. e-objectivese-domain-videoe-discussion

This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.

4. Problem formulation

4.1 Inputs and outputs

InputsOutputs
  • Images from curated and web-derived sources; augmented global/local crops during pretraining
  • Task-labeled examples for supervised downstream probes
  • Global class-token embeddings and spatial patch-token embeddings
  • Task-head predictions or feature-similarity rankings

4.2 Equations and their role

LDINO=−∑ptlog⁡ps\mathcal{L}_{\mathrm{DINO}}=-\sum p_t\log p_s
The source's unnumbered cross-entropy matches teacher prototype probabilities p_t to student probabilities p_s across different crops of one image. e-objectives
LiBOT=−∑iptilog⁡psi\mathcal{L}_{\mathrm{iBOT}}=-\sum_i p_{ti}\log p_{si}
Here i indexes masked student patches; p_ti and p_si are the corresponding teacher and student prototype distributions. Targets describe image patches, not future states. e-objectives
Lkoleo=−1n∑i=1nlog⁡dn,i,dn,i=min⁡j≠i∥xi−xj∥2\mathcal{L}_{\mathrm{koleo}}=-\frac{1}{n}\sum_{i=1}^{n}\log d_{n,i},\qquad d_{n,i}=\min_{j\ne i}\lVert x_i-x_j\rVert_2
For n normalized feature vectors x_i, d_n,i is the distance to the nearest other vector. Minimizing the loss encourages separation of nearby embeddings. e-regularization

5. Method in detail

5.1 Curation changes the learning problem before any gradient update

Reader analysis

Begin with the distinction between images and the distribution of images. DINOv2 does not simply feed every crawled image into the same objective. Curated examples become visual queries, while duplicate removal reduces repeated content in the candidate pool. Retrieval and cluster sampling then concentrate the pretraining corpus around a broad collection of visual domains. Reader interpretation: this injects a useful prior through dataset choice even though class labels never enter the self-supervised loss. Table 2 tests the practical value of the entire distribution-building procedure at a matched image count and training schedule. Its iNaturalist gains support transfer beyond the query domains, but the ADE20k exception prevents a universal claim that curated data always produces better features. Appendix Tables 15 and 18 remain necessary for understanding which domains influenced selection. e-pipelinee-data-detailse-data-compositione-dataset-usee-curation-ablation

5.2 Teacher agreement needs both a global target and a spatial target

Reader analysis

Follow one training image through two complementary comparisons. Different crops produce class-token distributions that DINO asks to agree, encouraging an image-level representation that survives augmentation. At the same time, a masked student patch predicts the corresponding visible teacher patch distribution through iBOT. The teacher is an exponential moving average of the student, and separate heads allow the two objectives to use different projections. Sinkhorn–Knopp normalizes teacher targets, while KoLeo spreads normalized global features using nearest-neighbor distances. Reader interpretation: the objectives preserve different forms of useful structure rather than optimizing one downstream score directly. Table 3 is consistent with this account: removing KoLeo hurts retrieval strongly, whereas removing masked image modeling hurts linear segmentation. Small cross-task decreases and missing uncertainty keep this interpretation conditional. e-objectivese-regularizatione-traininge-loss-ablation

5.3 Distillation produces an encoder; downstream decoding remains a separate stage

Reader analysis

After large-model pretraining, the teacher changes roles. The ViT-g becomes fixed, a smaller student imitates it without patch masking or stochastic depth, and the student's EMA supplies the final encoder. Figure 5 shows a useful distilled ViT-L, but Tables 16 and 17 warn that this comparison also changes optimization and feed-forward-layer choices. At deployment, the output is still a set of visual features. A classifier maps global features to labels, a segmentation or depth head decodes patch features, and retrieval directly compares embeddings. Frozen refers to the backbone during these task evaluations; it does not mean that every downstream component is untrained. Video action recognition similarly classifies aggregated frame features. Reader interpretation: these results motivate DINOv2 as a perception component, while leaving action selection and environment dynamics to another system. e-distillatione-distill-ablatione-traininge-probinge-retrievale-segmentatione-depthe-domain-videoe-discussion

5.4 Training and inference

During training

Source description

The scratch ViT-g uses 1536-dimensional embeddings, 24 attention heads and 40 blocks. Table 16 specifies 625k AdamW iterations, batch 3072, learning rate 3.5×10⁻⁴, 100k warmup and EMA momentum 0.994→1. KoLeo has weight 0.1 and uses only first-global-crop class tokens within each GPU. e-efficiencye-traininge-regularization

Source description

A final 10k-iteration phase adapts to 518×518 images with compressed schedules and a reduced, unspecified base learning rate. Distilled models use batch 2048 and learning rate 10⁻³. The reported implementation uses A100 GPUs and PyTorch 2.0; its claimed 2× speed and one-third memory are relative to the authors' iBOT implementation. e-traininge-regularizatione-efficiency

During inference

Reader analysis

Encode an image once and expose global or patch features. Frozen-backbone evaluations train linear classifiers, segmentation heads or depth decoders; retrieval ranks cosine similarities directly. Video recognition aggregates eight frame embeddings. There is no predicted action to execute, recurrent environment state or planning feedback step. e-probinge-retrievale-segmentatione-depthe-domain-videoe-discussion

5.5 Implementation flow

  1. Build the image distribution

    Embed queries and web images with a self-supervised ViT-H/16 pretrained on ImageNet-22k. Remove duplicates, then retrieve nearby images or sample associated clusters. Curated sources guide the distribution without supplying class labels to the representation loss. e-pipelinee-data-detailse-data-composition

  2. Match global and local teacher targets

    Different crops of one image supply class tokens to DINO heads. For iBOT, the student receives masked patches and predicts the teacher's visible-patch targets. Student weights are optimized; the teacher tracks the student by EMA. Separate heads serve the two objectives, with Sinkhorn–Knopp teacher normalization and student softmax. e-objectivese-training

  3. Keep features spread and training efficient

    KoLeo discourages nearby normalized class embeddings from collapsing together. Sequence packing uses a block-diagonal attention mask to isolate crops; memory-efficient attention, skipped stochastic-depth residuals and sharded training state make the large encoder practical. e-regularizatione-efficiency

  4. Distill the large encoder

    A frozen ViT-g supervises smaller students. Masking and stochastic depth are removed, iBOT applies to both global crops, and an EMA of the student becomes the final model. Figure 5 reports ViT-L ImageNet accuracy rising from 84.5 to 86.3; this comparison also changes recipe and feed-forward-layer choices. e-distillatione-distill-ablation

6. Experiments & results

DINOv2 learns reusable image and patch representations through curated image data, DINO/iBOT self-supervision, efficient large-model training and distillation. Its strongest evidence concerns frozen visual features used by task-specific heads: ImageNet linear accuracy reaches 86.5%, while dense prediction and retrieval expose complementary benefits of the patch objective and feature-spreading regularizer. These are perception results, with no demonstrated action-conditioned dynamics or control loop.

6.1 Read the original evidence

Table 4. Frozen DINOv2-g features are competitive with the strongest listed text-supervised encoders. Original paper, p. 12 ↗

Excerpt from the authors’ paper; cropped without altering the figure or table.

How to read it. Keep the kNN validation column separate from the three linear columns. The latter report ImageNet validation, ReaL and V2 accuracies after fitting classifiers on frozen features. The rows vary architecture, pretraining dataset and text supervision; DINOv2 occupies the four bottom rows. The default input resolution is 224×224, with the CLIP 336 exception shown in its architecture label. The asterisk on EVA-CLIP's custom dataset refers to a custom mixture described in its cited paper, not a shared training set. Appendix B.3 selects each probe's learning rate, output-layer aggregation and pooled-patch option by highest validation accuracy. e-imagenete-probing

What it supports. DINOv2-g reaches 86.5% validation accuracy against OpenCLIP-G's 86.2% and iBOT-L's 82.3%. Its ReaL and V2 scores are 89.6% and 78.4%. The result establishes strong frozen-feature classification under this evaluation procedure, with improvements of 0.3 and 4.2 validation percentage points over those two references.

Where the evidence stops. Common evaluation code does not make pretraining compute, architecture or data equivalent. The table concerns supervised probe fitting, not zero-shot label prediction. It also supplies no uncertainty interval for the small advantage over OpenCLIP.

Table 10. Patch tokens expose segmentation information to simple heads, but evaluation choices matter. Original paper, p. 15 ↗

Excerpt from the authors’ paper; cropped without altering the figure or table.

How to read it. Within each dataset, compare like columns before moving across methods. The lin. setting fits a linear layer to patch tokens, then upsamples its low-resolution predictions. The +ms setting concatenates the last four layers, uses image resolution 640 and adds multiscale test-time augmentation. Thus +ms is a combined evaluation recipe rather than one additional architectural block. Values in parentheses below the dataset names are separately cited state-of-the-art reference results, not another frozen-feature row. The method text also describes a ViT-Adapter with Mask2Former; that separately trained setup is outside the table's linear/+ms columns. e-segmentation

What it supports. On ADE20k, DINOv2-g achieves 49.0 mIoU with the linear head, compared with 39.3 for OpenCLIP-G and 44.6 for iBOT-L. Its +ms score is 53.0. ViT-L narrowly leads the DINOv2 ADE20k +ms column at 53.1, showing that larger capacity does not guarantee the highest entry.

Where the evidence stops. Labels train the downstream segmentation heads. The +ms gain cannot be attributed solely to multiscale testing because feature aggregation and resolution also change. The separate 60.2 adapter/Mask2Former result should not be presented as linear probing.

Figure 10. Qualitative patch matches reveal reusable semantic structure across appearance changes. Original paper, p. 19 ↗

Excerpt from the authors’ paper; cropped without altering the figure or table.

How to read it. Inspect each image pair and trace a colored line to its two endpoints. Lines connect selected patch correspondences; they are not motion trajectories or confidence curves. The examples span vehicles, birds and airplanes, elephants, and drawings paired with animals. Section 7.5 first filters foreground patches using PCA, then forms Euclidean distances between features, solves an assignment problem and applies non-maximum suppression to retain salient matches. This processing matters: the displayed lines are the output of a selection procedure over features, rather than a direct prediction of all correspondences by a separately trained matching head. e-qualitativee-discussion

What it supports. Several displayed matches connect regions with related semantic roles across large changes in texture, style and pose. These examples make the usefulness of spatial features tangible and complement the dense-prediction tables. Their evidence is qualitative: they show selected successful relationships without supplying correspondence accuracy or a distribution of failures.

Where the evidence stops. Foreground filtering and match suppression shape what is visible. No quantitative matching benchmark or physical interaction test accompanies this figure, so semantic correspondences cannot establish reliable geometry, causal understanding or action-conditioned prediction.

6.2 Results and evaluation conditions

Task & protocolReported resultComparison & interpretation
ImageNet-1k frozen linear classification

ImageNet training labels; frozen ViT-g/14, 224×224; best validation-selected linear probe. ReaL/V2 are additional evaluation sets.

86.5 validation; 89.6 ReaL; 78.4 V2

Top-1 accuracy (%)

OpenCLIP-G: 86.2/89.4/77.2; iBOT-L validation: 82.3.

Validation gains are 0.3 and 4.2 percentage points, respectively. Shared evaluation code does not equal matched architecture or pretraining data. e-imagenete-probing

Pretraining data ablation on iNaturalist 2018

ViT-g/14; curated versus random 142M images from the same source, equal shortened training, no high-resolution adaptation; downstream linear evaluation.

82.3

Top-1 accuracy (%)

Uncurated: 68.0; ImageNet-22k: 81.1.

Curation improves this transfer task by 14.3 points over random data, but does not win every metric in Table 2. e-curation-ablation

KoLeo ablation on Oxford-M retrieval

Table 3(a), equal shortened training; frozen-feature instance retrieval on the Oxford Medium protocol.

63.9 with KoLeo

mAP (points)

55.6 without KoLeo.

An 8.3-point improvement supports the feature-spreading mechanism for retrieval; uncertainty is unreported. e-loss-ablation

iBOT masked-image-modeling ablation on ADE20k

Table 3(b), equal shortened training; frozen features with a linear segmentation layer.

47.1 with MIM

mIoU (points)

44.2 without MIM.

The 2.9-point gain supports retaining spatial information. Oxford-M retrieval slightly decreases in the same ablation. e-loss-ablation

ADE20k frozen-feature segmentation

Task-supervised linear head; separate linear and boosted +ms protocols. The source does not restate detailed split construction here.

49.0 linear; 53.0 +ms

mIoU (points)

OpenCLIP-G: 39.3/46.0; iBOT-L: 44.6/47.5.

The boosted setting changes layer aggregation, resolution and test augmentation together. A separate adapter/Mask2Former setup reaches 60.2; it is not the linear result. e-segmentation

NYUd depth estimation with a frozen backbone

NYUd evaluation following the cited depth protocol; DPT decoder trained over frozen ViT-g/14 features. Detailed split definitions are delegated to prior work.

0.279

RMSE (lower is better)

OpenCLIP-G: 0.414; iBOT-L: 0.358, also with DPT.

This measures task-trained depth decoding from frozen features, not unlabeled metric-depth prediction or physical control. e-depth

Oxford-Hard instance retrieval

Frozen features; cosine-similarity ranking on the Oxford Hard protocol.

54.0 for ViT-L/14; 52.3 for ViT-g/14

mAP (points)

OpenCLIP-G: 19.7; iBOT-L: 12.7.

Transfer is strong, but the largest model is not best on this retrieval task. e-retrieval

6.3 Ablations and diagnostic examples

Read component removals and qualitative examples within their stated evaluation conditions.

Table 2. Controlled data-source comparisons support curation, with a visible segmentation exception. Original paper, p. 8 ↗

Excerpt from the authors’ paper; cropped without altering the figure or table.

How to read it. Read rows as alternative pretraining datasets and columns as downstream evaluations. The most direct curation comparison is between Uncurated data and LVD-142M: both contain 142 million images from the same underlying web source, and both train ViT-g/14 for the same shortened number of iterations. High-resolution adaptation is absent. Compare each column independently because accuracy, segmentation mIoU and retrieval mAP are different metrics. The two ImageNet-22k rows provide additional reference distributions, including a variant excluding ImageNet-1k synsets. iNaturalist and Places205 are especially informative because Table 18 marks them as outside the curation query sources. e-curation-ablatione-dataset-use

What it supports. LVD improves iNaturalist 2018 accuracy from 68.0 to 82.3 and ImageNet-A from 59.4 to 73.9 relative to random web data. Yet ADE20k is 47.7 for LVD versus 48.5 for uncurated data. The source's boldface on 47.7 does not mark the largest value in that column.

Where the evidence stops. This experiment supports the complete curation procedure, not an isolated retrieval or deduplication component. It uses a shortened schedule, and its absolute scores should not be substituted for the final-model tables. No repeated-run uncertainty is shown.

Table 3(a–b). The global spreading loss and masked-patch objective help different downstream tasks. Original paper, p. 9 ↗

Excerpt from the authors’ paper; cropped without altering the figure or table.

How to read it. Each panel changes one loss component from the authors' stronger recipe. A cross marks omission and a check marks inclusion. In the left panel, start at Oxford-M: KoLeo changes retrieval from 55.6 to 63.9 mAP. In the right panel, start at ADE20k: masked image modeling changes segmentation from 44.2 to 47.1 mIoU. The ImageNet columns measure linear-classifier accuracy, whereas Oxford-M uses nearest-neighbor retrieval. Section 4 explains the asymmetry: KoLeo separates normalized class-token embeddings, while iBOT trains masked patch tokens against visible teacher targets. Appendix B.1 further specifies that KoLeo sees examples only within each GPU. e-loss-ablatione-objectivese-regularization

What it supports. The selective gains are consistent with complementary representation pressures: KoLeo adds 8.3 mAP points on retrieval, and iBOT adds 2.9 mIoU points on segmentation. Their benefits are not universal. KoLeo changes ADE20k from 47.2 to 47.1, and iBOT changes Oxford-M from 64.3 to 63.9.

Where the evidence stops. These shortened equal-schedule runs lack uncertainty estimates. They support task-dependent benefits, but cannot establish that tiny decreases are statistically meaningful or that identical gains persist at another batch size, data scale or training duration.

7. Analysis & limitations

7.1 What the evidence leaves open

Reader analysis

Pretraining is image-only, while many downstream heads require labels. Several evaluation domains guide retrieval, so broad transfer does not establish independence from all benchmark domains. Main cross-method tables vary architecture and data; the selected results omit confidence intervals. e-dataset-usee-probinge-imagenete-loss-ablation

Reader analysis

Table values limit broad prose claims: uncurated data beats LVD on ADE20k (48.5 versus 47.7), and iBOT beats DINOv2-g on SSv2 (38.7 versus 38.3). DINOv2 also trails OpenCLIP on ImageNet-R, Sketch and Places205. e-curation-ablatione-domain-video

Reader analysis

Dollar Street still shows income and regional disparities. Table 12's Africa/Europe scores are 74.0/89.7; the prose's 25.7% difference does not match their 15.7-point gap. The narrower harmful-label study cannot establish general fairness. Selected patch correspondences likewise lack a quantitative accuracy measure. e-fairnesse-label-biase-qualitative

7.2 Questions for discussion

  1. Would gains from data curation persist when every evaluation-domain query source is withheld?
  2. Can the KoLeo retrieval benefit survive changes in per-GPU batch size without weakening patch-level decoding?

8. Reproducibility audit

8.1 Requirements and known gaps

Reader analysis

Exact recreation requires the underlying web corpus and query mixture, copy-detection and retrieval encoders, deduplication thresholds, training schedules and probe search. The PDF leaves the web repository unnamed, conflicts on 1.2B versus 1.3B starting images, and omits the reduced adaptation learning rate. Appendix A.4 leaves its k=32 query dataset unnamed; Tables 15/18 disagree on whether CIFAR retrieval is represented in the composition. e-pipelinee-data-detailse-data-discrepanciese-data-compositione-dataset-usee-training

Reader analysis

Preserve the probe's 12,500 SGD iterations and search over learning rate, output layers and pooled patches. Table 14's 22,016 GPU-hours and 3.7 tCO2eq are a modeled reproduction estimate, not measured emissions; its A100-40GB label conflicts with nearby A100-80GB prose. Proposed controlled checks appear in the illustrated edition. e-probinge-carbon

8.2 Proposed reproduction checks

The following checks are proposals motivated by the paper. They have not been run as part of this reading.

Check 1: Does KoLeo selectively improve retrieval under controlled batching?

Proposed check, not performed: train matched small-scale DINO/iBOT runs with KoLeo weight 0 versus the stated 0.1. Hold image subset, crops, masking, optimizer, iteration count and per-GPU batch size fixed; repeat with several seeds. Evaluate frozen features on Oxford-M retrieval and ADE20k linear segmentation using identical probe settings. Compare the distribution of retrieval gains and segmentation changes. Failure to obtain a consistent retrieval advantage would challenge transfer of Table 3's mechanism to the chosen scale. Keep the within-GPU neighbor pool fixed because changing it changes the regularizer itself. e-regularizatione-loss-ablatione-training

Check 2: Does curation help when evaluation-domain queries are withheld?

Proposed check, not performed: at one explicitly reduced image and compute budget, compare random web sampling, curation with the full permitted query mixture, and curation after removing a selected evaluation domain's queries, such as ADE20k. Keep the similarity encoder, deduplication procedure, model, training schedule and downstream head identical; retain iNaturalist as an evaluation-only reference domain. Repeat selection and training seeds. If improvement disappears only on the removed-query domain, the evidence would favor domain targeting over a universal diversity explanation. Report this as a mechanism check, not a reproduction of the paper's absolute scores. e-pipelinee-data-detailse-data-compositione-dataset-usee-curation-ablation

8.3 Reading coverage

Visual audit: Visually inspected the title/author/affiliation page; all original Figures 1–10 and Tables 1–18; objective equations, implementation and evaluation prose; fairness and modeled carbon estimates; and Appendices A–C, including data splits, hyperparameters and architecture details. These pages cover the source facts used by the report, visual explanations and proposed checks. All six final crops were separately viewed. Figure 3's arrows were checked against its caption and Section 3; objective masking and EMA details were checked in Section 4 and Appendix B.1. The report preserves relevant data-description, boldface and prose/table inconsistencies. Reference-only pages 22–28 were read as text but not visually inspected. No separate supplement, code, checkpoint or experiment was reviewed.

PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 29, 30, 31, 32. Appendix coverage: reviewed.

Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.

Text reading scope & known omissions
  • Abstract
  • 1 Introduction
  • 2 Related Work
  • 3 Data Processing
  • 4 Discriminative Self-supervised Pre-training
  • 5 Efficient implementation
  • 6 Ablation Studies (6.1–6.6)
  • 7 Results (7.1–7.5)
  • 8 Fairness and Bias Analysis (8.1–8.2)
  • 9 Estimating the Environmental Impact of Training our Models
  • 10 Future work and Discussion
  • Acknowledgments
  • References
  • A Data Processing (A.1–A.4)
  • B Implementation Details (B.1–B.3)
  • C List of Datasets used

Outside the original text pass

  • Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
  • Separate supplemental material availability has not been fully verified.
  • All nine supplied chunks were read, covering the complete 32-page PDF. Original figures and tables were separately inspected on PDF pages 1–21 and 29–32; reference-only pages 22–28 were read as text.
  • Identity/version note: the inspected source is arXiv:2304.07193v2, dated 2 February 2024, with a TMLR 01/2024 header. Its title and OpenReview identifier match the catalog. The author sequence substantially agrees, but the PDF credits Mahmoud Assran where the catalog says Mido Assran, and prints Hervé Jegou where the catalog says Hervé Jégou. Metadata preserves the observed title-page spelling. No other edition or revision history was supplied or compared.
  • Code, checkpoints, datasets, linked references and separate supplements were not inspected; no experiments were reproduced.

The visual audit above records the subsequent illustrated pass.

8.4 Traceable evidence

e-identityPDF p. 1, title, author block, affiliation footnote and arXiv marginInspect

The title is DINOv2: Learning Robust Visual Features without Supervision. The artifact is arXiv:2304.07193v2, 2 February 2024, bearing the TMLR 01/2024 publication header and the catalog's OpenReview identifier. The 26-author list includes Mahmoud Assran and spells Hervé Jegou without the catalog's acute accent on the surname. Affiliations and co-affiliations are stated in the footnote.

Go to primary source ↓
e-problemPDF pp. 1–3, Abstract and Section 1Inspect

The goal is general-purpose frozen image- and patch-level features learned from images without text supervision. The paper combines curated data, scaled discriminative self-supervision and distillation, and evaluates visual understanding tasks.

Go to primary source ↓
e-pipelinePDF pp. 4–5, Figure 3 and Section 3Inspect

Curated and uncurated images are embedded; web images are deduplicated and retrieved by visual similarity to curated queries. Section 3 specifies a self-supervised ViT-H/16 pretrained on ImageNet-22k, cosine similarity, clustering and Faiss. Data processing is reported to take less than two days on 20 nodes with eight V100-32GB GPUs each.

Go to primary source ↓
e-data-detailsPDF p. 29, Appendix A.2–A.4Inspect

Copy-detection embeddings yield a 64-neighbor graph; similarity >0.6 defines self-deduplication components, retaining one representative. Relative deduplication uses >0.45 and removes components matching evaluation train/test images. Sample retrieval and 100,000-cluster retrieval are described; the latter samples 10,000 images per cluster with more than three query images and caps each dataset retrieval at one million.

Go to primary source ↓
e-data-compositionPDF p. 30, Table 15, dataset/split, retrieval and final columnsInspect

LVD-142M totals 142,109,386 images. It contains ImageNet-22k directly and retrieved images, ImageNet-1k train-based retrieval, Google Landmarks v2, Mapillary and capped retrievals for several fine-grained and dense tasks. The ImageNet-1k row records 40,997,344 retrieved images from 1,281,167 queries.

Go to primary source ↓
e-dataset-usePDF p. 32, Table 18, retrieving-pretraining-data and evaluation columns; PDF p. 30, Table 15Inspect

Many evaluation domains also provide retrieval queries. iNaturalist 2018/2021, Places-205, video datasets and fairness datasets are marked evaluation-only. Table 18 marks CIFAR-10/100 as retrieval sources, although they have no rows in Table 15.

Go to primary source ↓
e-data-discrepanciesPDF p. 4, automatic data curation paragraph; p. 5, Section 3; p. 29, Appendix A.3–A.4; p. 30, Table 15Inspect

The related-work paragraph says no pretrained encoders are used, whereas Section 3 explicitly specifies a pretrained self-supervised ViT-H/16. Section 3 reports 1.2B unique images; Appendix A.3 instead starts from 1.3B, then reports 1.1B after self-deduplication and 744M after relative deduplication. A.4 leaves the query dataset for k=32 unnamed; Table 15 provides the ImageNet-1k retrieval counts. These descriptions are not silently reconciled.

Go to primary source ↓
e-objectivesPDF pp. 5–6, Section 4, image-level and patch-level objectives, displayed unnumbered lossesInspect

DINO matches teacher and student prototype distributions from different crops of one image. iBOT matches student masked-patch predictions to visible teacher patch targets. Student parameters are learned, with an EMA teacher. DINO and iBOT use separate projection heads; teacher normalization uses three Sinkhorn–Knopp iterations and student outputs use softmax.

Go to primary source ↓
e-regularizationPDF p. 6, Section 4, KoLeo equation and adapting resolution; p. 29, Appendix B.1, KoLeo regularizationInspect

KoLeo is minus the average log nearest-neighbor distance among normalized feature vectors. Its weight is 0.1, applied to first-global-crop class tokens within each GPU, without cross-GPU communication. The final pretraining phase uses 518×518 images.

Go to primary source ↓
e-efficiencyPDF pp. 6–7, Section 5, attention, sequence packing, stochastic depth and FSDP; p. 31, Table 17Inspect

The paper reports approximately twice the speed and one-third the memory of its iBOT reference implementation on the same hardware, using A100 GPUs and PyTorch 2.0. ViT-g has 1.1B parameters, embedding dimension 1536, 24 attention heads and 40 blocks. Packed crops are separated by a block-diagonal attention mask. Dropped residuals are skipped; FSDP shards training state and uses mixed-precision communication.

Go to primary source ↓
e-distillationPDF p. 7, Section 5, model distillation; p. 31, Tables 16–17Inspect

Smaller models use a frozen ViT-g teacher; a separate EMA of the student becomes the evaluated model. Masking and stochastic depth are removed and iBOT is applied to both global crops. Distilled models use MLP feed-forward layers, while scratch-trained L/g models use SwiGLU.

Go to primary source ↓
e-trainingPDF p. 31, Table 16 and Appendix B.1–B.2Inspect

Table 16 reports 625k AdamW iterations. Scratch ViT-L/g use batch 3072, learning rate 3.5e-4 and drop rate 0.4; distilled S/B/L use batch 2048, learning rate 1e-3 and no drop. The caption gives 100k learning-rate warmup, weight decay 0.04→0.2, EMA momentum 0.994→1 and float16 with float32 DINO-head gradient reduction. High-resolution adaptation lasts 10k iterations with compressed schedules and an unspecified reduced base learning rate.

Go to primary source ↓
e-probingPDF p. 11, Section 7.1; p. 31, Appendix B.3Inspect

ImageNet evaluation freezes the backbone. Linear heads train with SGD for 12,500 iterations and random-resized crops. A grid varies 13 learning rates, one versus four output layers, and inclusion of average-pooled patch features; the highest validation accuracy is reported.

Go to primary source ↓
e-recipe-ablationPDF p. 8, Table 1 and Section 6.1Inspect

The sequential ViT-L/ImageNet-22k recipe ablation moves from the authors' iBOT reproduction at 74.5 k-NN / 83.2 linear to DINOv2 at 82.0 / 84.5. Some additions lower linear accuracy. The authors prioritize k-NN and training stability; the cumulative sequence is not a factorial causal isolation.

Go to primary source ↓
e-curation-ablationPDF p. 8, Table 2 and Section 6.2Inspect

ViT-g/14 models use equal, shortened iteration schedules without high-resolution adaptation. LVD-142M and a random 142M subset of the same web source give iNat2018 accuracy 82.3 versus 68.0 and ImageNet-A 73.9 versus 59.4. ADE20k is 47.7 versus 48.5, despite boldface on 47.7; ImageNet-22k reaches 85.9 on ImageNet-1k versus 85.8 for LVD.

Go to primary source ↓
e-loss-ablationPDF p. 9, Table 3(a–b), caption and Section 6.4Inspect

With shortened equal schedules, adding KoLeo raises Oxford-M retrieval mAP from 55.6 to 63.9, while ADE20k changes 47.2→47.1. Adding iBOT masked image modeling raises ADE20k linear segmentation mIoU from 44.2 to 47.1, while Oxford-M changes 64.3→63.9. Classification columns are accuracy percentages; no uncertainty intervals are reported.

Go to primary source ↓
e-distill-ablationPDF pp. 9–10, Section 6.5 and Figure 5; p. 31, Tables 16–17Inspect

Figure 5 reports distilled ViT-L/14 ImageNet-1k accuracy 86.3 against 84.5 for scratch ViT-L/14, with the ViT-g teacher at 86.5, and gains across its displayed benchmarks. Training and FFN choices also differ between the scratch and distilled configurations.

Go to primary source ↓
e-resolutionPDF p. 10, Figure 6 and Section 6.6Inspect

The resolution ablation uses ViT-L/16 on ImageNet-1k, comparing fixed 224, fixed 416, and 224 followed by 10k iterations at 416. The brief high-resolution phase approaches the fixed-high-resolution behavior. This is a smaller experimental setup than final LVD-142M training at 518.

Go to primary source ↓
e-imagenetPDF p. 12, Tables 4–5; p. 11, Section 7.1Inspect

Table 4 at 224×224 reports frozen DINOv2-g linear top-1 86.5/89.6/78.4 on ImageNet validation/ReaL/V2, versus OpenCLIP-G 86.2/89.4/77.2; iBOT-L validation is 82.3. Architectures and pretraining data differ. Table 5 reports DINOv2-g supervised finetuning 88.5 at 224 and 88.9 at 448, distinct from linear probing.

Go to primary source ↓
e-domain-videoPDF p. 13, Tables 6–7 and Section 7.2; p. 12, robustness paragraphInspect

Frozen DINOv2-g is below OpenCLIP-G on ImageNet-R (78.8 versus 87.8), Sketch (62.5 versus 66.4) and Places205 (67.5 versus 69.8). Video classification uses eight evenly spaced frame features, averaging for UCF/K400 and concatenating for SSv2. Table 7 gives SSv2 38.3 for DINOv2-g, 38.7 for iBOT and 35.8 for OpenCLIP, limiting the prose claim of a new SSL best across video tasks.

Go to primary source ↓
e-retrievalPDF p. 14, Table 9, Oxford/H column and Section 7.3Inspect

Instance retrieval ranks database images by cosine similarity to query features. Oxford-Hard mAP is 54.0 for DINOv2-L, 52.3 for DINOv2-g, 19.7 for OpenCLIP-G and 12.7 for iBOT-L. The largest DINOv2 model does not lead every retrieval column.

Go to primary source ↓
e-segmentationPDF p. 15, Table 10 and Section 7.4, semantic segmentation and frozen-backbone pipelineInspect

DINOv2-g ADE20k mIoU is 49.0 for linear and 53.0 for +ms, versus OpenCLIP-G 39.3/46.0 and iBOT-L 44.6/47.5. The +ms recipe concatenates the last four patch-token layers, uses 640 resolution and multiscale test augmentation. Parenthesized scores below the dataset names in the header are separately cited state-of-the-art results. A separately trained ViT-Adapter/Mask2Former with frozen ViT-g reaches 60.2 mIoU.

Go to primary source ↓
e-depthPDF pp. 15–16, Section 7.4 and Table 11, NYUd/DPT columnInspect

Depth evaluation distinguishes one-layer and four-layer linear predictors from a DPT decoder, all on frozen features. DINOv2-g NYUd DPT RMSE is 0.279, compared with OpenCLIP-G 0.414 and iBOT-L 0.358. Table 11 labels transfer NYUd→SUN RGB-D; Section 7.4 initially calls the destination SUN3d. The report follows the table label.

Go to primary source ↓
e-qualitativePDF pp. 17–19, Section 7.5, Figures 7–10 and patch-matching paragraph; p. 2, Figure 1Inspect

Dense prediction examples compare linear heads. PCA foreground filtering precedes patch matching; Euclidean feature distances define an assignment, then non-maximum suppression retains salient matches. Figure 10 shows vehicles, birds/airplanes, elephants and drawings/animals with selected correspondence lines, without quantitative correspondence accuracy.

Go to primary source ↓
e-fairnessPDF pp. 18–19, Section 8.1 and Table 12Inspect

Dollar Street evaluates household concepts across income and geography. Table 12 gives DINOv2 scores 74.0 for Africa and 89.7 for Europe, and 67.4/90.5 for low/high income. The prose's 25.7% regional and 31.7% income differences do not equal the table's 15.7- and 23.1-point gaps. The authors acknowledge Western and wealthy-household bias.

Go to primary source ↓
e-label-biasPDF pp. 19–20, Section 8.2 and Table 13Inspect

A frozen-backbone classifier over 619 ImageNet-22k classes evaluates 2,955 Casual Conversations images with top-five labels filtered at probability 0.1. The authors describe two harmful-label instances and acknowledge that broader bias evaluation may reveal additional flaws.

Go to primary source ↓
e-carbonPDF pp. 20–21, Table 14 and Section 9Inspect

Table 14 gives a modeled reproduction cost of 22,016 GPU-hours, 9.7 MWh and 3.7 tCO2eq, assuming 400W, PUE 1.1 and 0.385 kg CO2eq/kWh. It lists A100-40GB, while nearby prose references A100-80GB. These are normalized potential emissions, not measured site emissions; the project estimate excludes manufacturing and disposal.

Go to primary source ↓
e-discussionPDF p. 21, Section 10Inspect

The authors propose future scaling and language-enabled use of visual features. The reported system is a family of image encoders; no action-conditioned dynamics model, action extraction procedure or closed-loop control experiment is introduced.

Go to primary source ↓

8.5 Primary sources

Scroll across the image to inspect details. Press Esc to close.