A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark
1. Paper overview
In one sentence: VTAB measures representation quality through adaptation to diverse unseen tasks, exposing how upstream supervision, tuning budget and the transfer strategy jointly determine apparent progress. e-identitye-protocole-taskse-modelse-heavy-resultse-lineare-validation-conflict
| At a glance | What to know |
|---|---|
| Research problem | Source description Single-dataset accuracy, linear probes and image-generation metrics do not directly measure whether a representation makes new visual tasks easier to learn. VTAB targets sample-efficient adaptation across natural images, specialist imagery and structured scene properties, using a common interface that permits comparable evaluation. e-identitye-protocole-tasks |
| Core mechanism | Source description The benchmark defines representation utility through adaptation performance on unseen tasks, forbids upstream exposure to their images, and standardizes 19 tasks as classification problems. This makes transfer strategy part of the evaluated algorithm. e-protocole-tasks |
| A key reported result | VTAB-1k with lightweight fine-tuning: Sup-Rotation-100%: 67.5 Mean top-1 accuracy (%). 19 task test sets; nominal 1,000-example regime; median of three runs per task, followed by a task mean. Validation-budget descriptions conflict. Sup-100%: 65.6; Sup-Exemplar-100%: 67.0; From-Scratch: 42.1. Adding rotation supervision improves the tabulated mean by 1.9 percentage points. This is a transfer result, not an action-execution result. e-light-resultse-validation-conflict |
| Reading caution | Reader analysis The finite task set risks meta-overfitting, and converting every target to image-level classification omits dense prediction and interactive control. Human evaluation validates learnability, not human-versus-model parity; its DMLab task removes reward type and uses a different training protocol. e-protocole-taskse-human |
Core contributions
- Source description
The benchmark defines representation utility through adaptation performance on unseen tasks, forbids upstream exposure to their images, and standardizes 19 tasks as classification problems. This makes transfer strategy part of the evaluated algorithm. e-protocole-tasks
- Source description
The study compares 18 supervised, semi-supervised, self-supervised, generative and from-scratch algorithms, and tests sensitivity to tuning, freezing, aggregation, architecture size and evaluation cost. e-setupe-rankinge-capacitye-budget
Figure 1. Representation quality is measured after adapting the same learning procedure to separate unseen tasks. Original paper, p. 2 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start in the dashed box at lower left. Upstream data supplies the model, while the adjacent adaptation algorithm determines how that model learns a new task. The arrows branch into independent task rows; they do not represent sequential control steps. Above them, P_T denotes the desired distribution of tasks. Within each row, the red outline marks adaptation data and the blue outline marks test images. E_T produces that task's score. Equation (1) averages such evaluations over tasks, approximated here by a fixed benchmark suite. Section 2.1 adds the crucial constraint: the upstream stage must not see evaluation-task images, even without labels. e-protocole-taskse-setupe-validation-conflict
What it supports. The protocol evaluates the complete adaptation algorithm, including the learned representation and its transfer procedure. Freezing features and fine-tuning them are therefore different evaluated choices. Its task diversity is intended to test whether a representation helps beyond familiar natural-image categories; the diagram itself does not establish performance.
Where the evidence stops. The displayed 92.4, 96.3 and 79.7 are schematic scores, not reported benchmark results. Also distinguish the intended 1,000-label protocol on p. 4 from Appendix I's conflicting description of regular validation sets.
2. Motivation
2.1 The problem and the proposed response
Single-dataset accuracy, linear probes and image-generation metrics do not directly measure whether a representation makes new visual tasks easier to learn. VTAB targets sample-efficient adaptation across natural images, specialist imagery and structured scene properties, using a common interface that permits comparable evaluation. e-identitye-protocole-tasks
2.2 What this reading follows
A useful visual representation should help when the next task has few labels and differs from its training data. VTAB turns that idea into a benchmark spanning natural images, medical and aerial imagery, and structured scene properties. Its ImageNet study compares supervised, self-supervised and generative representations through task-specific adaptation. Read the protocol first, then compare the two tuning budgets before interpreting the rotation and freezing diagnostics. The strongest lesson is that representation rankings depend on how adaptation is allowed to work. This edition follows the supplied February 2020 revision and preserves its unresolved validation-budget and numerical inconsistencies. e-identitye-protocole-taskse-modelse-heavy-resultse-lineare-validation-conflict
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Benchmarks & simulators |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
3.1 Evidence-based assessment
Supports the recorded classification
The recorded benchmark/evaluation classification is supported. VTAB measures adaptation of visual representations, including perception tasks motivated by control. It defines neither a unified world/action architecture nor a future/action prediction paradigm; architecture, paradigm and quadrant are therefore not applicable, rather than unverified. e-protocole-taskse-structured
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Treat adaptation as part of the algorithm
Equation (1) makes the evaluation target explicit: an algorithm receives a small labelled dataset, produces a prediction function, and is judged on that task's held-out data. The representation is one ingredient in that algorithm. VTAB therefore allows both upstream learning and a flexible transfer procedure, while forbidding prior exposure to the evaluation images. In the reported study, ImageNet supplies the upstream data and each task receives a separately adapted classifier. The intended low-label protocol reserves 800 examples for fitting and 200 for selection before retraining on their union. That accounting matters because hyperparameter selection also consumes labels. Appendix I describes a different validation procedure, so the published numbers must remain attached to an unresolved selection-budget qualification rather than being presented as a fully specified strict-budget experiment. e-protocole-modelse-setupe-validation-conflict
5.2 Read rotation supervision through the tasks it helps
The hybrid models use upstream ImageNet labels together with an auxiliary rotation or Exemplar objective. Their downstream output remains a task label; the auxiliary objective is not a control module. Table 6 shows that rotation supervision helps the overall mean, while Figure 4 locates much of the added benefit in structured tasks such as position and counting. A reader interpretation is that classification pre-training may discard variation useful for another task, and an auxiliary objective may preserve some of it. The figure is compatible with that explanation, but cannot prove it: different optimization histories could also produce the pattern. Heavyweight tuning further changes the picture, with Exemplar leading the four evaluated methods. Read these as evidence about particular representation-and-adaptation combinations, not as a universal ranking of pre-training objectives. e-modelse-light-resultse-rotation-diagnostice-heavy-results
5.3 Separate usable information from immediately linear information
Frozen-feature evaluation asks whether a linear classifier can use the representation without changing it. Fine-tuning asks whether the pretrained network can become useful after task-specific updates. Appendix K holds the lightweight hyperparameters constant while changing which parameters train, and Figure 16 shows that these tests can order models differently. The tabulated supervised mean rises from 57.1% with frozen features to 65.6% with fine-tuning; the source's alternative prose value is disclosed rather than averaged into a new estimate. Reader interpretation: poor linear accessibility need not imply poor adaptability. Conversely, strong fine-tuning does not prove that the initial features explicitly encode a desired property. A meaningful reproduction should compare both transfer strategies under documented label and search budgets, then inspect task-level changes instead of relying only on the aggregate. e-lineare-light-resultse-validation-conflicte-light-sweep
5.4 Training and inference
During training
ImageNet pre-training uses 1.28 million images. Supervised variants use 10% or 100% of labels; hybrid variants add rotation or Exemplar auxiliary losses on all images. These are upstream representation objectives, not inference-time controllers. e-models
Lightweight adaptation uses SGD, momentum 0.9, batch size 512, learning rates 0.1/0.01 and 2,500/10,000 updates. Learning rate decays tenfold at one-third, two-thirds and shortly before completion. Fine-tuning has no weight decay; from-scratch decay is 0.01 times learning rate. Images are resized to 224×224, except generative models at 128×128. e-light-sweep
During inference
After task-specific training, a test image passes through the adapted representation and classifier to produce a class prediction. Frozen-feature evaluation leaves the representation unchanged during adaptation. There is no future-frame rollout, action extraction or closed-loop policy evaluation in this benchmark. e-protocole-taskse-linear
5.5 Implementation flow
- Define the task distribution
Seven natural, four specialized and eight structured tasks approximate the desired distribution. Structured targets include counting, position, orientation and binned depth. DMLab is a frame-classification task; Appendix A specifies six distance/reward classes, whereas Section 2.2 describes navigation time. That semantic difference remains unresolved. e-taskse-structured
- Keep upstream and downstream separate
Upstream learning may use arbitrary data but must exclude evaluation-task images, including unlabelled images. Per-task hyperparameter selection is allowed; task-dependent search spaces and joint training on downstream tasks are forbidden. e-protocole-decathlon-protocol
- Extract a representation
Most evaluated backbones are ResNet50-v2 with 2,048-dimensional features. BigGAN discriminators use a custom ResNet-like network with 1,536-dimensional features. Autoencoders and BigBiGAN supply encoders. Patch models average representations from nine patches after a central crop, with reduced backbone stride. e-modelse-architecture
- Adapt and select
The main protocol trains on 800 examples, selects on 200, then retrains on their union. It fine-tunes the whole network and reports the median of three test runs, averaging across tasks. Appendix I instead describes regular validation sets, so the actual validation budget cannot be resolved from this PDF. e-setupe-validation-conflict
- Compare adaptation regimes
Lightweight transfer searches two learning rates and two schedules. Heavyweight transfer uses 100 random trials over additional choices. Frozen-feature evaluation changes the trainable component to a linear logistic-regression layer while retaining the lightweight hyperparameters. e-light-sweepe-heavy-sweepe-linear
6. Experiments & results
VTAB evaluates how well representations adapt to diverse unseen classification tasks with limited labels. Its ImageNet study finds that combining supervision with self-supervision improves transfer, while tuning budget and the choice between fine-tuning and frozen features materially affect conclusions. The supplied revision also contains unresolved protocol and numerical inconsistencies, documented below.
6.1 Read the original evidence
Table 6, 1000-example block. Combining supervision with rotation leads the lightweight comparison, while task columns reveal uneven transfer. Original paper, p. 21 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the rightmost Mean column first, then move left to the individual tasks. The dots identify natural tasks in red, specialized tasks in green and structured tasks in blue. Each task entry is the median of three test runs; Mean averages those task scores. Sup-Rot and Sup-Ex abbreviate supervision combined with rotation and Exemplar losses. Semi-Rot-10% is the table's name for the hybrid trained with only 10% of upstream labels. This crop includes the complete 1000-example block and excludes the separate full-data block. Rows run toward higher overall scores at the bottom, despite the caption describing best-to-worst sorting. e-light-resultse-light-narrativee-modelse-architecturee-light-sweepe-validation-conflict
What it supports. Sup-Rot-100% reaches 67.5%, compared with 65.6% for Sup-100% and 42.1% from scratch. Semi-Rot-10% reaches 64.8%, versus 61.6% with 10% supervision alone. These means support a benefit from combining the objectives under the reported lightweight settings, while the varied task columns show that no method dominates everywhere.
Where the evidence stops. The main text gives Rotation as 59.6%, while this table gives 59.5%. The validation-budget conflict remains unresolved. Generative models also differ in input resolution and sometimes architecture, so this is not a perfectly isolated comparison of pre-training losses.
Table 1. A larger adaptation search improves mean scores and changes the leading hybrid objective. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Keep the two horizontal blocks separate: the upper block uses the 1000-example setting and the lower block uses full downstream datasets. Rows compare from-scratch training with three fully supervised upstream variants. Column colors retain the natural, specialized and structured grouping. Unlike the lightweight table, this search evaluates only four selected methods, using 100 random configurations per task. Appendix I varies optimizer, batch size, training schedule, preprocessing, evaluation resolution and regularization. First compare the rightmost means within a block. Then inspect individual columns to see where the average conceals failures, such as the weak pretrained performance on smallNORB elevation in the upper block. e-heavy-resultse-heavy-sweepe-light-resultse-validation-conflicte-sweep-discrepancy
What it supports. Sup-Ex-100% leads the tested 1k methods at 72.7%, followed by Sup-Rot-100% at 71.5% and Sup-100% at 71.2%; from scratch reaches 59.2%. In the full-data block, their respective means are 91.4%, 91.3%, 91.1% and 86.4%. These improvements show that adaptation resources are part of the comparison.
Where the evidence stops. This search changes several factors together, so gains cannot be assigned solely to longer training. Appendix I's printed SGD interval is invalid, and its validation-set description conflicts with Section 3.1. The table gives no uncertainty for the small mean-score gaps.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| VTAB-1k with lightweight fine-tuning 19 task test sets; nominal 1,000-example regime; median of three runs per task, followed by a task mean. Validation-budget descriptions conflict. | Sup-Rotation-100%: 67.5 Mean top-1 accuracy (%) | Sup-100%: 65.6; Sup-Exemplar-100%: 67.0; From-Scratch: 42.1. Adding rotation supervision improves the tabulated mean by 1.9 percentage points. This is a transfer result, not an action-execution result. e-light-resultse-validation-conflict |
| Transfer with reduced upstream labels Same lightweight VTAB-1k evaluation; upstream ImageNet label fraction varies. | Semi-Rotation-10%: 64.8 Mean top-1 accuracy (%) | Sup-10%: 61.6; Sup-100%: 65.6. The 3.2-point gain closes 80% of the 4.0-point gap in the reported means. Downstream label availability is not reduced by this comparison. e-light-resultse-light-narrativee-validation-conflict |
| VTAB-1k with heavyweight fine-tuning 100-trial random search per task; four selected methods; test evaluation under the paper's general protocol. | Sup-Exemplar-100%: 72.7 Mean top-1 accuracy (%) | Sup-Rotation-100%: 71.5; Sup-100%: 71.2; From-Scratch: 59.2. Exemplar leads among the four tested methods under this budget. Rotation led the lightweight comparison; the best method is therefore budget-dependent. e-heavy-resultse-heavy-sweepe-light-resultse-validation-conflict |
| Frozen versus fine-tuned supervised transfer VTAB-1k; Sup-100%; lightweight hyperparameters; Table 9 frozen logistic regression versus Table 6 whole-network fine-tuning. | Frozen: 57.1; fine-tuned: 65.6 Mean top-1 accuracy (%) | Section 3.4 instead quotes 57.3 for frozen Sup-100%. The tables imply an 8.5-point gain. The prose/table discrepancy is unresolved; the evidence supports sensitivity to adaptation strategy without an exact reconciliation. e-lineare-light-resultse-validation-conflict |
| Predicting rankings on held-out tasks Appendix N: 16 baseline models; each target task excluded from VTAB and Visual Decathlon aggregates before comparison with its task-specific ranking. | VTAB: 0.76 Average Kendall rank correlation | Visual Decathlon: 0.70. A broader suite better predicts rankings in this tested task/model pool. The transfer experiment uses a modified evaluation setup and is not a fair claim of winning the original Decathlon competition. e-decathlon |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 4. Rotation supervision changes which visual properties transfer, with gains concentrated on several structured tasks. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. The actual page stacks these plots vertically, although its caption calls them left and right. Use the model names beside the right-hand arrows to read the sign: positive bars favor Sup-100% in the top plot and Sup-Rotation-100% in the bottom plot. Red, green and blue identify natural, specialized and structured tasks, as defined in Section 2.2 and the caption. Each plot sorts tasks by its own differences, so matching horizontal positions do not identify the same task. The vertical scales also differ substantially. Compare the pattern of signs and task identities before comparing bar heights, and use Table 6 for exact reported medians. e-rotation-diagnostice-light-resultse-modelse-tasks
What it supports. The top comparison shows large supervised-transfer benefits on many natural tasks, with some structured exceptions. The lower comparison highlights gains on dSprites location, CLEVR counting and DMLab after adding rotation supervision. Table 6 gives dSprites-location scores of 81.0% and 92.5% for supervision alone and its rotation hybrid.
Where the evidence stops. Some small plotted differences do not exactly match subtraction of Table 6's rounded medians; their aggregation is not reconciled. The pattern motivates a spatial-information hypothesis, but does not directly measure what features encode or show that an agent can act successfully.
Figure 16. Freezing the representation can change the ranking of representation-learning methods. Original paper, p. 30 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Each bar belongs to a downstream task and measures agreement between two rankings of the evaluated methods: one from a frozen representation with a trained logistic-regression layer, and one from whole-network fine-tuning. Higher positive Kendall correlation means closer agreement; a negative value indicates reversed ordering more often than agreement. The left panel uses 1000-example datasets and the right uses full datasets. Task order changes between panels. Colors again denote natural, specialized and structured tasks. Appendix K states that linear evaluation uses the lightweight hyperparameters, so the comparison tests the consequences of restricting which parameters adapt under that search. e-lineare-light-sweepe-tasks
What it supports. Many natural tasks show strong agreement, but several specialized or structured tasks do not. Retinopathy is negative in the 1k panel; CLEVR distance is negative in the full-data panel. Thus, a ranking from linear probes can fail to predict which pretrained model adapts best when its representation can change.
Where the evidence stops. These bars are rank correlations, not accuracy gains or uncertainty intervals. Section 3.4 and Table 9 disagree on some mean accuracies, including 57.3% versus 57.1% for frozen Sup-100%. Shared lightweight hyperparameters also do not establish an optimal search for every frozen model.
7. Analysis & limitations
7.1 What the evidence leaves open
The finite task set risks meta-overfitting, and converting every target to image-level classification omits dense prediction and interactive control. Human evaluation validates learnability, not human-versus-model parity; its DMLab task removes reward type and uses a different training protocol. e-protocole-taskse-human
Architecture and preprocessing controls are approximate: BigGAN differs structurally, patch models alter spatial processing, and generative models use lower resolution. Consequently, loss-family comparisons do not perfectly isolate the pre-training objective. e-architecturee-light-sweep
Numerical inconsistencies remain: Rotation is 59.6% in Section 3.2 but 59.5% in Table 6; frozen Sup-100% is 57.3% in Section 3.4 but 57.1% in Table 9; frozen Cond-BigGAN is 43.3% in prose but 42.5% in Table 9. Appendix F mentions 20 tasks despite the 19-task definition. These are not silently harmonized. e-light-narrativee-light-resultse-lineare-subsetse-tasks
Alternative weighting largely preserves rankings in this heterogeneous model pool, but Appendix F warns that subsets may be less reliable for closely matched methods. The authors regard diverse upstream images or video as future work; the reported ImageNet study does not establish those benefits. e-rankinge-subsetse-discussion
7.2 Questions for discussion
- How much of the apparent low-label advantage survives a strictly enforced 800/200 selection budget?
- Which spatial properties explain gains from rotation supervision, and which would survive evaluation on unseen task families?
8. Reproducibility audit
8.1 Requirements and known gaps
Reproduction requires the exact upstream checkpoints, task definitions/splits, image preprocessing and per-task selection procedure. The paper announces released code/data/models but does not supply a complete version-pinned environment or checkpoint fingerprints. Resolve the main-text/Appendix I validation conflict before claiming exact replication. e-releasee-setupe-architecturee-light-sweepe-validation-conflict
Appendix I prints an invalid log-uniform SGD interval [0.001, 0.0], while the sweep graphics show positive learning rates up to 1.0. The last learning-rate drop is specified only as shortly before completion. Both require clarification rather than a guessed repair. e-heavy-sweepe-sweep-discrepancy
The authors report three hours for VTAB-1k on one Nvidia P100 at batch size 64 and learning rate 0.01, and use TPU-v3-16 accelerators to reduce cost. This is a particular evaluation configuration, not total upstream-training or exhaustive-search cost. e-budget
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Measure the cost of the ambiguous validation budget
Reader-proposed check, not performed: freeze the upstream Sup-100% and Sup-Rotation-100% checkpoints and all four lightweight candidate configurations. On all 19 tasks, compare selection using an 800/200 split within 1,000 labels with selection using the same 800 fitting examples plus the regular validation set, keeping the test set sealed. Retrain selected configurations on the same fixed 1,000-image union in both arms so final fitting data stay constant. Repeat paired seeds and report total labels consulted, selected configurations, per-task test deltas and mean rankings. This isolates the effect of selection data rather than exactly reproducing Appendix I. Consistent score or ranking changes would falsify the assumption that the conflicting validation descriptions are immaterial. e-setupe-validation-conflicte-light-sweepe-light-results
Check 2: Test whether rotation's benefit depends on feature adaptation
Reader-proposed check, not performed: train matched ResNet50-v2 representations on the same ImageNet images with full supervision, with and without auxiliary rotation supervision, holding upstream update budget and preprocessing fixed and documenting the auxiliary-loss weight. Cross these two checkpoints with frozen logistic regression and whole-network fine-tuning. Use the strict 800/200 budget and identical lightweight candidate grids for all 19 tasks, treating dSprites location, CLEVR counting and DMLab as prespecified diagnostics and Pets as a natural-image comparison. Report paired seed distributions and the interaction between rotation's gain and the transfer strategy. A gain that appears only after fine-tuning would support adaptation-dependent utility; a persistent frozen-feature gain would support immediate linear accessibility. Neither outcome establishes control ability. e-modelse-architecturee-light-sweepe-setupe-light-resultse-rotation-diagnostice-lineare-structured
8.3 Reading coverage
Visual audit: Visually inspected the title/author/version page, protocol and equation, all 21 numbered figures and all nine tables, including architecture text in Appendix C, hyperparameter specifications and sweep graphics in Appendix I, and compute/schedule evidence on pp. 7 and 32. Every final crop was inspected at its native generated resolution. Figure 1's arrows agree with the adaptation/evaluation formulation and its scores are schematic. Figure 4 is vertically stacked despite left/right caption wording; its scales and sign arrows were checked, and small plot/table differences remain disclosed. Figure 3 includes a full-data panel despite a VTAB-1k-only caption description. Figure 7's caption says validation split while its body describes test metrics; no exact metric-comparison result is retained. No image content was recreated. Reference-only pp. 10–12 were read as text. External code, datasets and separate supplements remain outside this pass.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract
- 1. Introduction
- 2. The Visual Task Adaptation Benchmark (2.1–2.3)
- 3. Large-Scale Study (3.1–3.5)
- 4. Related Work
- 5. Discussion
- Acknowledgments and References
- A. Tasks
- B. Human Evaluation
- C. Network Architectures
- D. Alternative Metrics
- E. Alternative weighting and ranking schemes for models
- F. Representative Subset of Tasks
- G. Lightweight experiments
- H. Heavyweight experiments
- I. Hyperparameter Sweeps
- J. Lightweight versus Heavyweight Searches
- K. Linear Evaluation
- L. Larger Architectures
- M. Budget Analysis
- N. Comparison to Visual Decathlon
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- The reviewed artifact is arXiv:1910.04867v2 [cs.CV], dated 21 February 2020, and marked Preprint on PDF p. 1. The catalog's 2019 year describes an earlier bibliographic record; no earlier revision was supplied or compared. All 17 catalog authors match, with André's accent restored from the title page. The catalog BibTeX uses the shortened title The Visual Task Adaptation Benchmark; the catalog's main title matches this PDF.
- The supplied observedTitle and primary title concatenate the line break as 'theVisual'; these supplied provenance strings are preserved. The visible title reads 'A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark'.
- All ten supplied chunks were read individually, covering all 33 PDF pages. Text extraction does not reconstruct figure images; this was addressed by inspecting the PDF's original figures and tables on pages 1–9 and 13–33. Reference-only pages 10–12 were read as text.
- Separate supplemental material availability has not been fully verified. The supplementary material embedded in this PDF, Appendices A–N, was read; no separate supplement was supplied.
- Code, external references, linked datasets and model releases were not inspected. No experiments were run or reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e-identityPDF p. 1, title, author block, arXiv margin stamp, Abstract
The visible title matches the catalog's main title. Seventeen authors are credited to Google Research, Brain Team, Zürich; Zhai, Puigcerver and Kolesnikov share equal contribution. The stamp identifies arXiv:1910.04867v2, 21 February 2020. The abstract defines transfer to diverse unseen tasks as the evaluation goal.
Go to primary source ↓e-protocolPDF p. 2, Section 2, Eq. (1), Figure 1, Section 2.1
Algorithm A adapts to task data and is scored by E_T; expected performance is over P_T. Figure 1 separates upstream learning, adaptation and testing; its displayed scores are schematic. Evaluation images cannot be used upstream, and search spaces cannot depend on the task. Fixed tasks risk meta-overfitting.
Go to primary source ↓e-tasksPDF p. 3, Sections 2.1–2.3; p. 13, Appendix A, Table 2
The suite contains 19 classification tasks: seven natural, four specialized and eight structured. Table 2 lists dataset sizes and class counts. The classification interface excludes some dense-prediction problems. Fine-tuning is permitted, and upstream data type is unrestricted.
Go to primary source ↓e-structuredPDF p. 3, Section 2.2, Structured; p. 14, Appendix A, CLEVR, dSprites, SmallNORB, DMLab and KITTI descriptions
Structured tasks predict counts, binned locations/orientations, depth and related properties from images. Section 2.2 describes DMLab as navigation-time prediction; Appendix A instead specifies six classes from close/far/very far crossed with positive/negative reward. Neither description is an evaluated action policy.
Go to primary source ↓e-setupPDF p. 4, Section 3.1, Downstream Data Size, Tuning and Evaluation Protocol, Metrics
The main text specifies 1,000 labels, split 800/200 for training/validation, no extra tuning images, retraining on their union, three test seeds and median scores. Across-task aggregation is mean top-1 accuracy. VTAB-full uses full datasets.
Go to primary source ↓e-modelsPDF p. 4, Section 3.1, Representation Learning Algorithms and Hyperparameters
Eighteen methods are compared; pretrained models use ImageNet's 1.28 million images with varying label access. Rotation/Exemplar can accompany supervised losses. GAN discriminators, BigBiGAN's encoder and autoencoder encoders provide representations.
Go to primary source ↓e-architecturePDF p. 16, Appendix C, Network Architectures
ResNet50-v2 yields 2,048-dimensional features; Cond/Uncond-BigGAN use a custom architecture with 1,536 dimensions. Patch adaptation centrally crops 224×224 inputs to 192×192, partitions into nine 64×64 patches, processes them separately and averages representations; the source states reduced stride.
Go to primary source ↓e-light-sweepPDF p. 23, Appendix I, Lightweight sweep
SGD momentum 0.9, batch 512, learning rates {0.1,0.01}, schedules {2500,10000}, tenfold decays, no fine-tuning weight decay, and from-scratch decay 0.01 times learning rate are specified. Images are 224×224 except generative models at 128×128.
Go to primary source ↓e-validation-conflictPDF p. 4, Section 3.1, Downstream Data Size and Tuning and Evaluation Protocol; p. 23, Appendix I, paragraph following lightweight schedule
The main text enforces an 800/200 split within 1,000 images. Appendix I explicitly says regular task validation sets were used for model selection with 1,000 training examples, describing a larger validation set. The supplied PDF does not reconcile these statements.
Go to primary source ↓e-light-resultsPDF p. 21, Appendix G, Table 6, 1000 block, Mean column and named task columns; caption
Tabulated means are From-Scratch 42.1, BigBiGAN 59.1, Rotation 59.5, Sup-10% 61.6, Semi-Rot-10% 64.8, Sup-100% 65.6, Sup-Ex-100% 67.0, Sup-Rot-100% 67.5. Sup-100%/Sup-Rot-100% score 81.0/92.5 on dSpr-Loc and 41.8/48.2 on Clevr-Count. Entries are medians of three test runs. The caption says best-to-worst sorting although means increase down each block.
Go to primary source ↓e-light-narrativePDF p. 5, Section 3.2, Self-supervised and (Semi-)Supervised paragraphs
The prose quotes Rotation 59.6, labels the 64.8 versus 61.6/65.6 comparison as closing 80% of the gap, and reports 67.5 for added rotation supervision. It identifies BigBiGAN as an exception among generative representations.
Go to primary source ↓e-rotation-diagnosticPDF p. 7, Figure 4, both plots and caption; p. 21, Table 6
Top plot compares Sup-100% with From-Scratch; bottom compares Sup-Rotation-100% with Sup-100%. Red/green/blue denote natural/specialized/structured tasks. The caption refers to left/right despite vertical stacking. Positive direction is indicated by the named right-side arrows; scales differ. Small plotted deltas do not all exactly match differences of Table 6's rounded medians.
Go to primary source ↓e-heavy-resultsPDF p. 6, Table 1, 1000 and Full blocks, Mean and named task columns; p. 22, Table 7
Heavyweight 1k means: From-Scratch 59.2, Sup-100% 71.2, Sup-Rot-100% 71.5, Sup-Ex-100% 72.7. Full means: 86.4, 91.1, 91.3, 91.4 respectively. From-Scratch reaches 79.1 on 1k sNORB-Elev versus 57.8 for Sup-100%; Clevr-Count is 78.8 for Sup-100% and 96.4 for Sup-Ex-100%.
Go to primary source ↓e-heavy-sweepPDF p. 23, Appendix I, Heavyweight sweep
The 100-trial random search varies batch size, schedule, horizontal flips, color distortion, resizing/cropping, evaluation resolution, weight decay and SGD/Adam. The SGD interval is printed [0.001,0.0]. The final decay timing is only described as shortly before the end.
Go to primary source ↓e-sweep-discrepancyPDF pp. 25–28, Figures 8–13, SGD axes and binned SGD learning rates; p. 23, Appendix I
Sweep graphics include SGD learning rates through 1.0 and bins ending at 1.0, conflicting with the printed [0.001,0.0] interval. The graphics do not provide a complete corrected sampling specification.
Go to primary source ↓e-linearPDF p. 6, Section 3.4; p. 30, Appendix K, Figure 16 and Table 9, 1000 block, Mean column
Appendix K trains a logistic-regression layer on frozen representations using lightweight hyperparameters. Table 9 gives Sup-100% 57.1, Sup-Rot-100% 56.0, and Cond-BigGAN 42.5; prose quotes 57.3 and 43.3 for the first and third. Figure 16 shows task-dependent ranking agreement between linear evaluation and fine-tuning, including negative correlations.
Go to primary source ↓e-rankingPDF pp. 18–19, Appendix E and Table 4
Mean accuracy implicitly weights domains, group sizes and task headroom. Alternative aggregations have substantial Kendall agreement; the 1k average-rank comparison is 0.873±0.023, and group reweighting gives 0.969±0.017. Statistics sample the three test repetitions over 100 trials.
Go to primary source ↓e-subsetsPDF p. 20, Appendix F and Table 5; p. 3, Section 2.2
Appendix F describes exhaustive selection of five-task subsets from 20 tasks, inconsistent with the 19-task definition. It warns that diverse baseline rankings may exaggerate reliability for similarly performing methods and that repeated subset use risks meta-overfitting.
Go to primary source ↓e-humanPDF p. 15, Appendix B, protocol and Table 3
Human raters receive 20 examples per class on six selected tasks and outperform random guessing. DMLab is reduced to three distance classes. The source explicitly rejects direct human/machine performance comparison because protocols differ.
Go to primary source ↓e-capacityPDF p. 31, Appendix L, Figures 17–18 and accompanying text
The architecture study varies ResNet depth and width; a twice-wide ResNet152 exceeds the standard ResNet50 on the plotted VTAB comparison. It establishes architecture as another experimental factor.
Go to primary source ↓e-budgetPDF p. 7, Section 3.5, Limiting Evaluation Cost and Figure 6; p. 32, Appendix M, Figures 19–20
The schedule study spans 40–40,000 steps at batch 512; the authors describe most curves as near optimal by 1,000 steps. They separately report three hours on a single Nvidia P100 for VTAB-1k at batch 64 and learning rate 0.01, and use Google Cloud TPU-v3-16 accelerators.
Go to primary source ↓e-decathlon-protocolPDF pp. 7–8, Section 4, Vision Benchmarks
Visual Decathlon allows joint optimization on its tasks but forbids external data; VTAB permits external-data transfer and forbids multi-task training on downstream evaluation tasks.
Go to primary source ↓e-decathlonPDF p. 33, Appendix N, Figure 21 and experimental procedure
Sixteen baseline models are fine-tuned on resized Decathlon data. Each task is excluded from benchmark aggregates before comparing against its own ranking. Average Kendall correlations are 0.76 for VTAB and 0.70 for Decathlon. The source cautions that architecture and augmentation differ from published competition entries.
Go to primary source ↓e-discussionPDF p. 8, Section 5, Discussion
The authors see ImageNet supervision as helpful for natural and specialized tasks, limited for structured understanding, and propose diverse data sources including video as future work. Generative-loss conclusions are qualified by the BigBiGAN exception.
Go to primary source ↓e-releasePDF p. 9, Section 5 conclusion before Acknowledgments
The authors announce code and data at the task_adaptation project and released models. This statement does not independently verify present availability or a runnable version-pinned environment.
Go to primary source ↓8.5 Primary sources
A Large-scale Study of Representation Learning with theVisual Task Adaptation Benchmark ↗
PDF · 17,102 extracted words
Source fingerprint
a0624565b2eb3b08e6a453595c833f195b7780417ed15de152732546f4dbad3c