The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
1. Paper overview
In one sentence: Deep feature distances track local human similarity judgments, but improving their fit to synthetic distortions can weaken transfer relative to simple calibration. e-probleme-datasete-distancee-traininge-supervisione-transfer
| At a glance | What to know |
|---|---|
| Research problem | Source description Pixel error can undervalue perceptually severe blur, while similarity depends on structure and context. The paper asks whether features trained for visual prediction already capture human similarity judgments, which training signals matter, and whether fitting preferences improves transfer. BAPPS deliberately measures local patch similarity rather than no-reference image aesthetics. e-problem |
| Core mechanism | Source description BAPPS supplies diverse patch comparisons: Table 1 reports 349.8k distortion and 134.5k real-algorithm 2AFC judgments, plus a separate 28.8k same/not-same JND judgments. These protocols should not be pooled as one accuracy task. e-dataset |
| A key reported result | Calibration transfer to real algorithm outputs: Uncalibrated SqueezeNet/AlexNet/VGG: 64.0%/65.0%/62.6%. The text reports lin gains of 1.1%/0.3%/1.5%. 2AFC agreement and calibration gain. Average over superresolution, frame interpolation, video deblurring and colorization; calibration uses synthetic distortion judgments. Lin improves 11 of 12 architecture-by-task comparisons. Full tuning improves synthetic-distortion performance but trails lin on the real-algorithm panel. The gains are interpreted as percentage-point changes on the plotted agreement scale, not relative percentage improvements. Tuning is less transferable than linear calibration; it is not uniformly worse than every uncalibrated baseline. e-transfer |
| Reading caution | Reader analysis The 64-by-64 design emphasizes local appearance and reduces semantic context. Crowdsourced viewing conditions are uncontrolled, and 2AFC allows subjective choices about similarity. JND uses one-second displays separated by 250 ms; it provides converging evidence under that protocol, not universal perceptual validity. e-protocole-jnd |
Core contributions
- Source description
BAPPS supplies diverse patch comparisons: Table 1 reports 349.8k distortion and 134.5k real-algorithm 2AFC judgments, plus a separate 28.8k same/not-same JND judgments. These protocols should not be pooled as one accuracy task. e-dataset
- Source description
The study compares supervised, self-supervised, unsupervised and random features, then introduces LPIPS calibration variants that separate learning channel weights from changing the representation. e-backbonese-traininge-supervision
- Author claim
The authors interpret transfer across training objectives as evidence that perceptual similarity emerges from useful visual representations. This is their explanation of the BAPPS findings, rather than an established universal theory of perception. e-supervisione-conclusion
Figure 3. LPIPS compares feature activations; human preferences calibrate the distance. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow the left half from patches x and x_0 through the two copies of F. The aligned feature levels indicate corresponding layers, not successive candidate predictions. At each spatial position, normalize across channels, subtract the two feature vectors and weight their differences with w. Equation (1) specifies squared L2 aggregation, spatial averaging and a sum over layers. The right half starts with two already-computed distances, d_0 and d_1, and feeds them to G to predict judgment h through the displayed loss. For lin, F stays fixed while channel weights are learned; tune allows the representation itself to change. e-distancee-training
What it supports. The diagram separates the reusable distance calculation from the supervision used to calibrate it. LPIPS can exploit representations learned for another visual task, with lin adjusting their channel contributions. This is a reference-based patch comparison: the illustrated output is a distance and a training judgment prediction.
Where the evidence stops. The schematic labels L2 norm and a final Avg; its caption averages layers, while Equation (1) uses squared L2 and a layer sum. This edition follows the equation without altering the crop. The missing appendix leaves the detailed G/loss implementation unverified.
2. Motivation
2.1 The problem and the proposed response
Pixel error can undervalue perceptually severe blur, while similarity depends on structure and context. The paper asks whether features trained for visual prediction already capture human similarity judgments, which training signals matter, and whether fitting preferences improves transfer. BAPPS deliberately measures local patch similarity rather than no-reference image aesthetics. e-problem
2.2 What this reading follows
A perceptual metric should distinguish a harmless pixel change from a visually damaging one. This paper tests that idea with BAPPS, where people compare distorted image patches with a reference. It then turns internal convolutional features into a distance and asks how much learning is needed: none beyond pretraining, channel calibration, full fine-tuning, or training from scratch. The useful surprise is that several visual training objectives already produce strong perceptual representations. The practical warning is that better synthetic-distortion performance does not automatically transfer to real algorithm outputs. Read the metric diagram, evaluation design and transfer comparison together before interpreting any single LPIPS score. e-probleme-datasete-distancee-traininge-supervisione-transfer
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Evaluation metrics |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
3.1 Evidence-based assessment
Supports the recorded classification
The evaluation-metric category is supported: the output is a reference-based visual distance validated against human judgments. Architecture, prediction-paradigm and quadrant entries correctly remain not applicable to world-action modeling; multiple backbones and a training head do not constitute a joint future/action model. e-distancee-scoringe-correlations
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Build a patch distance, then decide what to learn
Begin with a fixed feature extractor F and two patches. Its intermediate activations preserve multiple levels of visual structure. Channel normalization makes the comparison depend on feature direction at each location; the weighted difference then determines which channels contribute strongly. Equation (1) aggregates those contributions into one distance. The learnable part can be small: lin keeps the visual representation and calibrates channel weights, whereas tune changes the representation and scratch learns from random initialization. During preference training, both candidates receive a distance to the same reference, and G turns the pair into a judgment prediction. At evaluation, the distance orders candidate similarity. Keep the source’s reduction ambiguity visible: the literal equation specifies squared differences and a layer sum, while the schematic uses different shorthand. e-backbonese-distancee-traininge-scoring
Table 1. BAPPS trades dense ratings of a few images for diverse local patch judgments. Original paper, p. 3 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the Input Type and Judgment Type columns. The earlier datasets use images and mean opinion scores; the BAPPS rows use 64-by-64 patches and either two-alternative preference or same/not-same judgments. Next distinguish input counts from distorted output counts and human-judgment counts: these are different units. In the distortion 2AFC row, each reference is compared through two candidates, whereas JND asks about one reference/distorted pair. The real-algorithm row supplies a separate evaluation domain. The continuously sampled distortion levels and broad operation set explain why coverage, rather than many ratings of each image, is central to the dataset design. e-datasete-distortionse-protocole-jnde-data-ambiguities
What it supports. The table reports 349.8k distortion 2AFC judgments, 134.5k real-algorithm 2AFC judgments and an additional 28.8k JND judgments. This gives the metric both a preference-training resource and different evaluation settings. The counts establish scale, not that the sampled patches exhaust human notions of similarity.
Where the evidence stops. Preserve the printed 425 distortion count: the component counts in Section 2.1 sum to 424, and the supplied text does not reconcile them. Table 3 also has inconsistent split labels and missing k suffixes; this edition does not silently repair those entries.
5.2 Define success through human disagreement
BAPPS does not replace human responses with a unanimous correct answer. For a reference and two candidates, a metric earns the fraction of judgments supporting its choice. If four of five people prefer one candidate, selecting it earns 80% credit. This makes the reported score interpretable as agreement with people, with a human-consistency reference computed from the same preference distribution. The dataset also separates training images from distortion-validation images and reserves real algorithm outputs as a transfer domain. JND asks a different question: can people distinguish a reference from a distorted patch after sequential presentation? The strong correlation between methods’ 2AFC and JND scores supplies converging evidence, but the protocols retain different inputs, response formats and performance measures. e-protocole-scoringe-jnde-correlations
5.3 Read improved fitting and improved transfer separately
Compare purple lin bars with brown tune bars for a single backbone in both Figure 4 panels. The authors report that allowing more of the network to change improves the synthetic-distortion evaluation, while linear calibration transfers better to the real-algorithm aggregate. Reader interpretation: preserving a broadly useful pretrained representation may protect information that direct preference fitting would otherwise change. The comparison supports that explanation but does not isolate its cause; optimization, regularization and distortion coverage could also matter. Nor does the negative transfer result mean that every tuned network falls below its uncalibrated counterpart. The meaningful follow-up fixes the data and backbone, examines individual transfer tasks, and measures uncertainty in the small score differences before treating the pattern as a general design rule. e-traininge-transfere-overall
5.4 Training and inference
During training
In lin, freeze pretrained F and learn channel weights w; the VGG example learns 1,472 parameters. In tune, initialize F from classification pretraining and allow all its weights to change. In scratch, initialize Gaussian random weights and train entirely on perceptual judgments. e-training
Figure 3 feeds distance pair d_0, d_1 into a small network G and a cross-entropy loss against human judgment h. Further loss details are deferred to the unavailable appendix. The main article does not specify a complete calibration optimizer, schedule, G architecture or compute configuration. e-training
During inference
Compute the distance for each reference/candidate pair and prefer the smaller value. G belongs to the illustrated preference-training path, whereas Equation (1) defines the distance. No action, rollout or controller is specified. For JND evaluation, rank pairs by increasing distance and measure retrieval of pairs judged the same. e-distancee-scoringe-correlations
5.5 Implementation flow
- Construct the comparison domain
Traditional operations include photometric changes, noise, blur, spatial transformations and JPEG artifacts. CNN distortions vary task, architecture and losses; 96 distortion-producing networks were trained for one ImageNet epoch each. These networks generate dataset corruptions and are distinct from the feature networks used to measure distance. e-distortions
- Collect preference and transfer data
Use 64-by-64 patches, MIT-Adobe 5k for distortion training and RAISE1k for validation. Table 3 gives aggregate splits of 151.4k training and 9.4k validation 2AFC examples, with two and five judgments respectively. Separate transfer sets use superresolution, frame interpolation, video deblurring and colorization outputs. e-protocole-real-tasks
- Extract and compare features
Pass both patches through F. Use five VGG convolutional layers, AlexNet conv1–conv5, or SqueezeNet’s initial convolution and fire modules. Unit-normalize each spatial feature vector across channels, subtract corresponding vectors, apply channel weights, then aggregate squared differences according to Equation (1). e-backbonese-distance
6. Experiments & results
LPIPS measures reference-to-image-patch dissimilarity through normalized deep features, optionally calibrated with human preferences. BAPPS tests agreement with people across synthetic distortions and real algorithm outputs. Pretrained features outperform common low-level metrics; modest calibration transfers more reliably than full fine-tuning. The evidence concerns local visual similarity with a reference patch, rather than general image quality or action competence.
6.1 Read the original evidence
Table 4. Two perceptual protocols agree more strongly across methods than the semantic-task pairs. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read an off-diagonal cell as a correlation between methods’ task scores, not as accuracy on either task. The 2AFC row gives 0.928 with JND, 0.640 with classification and 0.363 with detection. The symmetric cells repeat the same comparisons, and dashes mark self-comparisons. The caption restricts this analysis to AlexNet-like architectures and averages perceptual scores across traditional and CNN-based distortions. JND ranks patch pairs by increasing metric distance and evaluates retrieval of pairs judged the same. The Avg column summarizes a task’s correlations with the other listed tasks; it is not an average perceptual success rate. e-correlationse-jnd
What it supports. The 0.928 association supports using 2AFC as an informative perceptual test: methods that rank well under relative preference also tend to rank well under JND. The weaker semantic-task correlations are consistent with shared representational usefulness, while leaving considerable room for task-specific behavior and errors.
Where the evidence stops. This is correlation across a selected set of methods, not causal evidence or a guarantee for an individual patch. The surrounding prose refers to Table 5, but the actual printed label is Table 4. JND here concerns distortion sets, not the four real-algorithm transfer sets.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Human agreement across six BAPPS test sets Reported average across traditional and CNN distortions plus four real-algorithm validation sets; five judgments per triplet. | Uncalibrated SqueezeNet 68.6%; AlexNet 68.9%; VGG 67.0%. 2AFC agreement (%) ↑ | L2 63.2%; SSIM 63.1%; FSIM 63.8%; human consistency 73.9%. Deep classification features exceed these traditional metrics under the same aggregate protocol. This six-set average is distinct from either individual panel of Figure 4. e-overalle-scoring |
| Calibration transfer to real algorithm outputs Average over superresolution, frame interpolation, video deblurring and colorization; calibration uses synthetic distortion judgments. | Uncalibrated SqueezeNet/AlexNet/VGG: 64.0%/65.0%/62.6%. The text reports lin gains of 1.1%/0.3%/1.5%. 2AFC agreement and calibration gain | Lin improves 11 of 12 architecture-by-task comparisons. Full tuning improves synthetic-distortion performance but trails lin on the real-algorithm panel. The gains are interpreted as percentage-point changes on the plotted agreement scale, not relative percentage improvements. Tuning is less transferable than linear calibration; it is not uniformly worse than every uncalibrated baseline. e-transfer |
| Agreement between perceptual evaluation protocols Across AlexNet-like methods; perceptual scores averaged over traditional and CNN-based distortions. | 2AFC versus JND: 0.928. Task-score correlation (unitless) | 2AFC versus PASCAL classification: 0.640; versus detection: 0.363; classification versus detection: 0.429. Methods performing well on relative preference tend also to perform well on JND. Correlation across methods is not per-patch prediction accuracy or evidence of causation. e-correlations |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 4. Calibration and full tuning behave differently when the evaluation domain changes. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each panel on its own percentage axis: the left averages traditional and CNN distortions, while the right averages four real-algorithm tasks. Match repeated backbone names by color group. Blue is uncalibrated supervised features, purple is linear calibration, pink is training from scratch and brown is full tuning. Compare one backbone across those groups before comparing architectures. Lin freezes the representation; tune changes it. The black human bar reflects the paper’s judgment-consistency calculation, not a guaranteed maximum for every predictor. The report’s six-set average comes from the prose on page 7 and must not be mistaken for either panel’s average. e-traininge-scoringe-overalle-transfer
What it supports. Full tuning strengthens performance on synthetic distortions, yet its real-algorithm bars trail linear calibration. The accompanying page-8 text reports that lin improves 11 of 12 architecture/task comparisons. This is the key transfer finding: fitting the calibration domain more strongly does not guarantee the best performance on other algorithm artifacts.
Where the evidence stops. The panels have different axis ranges. Error bars are present, but their statistical definition is not stated in the supplied article. Per-task scores are deferred to the missing appendix. Brown bars should be compared with purple before claiming that tuning is uniformly worse than uncalibrated features.
Figure 6. Blur and correlated noise reveal different sensitivities in SSIM and BiGAN. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use the arrow labels before looking at individual pairs. SSIM calls patches close on the left and far on the right; BiGAN calls them close at the top and far at the bottom. Thus the upper-left and lower-right regions show agreement, while the other regions expose disagreements. In the lower-left region, examine the softened underwater scene and landscape: SSIM treats these pairs as close while BiGAN treats them as far. The upper-right region includes differences that BiGAN tolerates more. This is a categorical arrangement of paired examples, not a scatterplot with numerical distance coordinates or a new evaluation quadrant taxonomy. e-diagnostic
What it supports. The examples make the aggregate comparison interpretable: a deep representation can respond strongly to lost structure from blur while accepting some noise patterns more readily than SSIM. The authors explicitly describe these tendencies for BiGAN. They suggest a concrete distortion-family test, rather than establishing that one metric wins for every image.
Where the evidence stops. The source supplies selected qualitative pairs, without frequencies or per-example human vote counts in this figure. Do not generalize BiGAN’s pattern to every LPIPS backbone or treat the display as a measured blur/noise ablation. Additional examples are referenced in the unavailable appendix.
7. Analysis & limitations
7.1 What the evidence leaves open
The 64-by-64 design emphasizes local appearance and reduces semantic context. Crowdsourced viewing conditions are uncontrolled, and 2AFC allows subjective choices about similarity. JND uses one-second displays separated by 250 ms; it provides converging evidence under that protocol, not universal perceptual validity. e-protocole-jnd
Figure 6 suggests BiGAN penalizes blur more than SSIM and tolerates some correlated noise more. These selected examples diagnose different sensitivities without establishing their frequency or proving that every deep metric behaves identically. e-diagnostic
Source inconsistencies remain unresolved: Table 1 lists 425 distortions, while the stated 20 + 308 + 96 components total 424. Table 3 labels two validation-component rows Train and omits k on several real-algorithm counts. Aggregate totals are reported as printed; missing row details are not silently repaired. e-data-ambiguities
7.2 Questions for discussion
- How much of linear calibration’s transfer gain survives uncertainty estimated over source images?
- Would metric ordering hold for full-image semantic changes rather than local distortions?
- How sensitive are absolute LPIPS values to the layer-sum versus layer-average ambiguity?
8. Reproducibility audit
8.1 Requirements and known gaps
Calibration requires original triplets/judgments, exact backbone checkpoints, layer selection, normalization and learned weights. Further loss and per-task-result details are referred to an appendix. The linked release is a provenance lead, not verified implementation availability. e-identitye-backbonese-traininge-overall
Reader-proposed check: fix backbone initialization, triplets and validation protocol when comparing uncalibrated, lin and tune distances. Report synthetic sets and each real-algorithm task separately, and estimate paired uncertainty before interpreting small transfer gains. e-traininge-transfer
Reader-proposed check: test SSIM and BiGAN on matched blur/noise perturbations with fresh 2AFC and JND judgments. Agreement on held-out examples would support the sensitivity explanation; systematic preference reversals would limit it. e-diagnostice-jnd
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Does linear calibration preserve transfer under a controlled backbone?
Reader-proposed experiment, not performed: start lin and tune from the same AlexNet checkpoint, use the same BAPPS distortion-training triplets, and retain an uncalibrated control. Freeze F only in lin; document the loss, optimization budget and distance reduction used. Select settings without real-algorithm judgments. Evaluate each distortion set and each of the four transfer tasks with the original fractional-credit rule, and bootstrap paired differences by source image. The proposed discriminating observation is tune beating lin on distortions while lin beats tune on transfer. Reversal or uncertainty spanning zero would weaken the claimed practical advantage of calibration. e-backbonese-traininge-protocole-scoringe-transfer
Check 2: Do blur and noise disagreements predict fresh human judgments?
Reader-proposed experiment, not performed: choose held-out reference patches and create controlled severity series of Gaussian blur and correlated noise from the paper’s distortion families. Compare SSIM and the same BiGAN representation on identical patches, using a preregistered ranking rule. Collect fresh reference-based 2AFC choices and JND same/different responses with the paper’s presentation timing; use separate calibration and evaluation patches. Report disagreement rates and human agreement by distortion family, with uncertainty over reference images. Support would require BiGAN’s blur sensitivity to align with judgments beyond the selected Figure 6 examples. Consistent human preference for SSIM’s ordering would falsify that broader interpretation. e-distortionse-protocole-jnde-diagnostice-supervision
8.3 Reading coverage
Visual audit: All ten pages of the supplied CVF Open Access PDF were rendered and visually inspected, including the title/author/affiliation and edition notice on page 1, every method/training/evaluation page, Figures 1–6, Tables 1–4 and references. All five final crops were separately viewed: Table 1, Figure 3, Figure 4, Table 4 and Figure 6. The narrow Table 4 crop was rendered at 400 DPI; other crops use 200 DPI. Diagram directions and reductions were checked against the captions and Equation (1); the layer-average versus layer-sum and L2 versus squared-L2 shorthand differences are disclosed. Page 5 supplies uncropped protocol details and Table 3 anomalies; pages 7–8 supply numerical result prose and transfer interpretation. The PDF ends with references and contains no appendix. Referenced supplementary loss details, per-task tables and additional examples remain outside the supplied material; no external edition, code or experiment was inspected.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10. Appendix coverage: not present.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Title, author block and CVF Open Access edition notice (PDF p. 1)
- Abstract
- 1. Motivation
- Prior work on datasets and deep networks (PDF p. 3)
- 2. Berkeley-Adobe Perceptual Patch Similarity (BAPPS) Dataset
- 2.1. Distortions
- 2.2. Psychophysical Similarity Measurements
- 3. Deep Feature Spaces
- 4. Experiments
- 4.1. Evaluations
- 5. Conclusions and acknowledgements
- References (PDF pp. 9–10)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout. This extraction limitation was addressed by visually inspecting all ten PDF pages and all five final crops.
- Separate supplemental material availability has not been fully verified. The supplied ten-page article contains no appendix; the referenced appendix with further loss details, per-test-set scores and additional examples was not supplied or read.
- Code, linked model/data releases and cited external works were not inspected; no experiments were reproduced.
- Identity matches the catalog title and all five authors. This is the CVPR 2018 CVF Open Access article, proceedings pages 586–595. Its title-page notice states a watermark-only difference from IEEE Xplore; that other edition was not independently compared. No numbered revision or revision history was supplied.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e-identityPDF p. 1, title/author/affiliation block, CVF edition notice and Figure 1 caption; proceedings pp. 586–595
Title and authors match the catalog: Richard Zhang; Phillip Isola; Alexei A. Efros; Eli Shechtman; Oliver Wang. Affiliations are UC Berkeley, OpenAI and Adobe Research. The notice identifies the CVF Open Access edition and states watermark-only difference from IEEE Xplore. Figure 1 links a model/data release.
Go to primary source ↓e-problemPDF pp. 1–3, Abstract, Section 1 and prior-work discussion
Pixelwise errors can disagree with human judgments; the study asks whether deep features encode perceptual similarity. BAPPS focuses on local reference-based similarity rather than no-reference quality.
Go to primary source ↓e-datasetPDF p. 3, Table 1, BAPPS rows and caption
Table 1 lists 160.8k distortion input patches and 349.8k 2AFC judgments, 26.9k real-algorithm inputs and 134.5k judgments, plus 9.6k JND inputs and 28.8k same/not-same judgments. Patches are 64 by 64; the distortion count is printed as 425.
Go to primary source ↓e-distortionsPDF pp. 3–4, Section 2.1 and Table 2; PDF p. 2, Figure 2
The source specifies photometric, noise, blur, spatial and compression operations, 20 traditional and 308 composed distortions, plus 96 CNN distortion functions. Each CNN is trained for one epoch on 1.3M ImageNet images while task, architecture and learning parameters vary.
Go to primary source ↓e-real-tasksPDF p. 4, Section 2.1, real-algorithm task paragraphs
Transfer sets sample superresolution, frame interpolation, video deblurring and colorization outputs. Superresolution uses randomly located 64-by-64 triplets; some other tasks sample at random scales.
Go to primary source ↓e-protocolPDF pp. 4–5, Section 2.2, 2AFC paragraphs and Table 3 aggregate rows/caption
Judgments compare x_0 and x_1 to reference x. Distortion patches come from MIT-Adobe 5k for training and RAISE1k for validation. Aggregate counts are 151.4k train and 9.4k validation, with two versus five judgments; real-algorithm validation totals 26.9k. Mechanical Turk collection uses sentinels but cannot control all environmental factors.
Go to primary source ↓e-data-ambiguitiesPDF p. 3, Table 1, Num Distort. column; PDF p. 4, Section 2.1 counts; PDF p. 5, Table 3, Train/Val and # Examples columns
Table 1 prints 425, whereas the text lists 20 traditional, 308 composed and 96 CNN distortions. Table 3 prints Train for the two RAISE1k component rows above 2AFC–Distort [Val]. Its real-algorithm rows print 1.9, 9.4 and 4.7 without k, while the aggregate is 26.9k.
Go to primary source ↓e-jndPDF p. 5, Section 2.2, Just noticeable differences paragraphs
JND presents reference and distorted patches for one second each with a 250 ms gap, asks same/different, and collects three observations on each of the 4.8k patches in each distortion validation set. The source motivates this as a check on subjective 2AFC judgments.
Go to primary source ↓e-backbonesPDF pp. 5–6, Section 3, Network architectures
The paper uses five VGG convolutional layers, AlexNet conv1–conv5, and SqueezeNet’s first convolution and subsequent fire modules; it also evaluates self-supervised AlexNet variants.
Go to primary source ↓e-distancePDF p. 6, Figure 3 with caption and Section 3, Network activations to distance, Equation (1)
Equation (1) spatially averages squared differences of channel-normalized features weighted by w_l, then sums layers. Figure 3 routes x and x_0 through F and shows normalization/subtraction, weighting, distance and spatial averaging. Its final Avg and caption’s layer averaging differ from the equation’s layer sum; the diagram also labels L2 norm rather than explicitly squared L2.
Go to primary source ↓e-trainingPDF p. 6, Section 3, Training on our data and Figure 3 right
Lin fixes F and learns w, with 1,472 VGG parameters; tune starts from classification pretraining and changes F; scratch starts from Gaussian random weights. G maps distance pairs toward a cross-entropy judgment loss. Further loss details are referred to an appendix.
Go to primary source ↓e-scoringPDF p. 6, Section 4, evaluation-scoring paragraph
Validation triplets have five judgments. Choosing a candidate favored by four of five receives 80% credit. If preference fraction is p, expected human consistency is p squared plus (1 minus p) squared.
Go to primary source ↓e-overallPDF p. 7, Section 4.1, first evaluation paragraph; PDF p. 6, Figure 4 and caption
The six-set average is 73.9% human consistency, 68.6% SqueezeNet, 68.9% AlexNet, 67.0% VGG, 63.2% L2, 63.1% SSIM and 63.8% FSIM. Figure 4 separates two distortion sets from four real-algorithm sets. Per-set scores are referred to an appendix.
Go to primary source ↓e-supervisionPDF p. 6, Figure 4; PDF p. 7, Section 4.1, Does the network have to be trained on classification?
Several self-supervised representations perform comparably to supervised features; stacked k-means exceeds low-level metrics, and Gaussian random networks offer much less improvement. Different training objectives do not isolate every possible training factor.
Go to primary source ↓e-transferPDF p. 6, Figure 4, lin/scratch/tune bars in both panels; PDF pp. 7–8, training/generalization evaluation paragraphs
Tuning improves distortion-set performance. Real-algorithm uncalibrated starting scores are 64.0%, 65.0% and 62.6% for SqueezeNet, AlexNet and VGG; lin gains are printed as 1.1%, 0.3% and 1.5%, with 11 of 12 architecture/task scores improving. Tuned bars trail linearly calibrated bars in the real-algorithm panel.
Go to primary source ↓e-correlationsPDF p. 7, Table 4, Figure 5/caption and Section 4.1, perceptual/semantic and JND evaluation paragraphs
For AlexNet-like architectures, Table 4 gives 2AFC–JND .928, 2AFC–classification .640, 2AFC–detection .363 and classification–detection .429. Perceptual scores average distortion sets. JND ranks pairs by increasing distance and evaluates precision-recall area, called mAP. The prose misreferences Table 5 where the displayed table is labeled Table 4.
Go to primary source ↓e-diagnosticPDF p. 8, Figure 6/caption and Section 4.1, Where do deep metrics and low-level metrics disagree?
Figure 6 uses horizontal SSIM close-to-far and vertical BiGAN close-to-far directions. SSIM-close/BiGAN-far examples often contain blur; the text says BiGAN is more tolerant of correlated noise patterns than SSIM. Additional examples are referred to an appendix.
Go to primary source ↓e-conclusionPDF p. 8, Section 5, Conclusions
The authors propose that representations useful for visual tasks also capture perceptual judgments, relate this to representation-learning and neuroscience literature, and leave the extent of that explanation for future research.
Go to primary source ↓8.5 Primary sources
The Unreasonable Effectiveness of Deep Features as a Perceptual Metric ↗
PDF · 7,497 extracted words
Source fingerprint
6809251d37d9d9847335e27da3bd71fdab15197219841ebe9706e6e3d18cc35b