CIDEr: Consensus-Based Image Description Evaluation
1. Paper overview
In one sentence: CIDEr measures caption consensus by averaging informative n-gram matches across human references, gaining reliability from denser annotations while inheriting their definition of a typical description. e-goale-triplete-metrice-pilote-tablee-rankinge-ciderd
| At a glance | What to know |
|---|---|
| Research problem | Source description The target is human-likeness: similarity to how people typically describe an image. The authors distinguish this from whichever description humans like best. They replace an arbitrary combination of grammaticality, saliency and truthfulness ratings with relative sentence-similarity judgments, then test whether an automatic score predicts those judgments. e-goale-triplet |
| Core mechanism | |
| A key reported result | Overall agreement with human caption consensus: CIDEr: 84% on each dataset, as rounded in the text. Candidate-pair agreement accuracy (%). 48 references; PASCAL-50S: 4,000 pairs, 1,000 each HC/HI/HM/MM; ABSTRACT-50S: 400 pairs, 200 each HC/HI. Best aggregate metric at 48 references, followed by METEOR. At five references, METEOR exceeds CIDEr on PASCAL-50S; CIDEr exceeds METEOR on ABSTRACT-50S. This establishes consensus-label agreement under the stated reference budget, not unrestricted caption quality. No confidence intervals are supplied. e-pairse-overall |
| Reading caution | Reader analysis Consensus depends on the reference population. Workers were US-based, and caption instructions encouraged descriptions others would also provide. Reader assessment: minority but valid descriptions can be under-rewarded; similarity does not independently verify visual truth. e-goale-datae-metric |
Core contributions
- Source description
PASCAL-50S supplies 50 descriptions for each of 1,000 photographs; ABSTRACT-50S supplies 50 for each of 500 sampled clipart scenes. Both are intended for evaluation. e-data
- Author claim
CIDEr-D removes stemming, penalizes length mismatch and clips repeated content. The authors report Spearman correlation 0.94 with CIDEr and claim greater resistance to gaming. e-ciderd
Figure 2. Human consensus is assembled from many small comparisons against reference sentences. Original paper, p. 3 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the panels left to right. The photograph explains where the descriptions originate; the middle panel supplies alternative references to the same scene. The two candidates in the upper-right box stay fixed while sentence A in the lower box changes from one reference to another. B maps to C1 and C maps to C2; these roles agree with Section 3. The figure illustrates 50 references, while Section 6 uses at most 48 in the main evaluation. Section 3 adds the aggregation absent from the diagram: three workers vote per triplet, then the winning candidate is determined across reference triplets. e-triplete-pairse-goale-metric
What it supports. The human target is a preference between two candidates, grounded in many reference descriptions. This provides the labels for checking an automatic metric: agreement means selecting the same candidate. It does not require workers to assign a single absolute quality score to either caption.
Where the evidence stops. The photograph is explanatory context in the figure; the described judgment task presents three sentences. It therefore tests perceived sentence similarity. Consensus under this protocol should not be equated with an independent check of every claim against image pixels.
2. Motivation
2.1 The problem and the proposed response
The target is human-likeness: similarity to how people typically describe an image. The authors distinguish this from whichever description humans like best. They replace an arbitrary combination of grammaticality, saliency and truthfulness ratings with relative sentence-similarity judgments, then test whether an automatic score predicts those judgments. e-goale-triplet
2.2 What this reading follows
A caption can be plausible yet describe an image differently from most people. CIDEr makes that distinction central to evaluation: first collect relative human judgments of sentence similarity, then ask whether an automatic score predicts the resulting consensus. The score uses TF-IDF weighted n-grams and averages over references, so the reference collection is part of the measurement. Read the figures as a sequence from defining the target to testing reference coverage, inspecting difficult pair types and comparing whole systems. Keep the original CIDEr experiments separate from CIDEr-D, the paper's modified score intended to discourage repetition and other gaming. e-goale-triplete-metrice-pilote-tablee-rankinge-ciderd
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Evaluation metrics |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
3.1 Evidence-based assessment
Supports the recorded classification
The recorded evaluation-metric classification is supported by the sentence-scoring definition and validation protocol. Architecture, prediction paradigm and world/action quadrant are not applicable: CIDEr neither models future states nor predicts or executes actions. This is an affirmative scope judgment, not an unverified model assignment. e-goale-metrice-triplet
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Turn a population of descriptions into an evaluation target
Begin with the distinction illustrated by Figure 1: several people can describe the same image differently, and the most typical description need not be the one an observer prefers overall. Figure 2 operationalizes typicality through repeated relative choices. Fix the candidate pair, change the reference, collect three votes, and aggregate twice: first within each triplet and then across triplets. CIDEr does not implement those votes directly. It computes continuous caption–reference similarities and averages them before comparing candidate scores. Reader interpretation: this averaging is a proxy for the population-based target, not an algebraic guarantee that cosine similarity reproduces majority preference. That gap is why the human-agreement experiments are essential. It also explains why the result should be described as consensus agreement rather than universal caption correctness. e-goale-triplete-metrice-overall
5.2 Follow the two roles played by reference captions
References enter CIDEr twice. Locally, each candidate is compared with every description of its image, so broadly repeated content can contribute through many comparisons. Globally, Equation (1) uses references across the image dataset to estimate which n-grams are common and should receive less weight. The document-frequency denominator counts images with any occurrence, rather than counting every caption separately. Equation (2) then normalizes the weighted vectors before averaging cosine similarities; Equation (3) combines n-gram lengths one through four. Reader interpretation: changing the reference collection can change both the local consensus estimate and the global weighting statistics. Figure 3 demonstrates sensitivity to reference count, but it does not independently separate these two pathways. A careful reproduction should specify the corpus used for IDF as well as the references used to score each image. e-metrice-pilote-overall
5.3 Read CIDEr-D as a changed measurement rule
Section 8 starts from an optimization problem: a caption generator might exploit the score by repeating high-confidence words. CIDEr-D responds with three changes. Removing stemming preserves distinctions among word forms. A Gaussian factor penalizes differences between candidate and reference lengths, using sigma 6. Clipping then limits the candidate contribution to each reference's support. For implementation, follow Equation (4) precisely: the numerator contains the componentwise minimum of the two weighted vectors, while the denominator still contains their original norms. It is not simply cosine similarity after renormalizing a clipped candidate. Finally, a factor of ten rescales the score. The reported 0.94 Spearman correlation describes similarity to CIDEr's ranking; reader assessment: it does not isolate which defense works or establish general resistance to adversarial optimization. e-ciderde-metrice-external
5.4 Training and inference
During training
CIDEr has no learned encoder, training loss or action policy. Its preparation computes reference-corpus statistics; the authors empirically select uniform n-gram weights. The paper evaluates supplied outputs from captioning systems rather than training those systems as part of CIDEr. e-metrice-pairse-ranking
During inference
At scoring time, transform a new caption using the same representation and corpus IDF, compare it with its image's references, and aggregate. Comparing two scores yields the metric's preferred caption. There is no future-state rollout, action extraction, environment feedback or executed control in this procedure. e-metrice-triplet
5.5 Implementation flow
- Build the human target
For each candidate pair B and C, vary reference sentence A. Workers choose which candidate resembles A more, without an explicit definition of similarity. Three judgments determine each triplet's majority; a majority across triplets determines the pair winner. Accuracy is agreement with these winners. e-triplet
- Represent informative overlap
Stem both candidate and references, count ordered n-grams of lengths one through four, and apply TF-IDF. Document frequency counts images having the n-gram in any reference, rather than individual reference sentences. Common phrases across images consequently receive less weight. e-metric
- Aggregate consensus
For each n-gram length, average candidate–reference cosine similarities over every reference. Average the four resulting scores uniformly. Longer n-grams are intended to capture richer semantics and grammatical structure; the metric uses text rather than image pixels. e-metric
- Separate the defended variant
CIDEr-D uses unstemmed words, a Gaussian length penalty with sigma 6, and Equation (4)'s componentwise minimum of candidate and reference TF-IDF vectors in the numerator. The denominator retains the original vector norms. Scores are multiplied by ten and combined across n-gram lengths with uniform weights. e-ciderd
6. Experiments & results
CIDEr evaluates how closely an image caption matches the consensus of human reference descriptions. It combines TF-IDF weighted n-gram similarities with a triplet-based human validation protocol and densely captioned evaluation datasets. Its gains depend on reference coverage and pair type; the paper separately introduces CIDEr-D to discourage gaming.
This metric paper has an annotation-method diagram rather than a learned-model architecture. Figure 3 supplies the reference-count diagnostic; the supplied PDF contains no component-isolating CIDEr-D ablation or controlled quantitative gaming table. Section 8 defers qualitative examples to Reference [42], which was not supplied. The illustrated edition therefore cannot establish the separate efficacy of the proposed defenses. e-triplete-metrice-pilote-ciderde-external
6.1 Read the original evidence
Figure 3(b–c). Compare metrics at the same reference count before attributing an improvement to the scoring rule. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use the left panel for PASCAL-50S and the right for ABSTRACT-50S. Their vertical ranges differ, so read the labeled accuracies rather than comparing apparent heights across panels. Within either panel, compare the black CIDEr curve with cyan METEOR points at the same horizontal position. METEOR has fewer evaluated points because of runtime, according to Section 7.2. The dashed vertical marker highlights five references. The horizontal axes extend to fifty, but Section 6 sets the main evaluation maximum at 48. Unlike the pilot panel, these black curves use the combined CIDEr score, not CIDEr1 alone. e-pairse-overalle-pilote-metrice-human
What it supports. At 48 references, the text reports rounded CIDEr accuracy of 84% on both datasets, followed by METEOR. The ordering depends on budget: at five references, METEOR leads on PASCAL-50S and CIDEr leads on ABSTRACT-50S. Dense references and the metric choice both matter.
Where the evidence stops. PASCAL includes four pair types, while ABSTRACT includes only two human-caption pair types. Their aggregate accuracies therefore summarize different tests. These curves also differ from Section 7.4's disjoint 24/24-reference human comparison, where CIDEr reaches 82% on each dataset.
Table 1. The difficult pair types reveal CIDEr's gains and the human–machine exception. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Decode the columns before following the boldface. HC compares two human captions for the same image; HI replaces one with a human caption for another image. HM compares a correct human caption with a machine caption, and MM compares two machine captions. All cells are agreement percentages, so higher is better. PASCAL contributes 1,000 pairs per type; ABSTRACT contributes 200 each for HC and HI. Compare rows within one column rather than treating these columns as interchangeable tasks. Keep the retained note: ROUGE is ROUGEL for PASCAL but ROUGE1 for ABSTRACT. Bold values include a tie in ABSTRACT HC. e-pairse-tablee-overall
What it supports. CIDEr reaches 72.2% on PASCAL MM versus METEOR's 67.7%, a calculated 4.5 percentage-point gain. PASCAL HC also improves, from 65.2% to 71.8%. However, CIDEr's 92.1% on HM trails METEOR's 96.4%. It wins four columns outright and ties for best in another.
Where the evidence stops. The paper's claim of being best in five of six cases includes the ABSTRACT HC tie. No confidence intervals or significance test accompany these cells. Near-ceiling HI accuracy should not be taken as evidence that subtle semantic differences are solved.
Figure 4. Agreement at the system level is strong even though two middle-ranked systems exchange places. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Each group shows the same five captioning methods, identified by color in the legend. The left group uses human consensus judgments and the right uses CIDEr. The vertical axis is how often a system beats another system, not the fraction of captions that are correct. Section 6 constructs all ten system pairs on each of 100 images, producing 1,000 comparisons, and uses 20 random reference sentences per pair. Follow a color across the two groups to compare its aggregate performance, then compare the green Video+ and purple Video bars to see the ordering disagreement. e-rankinge-pairse-table
What it supports. Both assessments place Midge first, Babytalk second and Story last. Section 7.3 reports Pearson correlation 0.98 between the five system-level win fractions. The visible reversal of Video and Video+ shows why high aggregate correlation can coexist with a meaningful ranking disagreement.
Where the evidence stops. This is a separate 100-image, 20-reference system comparison, not Table 1's full pair-type evaluation. Correlation over five aggregate scores does not measure per-caption agreement, and the figure supplies neither exact bar labels nor uncertainty intervals.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Overall agreement with human caption consensus 48 references; PASCAL-50S: 4,000 pairs, 1,000 each HC/HI/HM/MM; ABSTRACT-50S: 400 pairs, 200 each HC/HI. | CIDEr: 84% on each dataset, as rounded in the text. Candidate-pair agreement accuracy (%) | Best aggregate metric at 48 references, followed by METEOR. At five references, METEOR exceeds CIDEr on PASCAL-50S; CIDEr exceeds METEOR on ABSTRACT-50S. This establishes consensus-label agreement under the stated reference budget, not unrestricted caption quality. No confidence intervals are supplied. e-pairse-overall |
| Consensus agreement by candidate-pair type Table 1; the PASCAL-50S and ABSTRACT-50S pair sets from Section 6, using the maximum 48-reference protocol. | PASCAL CIDEr HC/HI/HM/MM: 71.8/99.7/92.1/72.2. ABSTRACT CIDEr HC/HI: 71.5/96.0. Candidate-pair agreement accuracy (%) | PASCAL METEOR: 65.2/99.3/96.4/67.7; ABSTRACT METEOR: 69.5/94.0. ABSTRACT ROUGE1 ties CIDEr's 71.5 HC score. CIDEr wins four columns outright and shares a fifth; it loses HM. Its PASCAL MM gain over METEOR is 4.5 percentage points, calculated from the table. e-pairse-table |
| Reference-count sensitivity Pilot: 120 collected descriptions for each of 50 sampled images; 179 pooled candidate pairs; scoring with up to 100 references. | CIDEr1 and ROUGE1 improve toward roughly 50 references, with diminishing gains beyond that region. Human-consensus agreement versus reference count | BLEU1 declines at larger reference counts in Figure 3(a). A diagnostic supporting denser references; it does not isolate TF-IDF, higher-order n-grams or each CIDEr-D modification. e-pilot |
| Predicting consensus from disjoint reference halves Same 4,000 PASCAL and 400 ABSTRACT pairs; 24 references define ground truth, and the other 24 support human and metric predictions. | CIDEr: 82% on PASCAL-50S and 82% on ABSTRACT-50S. Candidate-pair agreement accuracy (%) | Humans: 90% and 83%, respectively. This matched split differs from the 48-reference result. Human agreement provides a protocol-dependent comparison, not a mathematical ceiling. e-human |
| Ranking five captioning systems by consensus 100 images with all five systems' outputs; all ten system pairs per image, totaling 1,000 pairs, each compared using 20 random references. | 0.98 Pearson correlation between system-level human and CIDEr win fractions | Both rank Midge first, Babytalk second and Story last; Video and Video+ switch order. Correlation summarizes five aggregate system scores and does not imply agreement on every caption pair. e-ranking |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 3(a). Reference coverage changes the reliability of the measurement itself. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the horizontal axis: equal spacing represents multiplicative changes in reference count because the axis is logarithmic. The vertical axis is agreement accuracy, not the numerical caption score. All three curves here use unigrams; this panel does not show the full four-order CIDEr metric. The black CIDEr1 and red ROUGE1 curves broadly improve as references accumulate, while the blue BLEU1 curve eventually declines. The green arrows annotate reference-count regions rather than algorithmic branches. Section 7.1 describes a pilot with 120 descriptions for each of 50 images and 179 candidate pairs, scored with up to 100 references. e-pilote-metrice-ciderd
What it supports. The diagnostic motivates collecting many descriptions per image: CIDEr1 and ROUGE1 continue improving beyond the then-common five-reference setting, with smaller gains around and beyond fifty. More references are not automatically beneficial for every aggregation rule, as BLEU1 demonstrates in this pilot.
Where the evidence stops. These are fluctuating empirical curves without uncertainty bands. The apparent saturation is specific to this pilot and its pair mixture. This reference-count diagnostic does not isolate the contributions of TF-IDF, higher-order n-grams or the separate CIDEr-D defenses.
7. Analysis & limitations
7.1 What the evidence leaves open
Consensus depends on the reference population. Workers were US-based, and caption instructions encouraged descriptions others would also provide. Reader assessment: minority but valid descriptions can be under-rewarded; similarity does not independently verify visual truth. e-goale-datae-metric
The headline comparison with five-reference BLEU1/ROUGE1 changes both metric and reference budget. Same-budget curves and Table 1 provide better evidence for metric differences. The plots and table contain no uncertainty intervals or component-isolating CIDEr ablations. e-overalle-tablee-pilot
CIDEr-D addresses specific repetition and length exploits. Its reported 0.94 rank correlation with CIDEr does not establish robustness against arbitrary optimization; detailed qualitative examples are deferred to another version. e-ciderde-external
7.2 Questions for discussion
- Would CIDEr's advantage persist under repeated disjoint-reference sampling rather than one reference pool? (e-pilot, e-human)
- Which CIDEr-D modification suppresses repetition most effectively without penalizing valid morphological or length variation? (e-ciderd)
8. Reproducibility audit
8.1 Requirements and known gaps
Preserve reference/candidate separation, pair categories, image-level document frequencies and the 48-reference versus 24/24 protocols. Reconstructing human labels requires three votes per triplet. Report baseline variants explicitly: Table 1 uses ROUGEL for PASCAL and ROUGE1 for ABSTRACT; METEOR is version 1.5. e-triplete-metrice-pairse-baselinese-tablee-human
The supplied text leaves exact tokenization, stemmer choice, zero-vector handling, pair-tie handling and random seeds unspecified. It gives no hardware or runtime benchmark. Code/dataset availability is an author statement; implementation provenance would need verification before reproduction. e-metrice-triplete-pairse-baselinese-availability
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Separate reference coverage from changes in IDF
Reader-proposed check, not run: retain the PASCAL HC/HI/HM/MM pair identities and use the paper's 24-reference ground-truth half. From the other half, repeatedly sample five, ten and 24 scoring references with identical subsets for CIDEr and METEOR. First hold image-level IDF fixed to that prediction half; then repeat with IDF recomputed from each sampled reference collection. Keep candidates excluded and report per-type agreement plus uncertainty from resampling images. If CIDEr's gains appear only when IDF changes, the explanation that denser local reference coverage drives the gain would be weakened. A stable gain with fixed IDF would support that explanation while preserving the disjoint-reference test. e-metrice-pairse-pilote-humane-baselines
Check 2: Isolate CIDEr-D's defenses against repeated phrases
Reader-proposed check, not run: for a fixed set of original captions and references, create repeated-phrase variants, including variants matched to reference lengths, and grammatical singular/plural or tense alternatives. Compare CIDEr, removal of stemming alone, addition of the Gaussian penalty, addition of Equation (4)'s clipping, and the complete CIDEr-D rule. Retain original vector norms in the clipped formula and remove the tenfold scale when comparing raw changes across variants. Collect blinded triplet preferences for original versus altered captions. A defense fails this check if a repetition variant gains score while losing human preference; a harmful tradeoff appears if grammatically valid variants lose agreement more than repetition exploits do. Report effects separately by edit type and sentence length. e-ciderde-metrice-triplete-external
8.3 Reading coverage
Visual audit: All ten PDF pages were rendered and visually inspected, including the title and edition notice on p. 1; Figure 1 on p. 2; the Figure 2 annotation roles on p. 3; Equations (1)–(3) on p. 4; dataset, pair and baseline protocols on pp. 3–6; all Figure 3 panels, Table 1 with its ROUGE note and Figure 4 on p. 7; the human split and CIDEr-D equations on p. 8; and references on pp. 9–10. The Figure 2 role labels were checked against Section 3, and Figure 3 axes and reference counts against Sections 6–7. All five final original crops were viewed; the metric-curves crop was widened and viewed again to retain the rightmost tick label. The table crop retains its caption because the necessary ROUGE-variant note is at its end. No external version, supplement, implementation or server was inspected.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10. Appendix coverage: not present.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- PDF pp. 1–2: title, authors, Abstract, 1. Introduction
- PDF pp. 2–3: 2. Related Work
- PDF pp. 3–4: 3. Consensus Interface
- PDF p. 4: 4. CIDEr Metric, Equations (1)–(3)
- PDF pp. 4–5: 5. New Datasets
- PDF pp. 5–6: 6. Experimental Setup
- PDF pp. 6–8: 7. Results, including Sections 7.1–7.4
- PDF p. 8: 8. Gameability and Evaluation Server, Equations (4)–(5); 9. Conclusion; Acknowledgements
- PDF pp. 9–10: References
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- The extraction limitation was addressed by visually inspecting all ten supplied PDF pages, Figures 1–4, Table 1 and all five final crops. No appendix is present in this PDF.
- Identity: the title page prints 'CIDEr: Consensus-based Image Description Evaluation'; the supplied observedTitle and catalog capitalize 'Based'. This is a capitalization difference, with exact author agreement. The supplied provenance title is preserved. The header identifies the CVPR 2015 CVF Open Access version and points to IEEE Xplore as authoritative; that edition was not supplied or compared. No numbered revision or revision date is printed.
- Reference [42] identifies a 2014 CoRR version, abs/1411.5726. Interface snapshots, additional metric comparisons and qualitative gaming examples deferred to that work were not supplied or read. Differences from that version are unverified.
- Code, datasets outside the PDF and the evaluation server were not inspected. No experiments were run. The title page prints Devi Parikh's affiliation as 'Virgnia Tech'; that spelling is preserved in metadata.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e-identityPDF p. 1 (printed p. 4566), CVF header and title/author block
The header identifies the CVPR 2015 Open Access version. Title and author order match the supplied identity apart from 'Consensus-based' capitalization. Credits are Ramakrishna Vedantam, C. Lawrence Zitnick and Devi Parikh; affiliations print Virginia Tech, Microsoft Research and Virgnia Tech, respectively.
Go to primary source ↓e-goalPDF p. 1, Section 1 and footnote 1; PDF p. 2, Figure 1 and caption
The paper distinguishes similarity to typical human descriptions from preferred overall quality. Figure 1 marks representative consensus descriptions in bold for a photograph and a clipart scene.
Go to primary source ↓e-tripletPDF p. 3, Figure 2 and Section 3; PDF p. 4, Section 3 continuation
A is a reference and B/C are candidates. The figure illustrates cycling through 50 references. Three votes establish each triplet preference; majority across triplets establishes a pair preference. Metric accuracy measures agreement on the preferred candidate.
Go to primary source ↓e-metricPDF p. 4, Section 4, Equations (1)–(3) and adjoining definitions
CIDEr stems words, forms length-one through length-four n-grams, computes image-document TF-IDF weights, averages cosine similarities over references, then uses uniform n-gram weights with N=4.
Go to primary source ↓e-dataPDF p. 3, human-study qualification paragraph and footnote 3; PDF pp. 4–5, Section 5
PASCAL-50S contains 1,000 images and ABSTRACT-50S 500, each with 50 captions written by distinct subjects per image. Caption instructions seek main aspects others would mention. AMT workers are US-based with approval rate above 95% and at least 500 approved HITs.
Go to primary source ↓e-pairsPDF p. 5, Section 6, Candidate Sentences
ABSTRACT uses 48 references and two held-out candidates, with 200 HC and 200 HI pairs. PASCAL uses human and five systems' outputs, with 1,000 each HC, HI, HM and MM pairs; two references are dropped for a maximum of 48. HC compares correct human captions, HI introduces a caption for another image, HM compares human/machine, and MM compares machines.
Go to primary source ↓e-baselinesPDF pp. 5–6, Section 6, Metrics
BLEU and METEOR are aggregated at sentence level. The authors implement baselines except METEOR, for which version 1.5 is used. The text distinguishes best-reference matching in several baselines from CIDEr's averaging.
Go to primary source ↓e-pilotPDF p. 6, Section 7.1; PDF p. 7, Figure 3(a) and caption
The pilot collects 120 descriptions on 50 images and pools 179 pairs. It tests unigram metrics with up to 100 references. CIDEr1/ROUGE1 gains diminish around 50, while BLEU1 declines at larger counts. Figure 3(a) uses a logarithmic horizontal axis.
Go to primary source ↓e-overallPDF pp. 6–7, Section 7.2; PDF p. 7, Figure 3(b–c)
The text reports 84% CIDEr accuracy on each dataset at 48 references, ahead of METEOR. At five references METEOR leads on PASCAL, while CIDEr leads on ABSTRACT. The PASCAL comparison to five-reference BLEU1 and ROUGE1 reports 76% and 74%, respectively.
Go to primary source ↓e-tablePDF p. 7, Table 1, all rows/columns and ROUGE note
CIDEr PASCAL HC/HI/HM/MM is 71.8/99.7/92.1/72.2; METEOR is 65.2/99.3/96.4/67.7. ABSTRACT CIDEr HC/HI is 71.5/96.0, METEOR 69.5/94.0 and ROUGE 71.5/91.0. ROUGE means ROUGEL on PASCAL and ROUGE1 on ABSTRACT; the ABSTRACT HC best score is tied.
Go to primary source ↓e-rankingPDF p. 6, Section 6, Machine Approaches; PDF p. 7, Figure 4; PDF p. 8, Section 7.3 continuation
Five systems are paired exhaustively on 100 images, yielding 1,000 pairs with 20 random references. Human and CIDEr aggregate win fractions have Pearson correlation 0.98. Midge, Babytalk and Story share first, second and last ranks; Video/Video+ reverse order.
Go to primary source ↓e-humanPDF p. 8, Section 7.4
Of 48 references, 24 establish ground-truth consensus and 24 support prediction on all 4,000 PASCAL and 400 ABSTRACT pairs. CIDEr/human accuracies are 82%/90% for PASCAL and 82%/83% for ABSTRACT.
Go to primary source ↓e-ciderdPDF p. 8, Section 8, Gameability, Equations (4)–(5)
CIDEr-D removes stemming, introduces a Gaussian length penalty with sigma=6, uses a componentwise minimum in the weighted-vector numerator while retaining original norms, and multiplies by ten. Uniform weights combine n-gram scores. Reported Spearman correlation with CIDEr is 0.94; increased gaming robustness is an author claim.
Go to primary source ↓e-availabilityPDF p. 2, Contributions paragraph; PDF p. 8, Evaluation Server paragraph
The authors state that code and datasets are on their webpages and that CIDEr-D is available on the MS COCO caption evaluation server. These are statements in the paper, not independently verified availability.
Go to primary source ↓e-externalPDF p. 5, Section 5; PDF p. 6, Section 7.2; PDF p. 8, Section 8; PDF p. 10, Reference [42]
Interface snapshots, more comprehensive metric comparisons and qualitative gaming/ranking examples are deferred to [42], identified in the bibliography as the authors' 2014 CoRR work abs/1411.5726. The supplied PDF ends with references and contains no appendix.
Go to primary source ↓8.5 Primary sources
CIDEr: Consensus-Based Image Description Evaluation ↗
PDF · 7,848 extracted words
Source fingerprint
82ad58a206c5acf37e5147f0c3b0328eaf535c9aeb2ee6f58d8e8cd34fed4440