METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments
1. Paper overview
In one sentence: Explicit lexical alignment and recall-weighted coverage improve agreement with human translation judgments, while sentence-level agreement remains modest and sensitive to how those judgments are processed. e-identitye-alignmente-scoree-systeme-segmente-normalized
| At a glance | What to know |
|---|---|
| Research problem | Author claim Automatic evaluation should track human translation quality while remaining cheap enough for repeated system development. The authors target BLEU's indirect treatment of recall, implicit word matching, and fragile sentence-level geometric averaging. Their central objective here is sensitivity at the individual-translation level; the source acknowledges that one scalar cannot capture every aspect of translation quality. e-purpose |
| Core mechanism | Source description An extensible alignment procedure matches exact forms, stems, and synonyms while enforcing at most one partner per word occurrence. Scoring explicitly combines coverage with continuity of matched sequences. e-alignmente-score |
| A key reported result | Sentence-level translation quality correlation: Arabic-to-English: METEOR: 0.347 Mean within-system Pearson correlation with raw combined human judgments. TIDES 2003: 664 sentences, four references, six systems. Each translation has two judges' adequacy and fluency ratings on a 1–5 scale; average those ratings, correlate within each system, then average system correlations. Precision 0.287; recall 0.334; Fmean 0.340. Recall is the stronger standalone component. The full score exceeds Fmean by 0.007 correlation units, a small observed increment from fragmentation penalization. e-datae-protocole-segmente-arabic-components |
| Reading caution | Reader analysis Both evaluation datasets contain English outputs and English references. The Arabic/Chinese labels describe translation directions, not evidence of English-independent lexical resources. These results do not establish performance on other target languages, domains, or modern generation tasks. e-datae-modules |
Core contributions
- Source description
An extensible alignment procedure matches exact forms, stems, and synonyms while enforcing at most one partner per word occurrence. Scoring explicitly combines coverage with continuity of matched sequences. e-alignmente-score
- Source description
The evaluation separates system-level comparison with BLEU/NIST from within-system sentence correlations, then varies scoring components, lexical modules, and human-score normalization. e-protocole-systeme-segmente-modulese-normalized
2. Motivation
2.1 The problem and the proposed response
Automatic evaluation should track human translation quality while remaining cheap enough for repeated system development. The authors target BLEU's indirect treatment of recall, implicit word matching, and fragile sentence-level geometric averaging. Their central objective here is sensitivity at the individual-translation level; the source acknowledges that one scalar cannot capture every aspect of translation quality. e-purpose
2.2 What this reading follows
A translation can preserve meaning while changing word forms, and it can reuse many reference words while arranging them poorly. METEOR addresses these cases by constructing word-level matches before assigning a score. This reading follows the original 2005 formulation: exact forms, stems, and WordNet synonyms provide matches; recall receives extra weight; fragmented matches incur a penalty. The tables then ask two separate questions: can the metric distinguish whole translation systems, and can it track quality sentence by sentence? Keeping those protocols separate is essential. The final diagnostic also shows that changing the human assessment target changes the measured correlation. e-identitye-alignmente-scoree-systeme-segmente-normalized
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Evaluation metrics |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
3.1 Evidence-based assessment
Supports the recorded classification
The recorded evaluation-metric category is supported: METEOR scores language generation against references. World/action-model architecture, prediction paradigm, and quadrant are not applicable because the proposed object is an evaluation rule without future-state prediction or an action policy. This is an affirmative resource-type assessment, not an unverified architecture judgment. e-purposee-alignmente-score
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Build matches before assigning credit
METEOR begins with two strings: an existing machine translation and one human reference. Each word occurrence can have at most one partner. This matters when the same word appears repeatedly: listing possible matches is only the first phase, and the second must choose a valid alignment. At each stage, the source specifies maximum matching cardinality first and the fewest pairwise crossings among tied alternatives. Default stages then progress from exact forms through Porter stems to WordNet synonyms, using only words not matched earlier. A lexical relation therefore creates a candidate link; it does not guarantee that every possible link survives. The experimental description additionally excludes stop words from exact matching. Synonymy is deliberately simple: a shared synset under any sense suffices, and the method does not resolve the sentence's intended sense. e-alignmente-modules
5.2 Separate lexical coverage from contiguous ordering
After alignment, METEOR computes precision using translation length and recall using reference length. The displayed formula, Fmean = 10PR/(R + 9P), gives recall greater weight. Use that explicit fraction: the nearby phrase describing a harmonic mean of P and 9R is not a precise substitute for it. The next operation counts chunks contiguous in both strings. In the paper's example, the reference inserts 'then' between 'the president' and 'spoke to the audience', producing two chunks. The penalty depends on the cube of chunks divided by matched words, and the final score multiplies Fmean by one minus that penalty. Tables 4 and 5 consequently separate coverage from the added fragmentation term. Scoring repeats independently for each reference; the highest-scoring match is retained. e-scoree-arabic-componentse-chinese-components
5.3 Ask which agreement the experiment actually measures
For sentence evaluation, the authors average adequacy and fluency ratings from two judges into a combined human score. They correlate these scores with automatic scores separately within each system, then average the correlations for each language dataset. This is different from Table 1, which correlates one aggregate automatic score with one mean human score across systems. Reader analysis: a high aggregate correlation can coexist with much weaker agreement on individual translations, so the system-level number should not stand in for sentence-level reliability. Table 8 adds a second distinction: identical metric outputs can correlate differently when the human target is normalized. The report therefore retains each protocol and judgment version with its result, and treats small ablation gains as descriptive because uncertainty estimates are absent. e-datae-protocole-systeme-segmente-normalizede-reporting
5.4 Training and inference
During training
The scoring formulas were manually crafted using empirical tests on separate development data. The penalty parameters were not trained to optimality; optimizing the formulas is proposed future work. The paper does not identify that development set or provide a tuning search. It proposes no learned prediction model or neural training stages. e-calibration
During inference
Score each reference independently and retain the highest-scoring match. For system-level scoring, accumulate statistics from the selected matches and recompute aggregate precision, recall, and penalty with the same formula, rather than simply averaging sentence scores. The operation evaluates an existing translation; it generates neither translations nor executable actions. e-purposee-score
5.5 Implementation flow
- Generate candidate word mappings
For each translation–reference pair, run exact matching, Porter stemming, then WordNet synonymy by default. Each stage proposes mappings only for words left unmatched earlier. Synonymy accepts a shared WordNet synset under at least one sense, without sense disambiguation. Section 3.6 says the experimental exact matcher excludes a stop-word list consisting mainly of function words. e-alignmente-modules
- Select a constrained alignment
Within each stage, select the largest subset of proposed mappings satisfying the one-to-one constraint. Among equal-cardinality alignments, minimize pairwise mapping crossings. A crossing reverses the relative order of the two matched word pairs. This criterion selects the alignment; the later fragmentation penalty scores its continuity. e-alignment
- Measure coverage and fragmentation
Precision divides matched words by translation length; recall divides them by reference length. Combine these into Fmean, then group matches into the fewest chunks contiguous in both strings. More fragmented matching increases the cubic penalty. All accepted lexical match types receive equal credit after alignment. e-scoree-future
6. Experiments & results
METEOR evaluates English machine translations through explicit word alignment to human references, recall-weighted matching, and a fragmentation penalty. This 2005 paper tests agreement with human translation judgments and diagnoses the contributions of scoring and lexical matching. Its strong system-level correlation and much lower sentence-level correlations answer different evaluation questions.
The supplied eight-page metric paper contains eight quantitative tables but no architecture or method figure. Its alignment and scoring mechanism is given in prose and unnumbered equations on PDF pp. 3–5, all visually inspected. The six original crops therefore cover system results, scoring-component ablations, lexical-module ablations, and a human-judgment diagnostic; the walkthrough explains the method without fabricating a diagram. This is the June 2005 formulation, with no later edition substituted. e-identitye-visual-scopee-alignmente-score
6.1 Read the original evidence
Table 1. System-level agreement is high, but it answers an aggregate evaluation question. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the rows as scoring methods despite the original left header, which says System ID. The right column is Pearson correlation across the seven Chinese-to-English systems: each contributes one aggregate automatic score and one mean human score. First compare BLEU and NIST with METEOR. Then follow Precision, Recall, F1, Fmean, and METEOR to see how the scoring components relate to agreement with humans. F1 gives the familiar equal-weight combination, while Fmean uses the printed recall-weighted formula. The last step includes fragmentation. These are correlations of system aggregates, not averages of the sentence correlations in later tables. e-datae-protocole-systeme-score
What it supports. METEOR reaches 0.964 compared with BLEU's 0.817 and NIST's 0.892 in this Chinese system-level comparison. Recall alone reaches 0.941, with further increases for Fmean and the full score. The table supports the importance of coverage in this evaluation setting, while the later tables test finer-grained agreement.
Where the evidence stops. Only seven system aggregates underlie this comparison. A correlation of 0.964 is neither a translation success percentage nor the sentence-level correlation. The table supplies no uncertainty interval.
Table 8. The human assessment target changes measured correlation while METEOR stays fixed. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read down each language column while keeping the automatic metric configuration fixed. The top row correlates the exact–Porter–WordNet-synonym metric with the raw combined human assessments used earlier. The bottom row substitutes normalized assessments supplied by MITRE. The experimental question is therefore about the judgment target, not an added matching module or better translation output. The surrounding text motivates normalization by disagreement among sentence-level judges. It does not provide the transformation, so this table tells us the resulting correlations but not how to regenerate the normalized labels from the original adequacy and fluency scores. e-protocole-modulese-normalized
What it supports. Correlation increases from 0.347 to 0.403 for Arabic and from 0.331 to 0.365 for Chinese when normalized human assessments are used. The authors interpret this as evidence that noisy human judgments depress correlation. The directly observed finding is that the metric's measured agreement depends on the assessment representation.
Where the evidence stops. The normalization procedure is absent, so its assumptions and reproducibility remain unresolved. Higher agreement with transformed judgments alone cannot establish that the transformed target is a more faithful measure of translation quality.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| System-level translation quality correlation: Chinese-to-English TIDES 2003: 920 source sentences, four references, seven systems. Correlate one aggregate metric score per system with that system's mean combined human adequacy/fluency score. | METEOR: 0.964 Pearson correlation across systems | BLEU 0.817; NIST 0.892; precision 0.752; recall 0.941; F1 0.948; Fmean 0.952. The strongest displayed correlation concerns seven system aggregates, not sentence-level accuracy or a percentage of correct translations. e-datae-protocole-system |
| Sentence-level translation quality correlation: Arabic-to-English TIDES 2003: 664 sentences, four references, six systems. Each translation has two judges' adequacy and fluency ratings on a 1–5 scale; average those ratings, correlate within each system, then average system correlations. | METEOR: 0.347 Mean within-system Pearson correlation with raw combined human judgments | Precision 0.287; recall 0.334; Fmean 0.340. Recall is the stronger standalone component. The full score exceeds Fmean by 0.007 correlation units, a small observed increment from fragmentation penalization. e-datae-protocole-segmente-arabic-components |
| Sentence-level translation quality correlation: Chinese-to-English The same sentence-level protocol on 920 sentences, four references, and seven systems; average per-system correlations rather than pooling translations. | METEOR: 0.331 Mean within-system Pearson correlation with raw combined human judgments | Precision 0.286; recall 0.320; Fmean 0.327. The ordering of components repeats on this dataset. The 0.004 increment over Fmean does not by itself establish statistical significance. e-datae-protocole-segmente-chinese-components |
| Lexical mapping-stage ablation Same TIDES datasets and raw sentence-level correlation protocol; vary the sequence of mapping modules while retaining METEOR scoring. | Exact → Exact+Porter → Exact+Porter+WN-Synonym: Arabic 0.312 → 0.329 → 0.347; Chinese 0.293 → 0.318 → 0.331. Mean within-system Pearson correlation | Exact+WN-Stem: Arabic 0.330 and Chinese 0.312. WN-Stem replaces Porter; it is not an added third stage. Stemming and then synonym matching improve the reported correlations in both datasets. The preferred stemmer differs between datasets, and these comparisons do not test sense disambiguation. e-protocole-modulese-arabic-modulese-chinese-modules |
| Sensitivity to human-score normalization Hold the exact–Porter–WordNet-synonym metric configuration fixed and compare raw judgments with MITRE-provided normalized judgments for both TIDES datasets. | Normalized: Arabic 0.403; Chinese 0.365. Mean within-system Pearson correlation | Raw: Arabic 0.347; Chinese 0.331. Changing the human target changes measured agreement without an improved translation system or metric configuration. The normalization transformation is not specified. e-protocole-normalized |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 4. Arabic component comparisons locate the larger gain in recall and a smaller gain in fragmentation scoring. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read downward from a precision-only score to recall, then to Fmean and the complete METEOR score. Each entry first correlates sentence scores with the combined human adequacy/fluency judgments within one translation system, then averages those correlations over the six Arabic-to-English systems. The Fmean row already includes the recall emphasis, so the difference between that row and METEOR concerns the additional fragmentation penalty. Keep the decimal values in correlation units: the bold 0.347 is the average Pearson correlation, not a METEOR score for an individual sentence or a proportion of acceptable translations. e-datae-protocole-scoree-segmente-arabic-componentse-reporting
What it supports. Recall correlates better than precision here, 0.334 versus 0.287. Combining coverage terms produces 0.340, and the full metric reaches 0.347. The last increment is 0.007 correlation units, so the displayed evidence favors the penalty but indicates a comparatively small observed contribution in this dataset.
Where the evidence stops. These averages hide variation among systems, shown separately in Table 2. No uncertainty estimates accompany the component differences, and the introduction's significance assertion is not backed by a reported test procedure.
Table 5. The same component ordering holds for Chinese-to-English outputs. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use the same row interpretation as in the Arabic component table, but keep this dataset separate. These values come from 920 source sentences translated into English by seven systems, with four English references per source sentence. The authors correlate automatic and human sentence scores within each system and average the resulting correlations. Follow Recall to Fmean to see the effect of combining coverage terms, then Fmean to METEOR to see the reported increment after penalizing fragmented matches. Do not compare the final 0.331 directly with Table 1's 0.964 as though they were measurements under the same protocol. e-datae-protocole-scoree-chinese-componentse-arabic-componentse-system
What it supports. METEOR's 0.331 exceeds recall's 0.320 and Fmean's 0.327. The gain over Fmean is 0.004 correlation units. Together with the Arabic table, this gives two datasets with the same component ordering, although neither table establishes how stable these small increments would be under resampling or other data.
Where the evidence stops. The Chinese label identifies the source-language dataset; both evaluated strings and references are English. This result therefore does not validate a Chinese tokenizer, Chinese WordNet resource, or language-independent lexical matching.
Table 6. Broader lexical matching raises Arabic correlation, with synonymy adding to Porter stemming. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. The left column lists ordered mapping stages rather than alternative scoring equations. Begin with Exact, which the experimental description says excludes an unspecified stop-word list. Compare the next two rows as competing stemming choices: Porter reduces word forms algorithmically, whereas WN-Stem uses WordNet base forms. WN-Stem replaces Porter in that row; it does not follow it. The final row returns to Porter and adds WordNet synonym matching. Each stage works only on words unmatched by earlier stages. The right column retains the same average of within-system sentence correlations, allowing the effect of changing lexical matching to be examined. e-alignmente-protocole-modulese-arabic-modules
What it supports. Both stemming alternatives improve on exact matching: 0.329 and 0.330 versus 0.312. Adding synonym matching after Porter raises the reported correlation to 0.347. This supports broader lexical matching under this protocol, but the near tie between the two stemmers gives little basis for declaring one consistently preferable.
Where the evidence stops. WordNet synonym matching accepts any shared sense without disambiguating context. This ablation does not measure incorrect-sense matches separately, and the unspecified stop-word list limits exact reconstruction of the first stage.
Table 7. Chinese-to-English outputs favor Porter over WordNet stemming in this comparison. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the rows in parallel with Table 6, retaining the Chinese dataset's own values. Exact matching supplies the starting alignment. The two middle rows separately extend it with Porter stemming or WordNet base-form matching. The final row adds synonym matching after exact matching and Porter. Because earlier stages reserve matched words, this sequence imposes priorities; it is not an unordered union of lexical relations. Section 4 also clarifies that accepted matches subsequently receive equal scoring credit regardless of which module produced them. Thus these rows test changed alignment opportunities and priorities rather than trained weights for different match types. e-alignmente-protocole-modulese-chinese-modulese-arabic-modulese-future
What it supports. Exact plus Porter reaches 0.318, above exact plus WN-Stem at 0.312; the ordering reverses the small Arabic preference for WN-Stem. The full sequence reaches 0.331. Both datasets support the tested synonym extension, but their differing stemmer rankings argue for checking dataset dependence before generalizing the lexical choice.
Where the evidence stops. WN-Stem and WN-Synonym have distinct roles. The table does not compare synonymy against a sense-disambiguated alternative or evaluate module-specific match weights; those weights are a future-work proposal in this edition.
7. Analysis & limitations
7.1 What the evidence leaves open
Both evaluation datasets contain English outputs and English references. The Arabic/Chinese labels describe translation directions, not evidence of English-independent lexical resources. These results do not establish performance on other target languages, domains, or modern generation tasks. e-datae-modules
The introduction claims statistically significant improvements, but the reported experiments provide correlations without a named significance test, confidence intervals, or p-values. Small component increments should therefore remain descriptive. The paper explicitly avoids a sentence-level BLEU/NIST comparison because those metrics were designed for system aggregates. e-reportinge-protocol
Undisambiguated synonym matching can accept a shared sense irrelevant to the sentence; that failure risk follows from the rule, rather than a reported error analysis. The authors also acknowledge weak use of multiple references and propose, but do not evaluate here, semantic relatedness, synthetic references, and module-specific match weights. e-modulese-future
7.2 Questions for discussion
- Would the lexical-module gains persist with fixed preprocessing and paired uncertainty estimates across source sentences?
- How much does best-reference selection hide valid translations whose wording differs from every supplied reference?
- Does a higher correlation after human-score normalization reflect a better target, and how would that be independently checked?
8. Reproducibility audit
8.1 Requirements and known gaps
Reproduction requires the TIDES 2003 system outputs, all four references, and the two judges' adequacy/fluency ratings. Preserve the distinction between corpus-statistic scoring and averaging within-system correlations. The normalized-score result additionally requires the MITRE judgments or their documented transformation, which this paper does not supply. e-datae-protocole-normalized
Implementation gaps include the stop-word list, tokenization/case policy, WordNet release, remaining alignment tie-breaking, and zero-match handling. Porter and WordNet modules are named, but software versions, runtime, and hardware requirements are not reported. Any reconstructed choices should be documented as such. e-alignmente-scoree-modulese-calibration
A proposed mechanism check should hold data and preprocessing fixed while crossing synonym-stage inclusion with fragmentation-penalty inclusion, then compare per-system and mean Pearson correlations. This separates added lexical coverage from the effect of scoring continuity; the illustrated edition specifies this unrun check. e-scoree-protocole-arabic-componentse-chinese-componentse-arabic-modulese-chinese-modules
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Cross lexical expansion with fragmentation scoring
Reader-proposed, not run: on the same TIDES outputs, four references, and raw combined human judgments, compare exact+Porter against exact+Porter+WN-Synonym under both Fmean alone and the full penalized score. Fix tokenization, stop words, WordNet version, and tie-breaking across the four conditions; cache each configuration's alignment before scoring. Report Pearson correlation within every system and its dataset mean. Use paired resampling of source sentences, retaining their outputs from all systems, to estimate uncertainty in differences. A robust positive penalty increment in both lexical conditions would support an independent continuity contribution; a gain confined to one condition would reveal an interaction hidden by the separate published ablations. e-scoree-datae-protocole-modulese-arabic-componentse-chinese-componentse-arabic-modulese-chinese-modules
Check 2: Reproduce the judgment-target change without changing METEOR
Reader-proposed, not run: freeze the full matching configuration and cache every automatic sentence score. Recompute per-system and mean correlations once with raw combined judgments and once with the original MITRE normalized judgments, using identical sentence and system membership. Record the per-system differences as well as the two dataset means. Table 8 supplies comparison targets of 0.347/0.403 for Arabic and 0.331/0.365 for Chinese, raw/normalized. If the original normalized labels or a documented transformation cannot be obtained, mark this check unresolved instead of inventing a normalization. Failure to recover the direction of change would motivate checking labels, alignment of examples, and aggregation before attributing the difference to metric quality. e-datae-protocole-normalized
8.3 Reading coverage
Visual audit: All eight PDF pages (printed pp. 65–72) were rendered and visually inspected. This includes the title/author/imprint page; the motivation; Section 2.2's alignment rule and Fmean, Penalty, and Score equations; the data and evaluation protocols; all eight tables; and future work, acknowledgements, and references. Every final crop was separately viewed: Tables 1, 4, 5, 6, 7, and 8. Crops preserve complete table cells and headers; no table footnotes are present. Method prose, caption context, and uncropped Tables 2–3 remain available through the declared pages. No separate supplement or code was inspected, and no appendix or architecture figure is present in the supplied paper.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8. Appendix coverage: not present.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract (PDF p. 1; printed p. 65)
- 1 Introduction (PDF pp. 1–2)
- 2.1 Weaknesses in BLEU Addressed in METEOR (PDF pp. 2–3)
- 2.2 The METEOR Metric (PDF pp. 3–5)
- 3.1 Data and 3.2 Comparison with BLEU and NIST MT Evaluation Algorithms (PDF p. 5)
- 3.3 Evaluation Methodology, 3.4 Correlation between METEOR Scores and Human Assessments, and 3.5 Comparison with Other Metrics (PDF pp. 6–7)
- 3.6 Comparison between Different Mapping Modules and 3.7 Correlation using Normalized Human Assessment Scores (PDF pp. 7–8)
- 4 Future Work, Acknowledgements, and References (PDF p. 8)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Identity/edition scope: the title and both authors match the catalog. The inspected title-page imprint identifies the June 2005 ACL workshop publication, printed pp. 65–72. No separate revision identifier or revision history is supplied; no comparison with later METEOR editions is established.
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout. This extraction limitation was addressed by visually inspecting all eight PDF pages, all eight tables, and the displayed method equations.
- Separate supplemental material availability has not been fully verified.
- No appendix is present in the supplied paper. Code and external resources were not inspected; no experiments were reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e-identityPDF p. 1 (printed p. 65), title, author blocks, and proceedings imprint
The exact catalog title appears with Satanjeev Banerjee and Alon Lavie, both at the Language Technologies Institute, Carnegie Mellon University, Pittsburgh, PA 15213. The imprint states Ann Arbor, June 2005, proceedings pp. 65–72; no revision identifier is displayed.
Go to primary source ↓e-purposePDF pp. 1–3 (printed pp. 65–67), Abstract, Section 1, and Section 2.1
METEOR evaluates translations with generalized word matching, recall, and fragmentation. The motivation concerns human correlation and BLEU weaknesses, emphasizing segment-level evaluation in this paper.
Go to primary source ↓e-alignmentPDF pp. 3–4 (printed pp. 67–68), Section 2.2, mapping stages and crossing definition
Each stage proposes word mappings, then selects a maximum-cardinality one-to-one subset with minimum crossings among ties. Later stages use unmatched words; the default order is exact, Porter stem, WordNet synonymy. Crossings have a negative product of the two word-position differences.
Go to primary source ↓e-scorePDF pp. 4–5 (printed pp. 68–69), Section 2.2, displayed Fmean, Penalty, and Score formulas and aggregation paragraphs
The displayed Fmean is 10PR/(R+9P); P and R use matched-word counts divided by translation and reference lengths. Penalty is 0.5 times the cube of chunks/matched unigrams; Score is Fmean times one minus Penalty. The best reference match is retained and corpus statistics are aggregated. The president example has two contiguous chunks; zero-match handling is not stated.
Go to primary source ↓e-calibrationPDF p. 4 bottom–p. 5 top (printed pp. 68–69), Section 2.2; PDF p. 8 (printed p. 72), Section 4, Train the Penalty and Score Formulas on Data
Parameters and formulas were chosen through empirical experimentation on separate development data and were not optimized by training. Formula training is proposed future work; the development data and tuning search are not specified.
Go to primary source ↓e-dataPDF p. 5 (printed p. 69), Section 3.1 Data, both columns
TIDES 2003 has 920 Chinese-to-English and 664 Arabic-to-English sentences, four references each, and judged outputs from seven and six systems respectively. Two judges assign adequacy and fluency ratings from one to five; the paper averages them into a combined score. Evaluation compares English strings.
Go to primary source ↓e-protocolPDF p. 5 (printed p. 69), Section 3.2; PDF p. 6 (printed p. 70), Section 3.3
System-level correlation compares aggregate algorithm scores with mean human scores across Chinese systems. Sentence-level evaluation computes Pearson correlation separately within each system, then averages those correlations per dataset. BLEU/NIST sentence-level comparison is considered unfair to their intended use.
Go to primary source ↓e-systemPDF p. 5 (printed p. 69), Table 1, all rows, Correlation column
Chinese system-level correlations are BLEU 0.817, NIST 0.892, precision 0.752, recall 0.941, F1 0.948, Fmean 0.952, and METEOR 0.964. The left header says System ID although its entries name scoring methods.
Go to primary source ↓e-segmentPDF p. 6 (printed p. 70), Section 3.4 and Tables 2–3, per-system and Average rows
Table 2 gives Arabic correlations 0.331, 0.278, 0.399, 0.363, 0.341, 0.371 and mean 0.347. Table 3 gives Chinese correlations 0.385, 0.299, 0.278, 0.307, 0.306, 0.385, 0.355 and mean 0.331.
Go to primary source ↓e-arabic-componentsPDF p. 6 (printed p. 70), Section 3.5 and Table 4, all rows
Arabic mean sentence correlations are precision 0.287, recall 0.334, Fmean 0.340, METEOR 0.347; the difference between the last two printed values is 0.007.
Go to primary source ↓e-chinese-componentsPDF p. 6 (printed p. 70), Section 3.5 and Table 5, all rows
Chinese mean sentence correlations are precision 0.286, recall 0.320, Fmean 0.327, METEOR 0.331; the difference between the last two printed values is 0.004.
Go to primary source ↓e-modulesPDF p. 7 (printed p. 71), Section 3.6, mapping-module definitions
Experiments compare exact, exact+Porter, exact+WordNet-base-form stemming, and exact+Porter+WordNet-synonym matching. The exact module excludes an unspecified stop-word list. Synonym matching requires some senses to share a synset, without disambiguation. WN-Stem is an alternative stemmer.
Go to primary source ↓e-arabic-modulesPDF p. 7 (printed p. 71), Table 6, all rows
Arabic mapping-sequence correlations are exact 0.312, exact+Porter 0.329, exact+WN-Stem 0.330, and exact+Porter+WN-Synonym 0.347.
Go to primary source ↓e-chinese-modulesPDF p. 7 (printed p. 71), Table 7, all rows
Chinese mapping-sequence correlations are exact 0.293, exact+Porter 0.318, exact+WN-Stem 0.312, and exact+Porter+WN-Synonym 0.331.
Go to primary source ↓e-normalizedPDF p. 7 (printed p. 71), Section 3.7 and Table 8; PDF p. 8 (printed p. 72), continuation and Acknowledgements
For the same metric configuration, raw/normalized human-target correlations are Arabic 0.347/0.403 and Chinese 0.331/0.365. MITRE supplied normalized scores; the transformation is not described. The authors attribute the increase to reduced judgment noise.
Go to primary source ↓e-futurePDF p. 8 (printed p. 72), Section 4, all four future-work topics
Future work includes trained scoring formulas, semantic relatedness, better multiple-reference use including synthetic references, and different weights for different matching modules. Current scoring treats accepted matches equally after module priorities determine alignment.
Go to primary source ↓e-reportingPDF p. 2 (printed p. 66), Section 1 significance assertion; PDF pp. 5–7 (printed pp. 69–71), Sections 3.2–3.7 and Tables 1–8
The introduction asserts statistical significance. The reported evaluation supplies correlation point values but no named significance test, confidence interval, or p-value.
Go to primary source ↓e-visual-scopePDF pp. 3–5 (printed pp. 67–69), Section 2.2; PDF pp. 5–7 (printed pp. 69–71), Tables 1–8
The method is expressed in prose and unnumbered equations, without an architecture figure or numbered method diagram. The original experimental visuals are eight numerical tables, including component/module ablations and a normalization diagnostic.
Go to primary source ↓8.5 Primary sources
METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments ↗
PDF · 5,327 extracted words
Source fingerprint
b6fad5895a6abbbf17b00eb02037b32b3256bd0f786bd39c4b70d563fa807082