Bleu: a Method for Automatic Evaluation of Machine Translation
1. Paper overview
In one sentence: BLEU makes translation comparison repeatable by pooling clipped phrase matches and penalizing corpus brevity, at the cost of dependence on reference wording and evaluation protocol. e02e03e04e08e09e12e15
| At a glance | What to know |
|---|---|
| Research problem | Source description Frequent human evaluation slows translation-system development. BLEU assumes that closeness to professional reference translations can serve as a reusable proxy for quality. It needs candidate translations and aligned human references; it measures textual agreement rather than directly checking source-language meaning. e02e08 |
| Core mechanism | |
| A key reported result | BLEU agreement with human system judgments: 0.99 for monolingual judgments; 0.96 for bilingual judgments. Reported linear correlation coefficient across five systems. Five systems; two-reference BLEU. Two groups of 10 nonprofessional judges evaluated 250 source/translation pairs sampled from the corpus, rating translations from 1 to 5. Monolingual judges were native English speakers assessing readability/fluency; bilingual judges were native Chinese speakers resident in the United States. Strong system-level association in this study. It is not sentence-level accuracy or an out-of-sample prediction test; no correlation confidence interval is reported. e12e15 |
| Reading caution |
Core contributions
Figure 2. Phrase length changes the scale of the evidence being combined. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the horizontal axis: each cluster uses one n-gram order, from single words to four-word phrases. Within a cluster, orange and yellow identify H2 and H1; green, blue and pale purple identify S3, S2 and S1. The vertical axis is modified precision, not final BLEU. Follow a color across clusters to see precision fall as longer exact matches become rarer. Sections 2.1.2–2.1.3 use this pattern to motivate combining orders through their logarithms. The two-reference experiment is distinct from Figure 1’s four-reference, 127-sentence human-versus-machine illustration on the preceding page. e05e06e08
What it supports. The chart and surrounding text support a consistent broad ordering, with both human translators above the machine systems and rapidly decreasing precision at higher orders. It motivates the geometric mean used in the baseline. It does not itself display the final brevity-penalized score or an isolated comparison of averaging rules.
Where the evidence stops. The bars carry no numerical labels or uncertainty intervals; near-equal high-order machine bars should not be assigned invented precision values. This is a method-motivation diagnostic, not an architecture diagram or controlled component ablation.
2. Motivation
2.1 The problem and the proposed response
Frequent human evaluation slows translation-system development. BLEU assumes that closeness to professional reference translations can serve as a reusable proxy for quality. It needs candidate translations and aligned human references; it measures textual agreement rather than directly checking source-language meaning. e02e08
2.2 What this reading follows
BLEU begins with a practical question: can developers compare translation systems frequently without commissioning a fresh human evaluation each time? Its answer is a fixed scoring rule over candidate translations and reusable human references. Read this paper by separating three issues: what counts as a match, how sentence evidence becomes a corpus score, and whether that score follows human preferences. The original experiments compare three machine systems and two human translators in Chinese-to-English translation. Their strong correlations are informative, but the reference-count experiment and the disagreement between judge groups show why a BLEU value needs its protocol attached. e02e03e04e08e09e12e15
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Evaluation metrics |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
3.1 Evidence-based assessment
Supports the recorded classification
The recorded evaluation-metric classification is supported by a reference-based text scoring rule. BLEU defines no world model, action predictor or control architecture, so the recorded Not applicable architecture, prediction paradigm and quadrant are appropriate. The suggestion of wider NLG use is prospective. e02e08e17
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 1. Spend each reference match only as often as it occurs
The first design problem is overgeneration. Ordinary unigram precision accepts every candidate occurrence if that word appears anywhere in a reference. Example 2 exploits this by repeating “the” seven times. BLEU instead finds the largest occurrence count in any one reference and clips the candidate count to it, giving 2/7. The maximum is taken separately for each word or n-gram; one need not choose a single reference for all lexical matches. The same construction extends to longer phrases. Matches are local to the aligned sentence, then clipped counts are pooled across the corpus. This preserves a distinction between matching a reusable vocabulary item and matching the amount of evidence actually present in a translation. e03e04
5.2 2. Repair the short-output incentive without recalling every synonym
Clipping alone does not solve omission. In Example 3, the tiny candidate “of the” has perfect unigram and bigram precision despite leaving almost the entire translation absent. Naive recall over all reference words is also problematic: Example 4 rewards a candidate that redundantly lists several synonymous alternatives. BLEU therefore handles length with a separate factor. For each sentence, it takes the reference length closest to the candidate length, then sums these best-match lengths into r. Candidate lengths sum into c. The exponential penalty applies when c is no larger than r and equals one at equality. Combining that factor with the geometric mean of corpus precisions produces the baseline score; the paper does not introduce a sentence-level smoothing rule. e06e07e08
5.3 3. Separate score scale, ranking stability and human agreement
Reader analysis: the evaluation answers three related questions with different evidence. Table 1 supplies a ranking under one shared reference protocol. Table 2 asks whether paired blocks preserve differences, using smaller-corpus BLEU values rather than independent test sets. Figures 3–6 compare that metric with human judgments, exposing both strong correlations and a disagreement over the two human translators. These findings should not be collapsed into one claim that BLEU measures quality perfectly. The reference-count experiment changes the same human translator’s score, showing that the numerical scale depends on available references. Figure 7 then normalizes each measure by its own observed range, which aids shape comparison but removes the original scales. A faithful comparison must state which of these questions and protocols it addresses. e09e10e11e12e13e14e15e16
5.4 Training and inference
During training
BLEU has no learned encoder, optimizer, training objective, frozen module or action policy. The baseline fixes N=4 and uniform weights. The authors selected this configuration through correlation experiments: orders 3 and 5 were comparable, and geometric averaging correlated slightly better than their best arithmetic average; exact alternative scores are not supplied. e06e08
During inference
Evaluation consists of deterministic reference matching and aggregation after translations have been produced. It neither generates translations nor executes actions. Reference construction remains a human prerequisite. No gradient update, planning loop or learned prediction stage is defined. e02e04e08
5.5 Implementation flow
- Match locally and clip counts
Apply case folding, the only stated text normalization. Within each sentence pair, match candidate n-grams against its references without requiring the same absolute position. Clip using the maximum count in one reference, not the sum across references. In Example 2, seven repetitions of “the” receive unigram credit 2/7 rather than 7/7. e03e08
- Pool counts before dividing
For each n-gram order, sum clipped matches across candidate sentences and divide by all candidate n-grams at that order. Matches do not cross sentence boundaries. This is corpus precision, not an average of individual sentence precisions or sentence BLEU scores. e04
- Combine phrase scales and penalize short output
Use orders 1 through 4 with equal weights in log space. For each candidate sentence choose the closest reference length, sum those lengths into the effective reference length, and compare that total with candidate corpus length. The penalty applies once to the corpus; it does not use source-sentence length. e06e07e08
6. Experiments & results
BLEU evaluates translated text through clipped reference n-gram overlap and a corpus brevity penalty. Its Chinese-to-English study reports strong agreement with aggregate human judgments across three machine systems and two human translators. The evidence supports efficient corpus-level comparison under a shared reference protocol, with explicit limits on sentence scoring and cross-reference comparisons.
This is a fixed evaluation metric, so the source contains equations and diagnostic examples rather than a learned-model architecture diagram. Figure 2 is used to explain the method’s n-gram-order rationale. There is no quantitative component-removal ablation: geometric versus arithmetic averaging and alternative maximum orders are discussed without alternative numerical scores. Table 2 and Figures 3–4 supply reliability and human-criterion diagnostics, explicitly labeled as such, rather than invented ablation results. e05e06e08e11e13e14
6.1 Read the original evidence
Table 1. The full-corpus ranking under a shared two-reference evaluation. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the columns as five translation producers, not five datasets. S1–S3 are commercial machine systems; H1 and H2 are human translators with different language backgrounds. Each entry is the final BLEU score computed from the approximately 500-sentence corpus against two professional references. The source describes 40 general-news stories. Higher values mean more reference agreement under the clipping, geometric-mean and brevity rules. Compare S2 with S3 first to identify the close machine pair, then compare S3 with H1 to see the larger machine–human separation. Keep these corpus scores separate from Table 2’s averages over smaller blocks. e05e08e09e10e11
What it supports. The reported ordering is H2 > H1 > S3 > S2 > S1. S3 scores 0.0930 against S2’s 0.0829, while H1 scores 0.1934. The table establishes this comparison on the shared corpus and reference set; the separate block analysis asks whether the close machine difference is reliable.
Where the evidence stops. These are scores on the original 0–1 scale, not success percentages. Section 3 reports a human score changing from 0.2571 with two references to 0.3468 with four, so reference-count changes invalidate a direct score comparison.
Figure 5. A strong association across five systems, with a limited calibration claim. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the horizontal axis as BLEU on the original decimal scale. Red diamonds identify monolingual-group observations; blue circles joined by a line identify the regression’s predicted values. Each observation represents a system, not an individual sentence or judge. Three low-BLEU points correspond to the machine systems, while the two higher-BLEU points correspond to human translators, as Table 1 establishes. Section 5 reports a correlation coefficient of 0.99 for this fit and 0.96 for the separate bilingual plot in Figure 6. Use those reported coefficients rather than estimating them from the drawn line or rounded plotted coordinates. e10e12e14e15
What it supports. The five system-level points closely follow a line, supporting BLEU’s agreement with the monolingual evaluation in this experiment. The bilingual correlation is also high, but its human-pair diagnostic differs. The source therefore supplies evidence of aggregate association, not a universal or sentence-level measure of translation correctness.
Where the evidence stops. Section 4 specifies 1–5 ratings, yet this plot reaches approximately zero without explicitly defining its axis transformation. Preserve that ambiguity: do not interpret the line as a calibrated raw-rating predictor. No independent regression test set or correlation interval is reported.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Chinese-to-English corpus system ranking About 500 sentences from 40 general-news stories; three commercial MT systems and two human translators, scored against two professional references. | S1 0.0527; S2 0.0829; S3 0.0930; H1 0.1934; H2 0.2571. Corpus BLEU, higher is better; original 0–1 scale | H2 > H1 > S3 > S2 > S1. H1 lacks native proficiency in both languages; H2 is a native English speaker. One shared evaluation ranks these five outputs. These are full-corpus scores, not block means, percentages or sentence-level accuracies. e05e09e10 |
| BLEU agreement with human system judgments Five systems; two-reference BLEU. Two groups of 10 nonprofessional judges evaluated 250 source/translation pairs sampled from the corpus, rating translations from 1 to 5. | 0.99 for monolingual judgments; 0.96 for bilingual judgments. Reported linear correlation coefficient across five systems | Monolingual judges were native English speakers assessing readability/fluency; bilingual judges were native Chinese speakers resident in the United States. Strong system-level association in this study. It is not sentence-level accuracy or an out-of-sample prediction test; no correlation confidence interval is reported. e12e15 |
| Close-system block reliability The same corpus partitioned into 20 blocks of 25 sentences, with BLEU computed separately on each block. | S2 mean 0.081, SD 0.025; S3 mean 0.090, SD 0.020; paired t=3.4. Block mean, block standard deviation and paired t-statistic | Neighbor-pair t values are 6 for S2/S1, 24 for H1/S3 and 11 for H2/H1. The authors report significant separation. Block SD is not a confidence interval for the 500-sentence corpus, and these means use a different aggregation from Table 1. e11 |
| Reference-count sensitivity The same human translator on the approximately 500-sentence test corpus; reference count changes. | 0.3468 with four references; 0.2571 with two references. Corpus BLEU | A separate single-reference simulation selected one of four translations per story and retained the system ordering; no scores for that simulation are given. Reference resources change the score scale. The qualitative rank result does not establish score comparability across reference counts. e09 |
| Human-criterion sensitivity for H2 versus H1 The two judge groups assess the same translation material using the Section 4 protocol. | Monolingual: 0.535 [0.400, 0.670]. Bilingual: 0.038 [−0.069, 0.145]. Mean rating difference and reported 95% confidence interval | The monolingual interval excludes zero; the bilingual interval includes zero. Human-group agreement is incomplete. The authors suggest greater bilingual emphasis on adequacy; the experiment does not isolate why the groups differ. e12e13e14 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 2. Block analysis tests the stability of differences between translation producers. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow the rows before comparing columns. Mean is the average of twenty BLEU scores, each computed on a 25-sentence block. StdDev describes variation among those block scores. The t row uses paired block observations and compares each column with its immediate left neighbor; consequently, 3.4 under S3 means S3 versus S2, not S3 versus S1. The dash under S1 indicates no preceding comparator. The source explicitly distinguishes these block means from Table 1’s one score over the whole corpus. This distinction matters because averaging a nonlinear score over blocks is not the same operation as pooling all counts first. e04e10e11
What it supports. S2’s block mean is 0.081 with SD 0.025; S3’s is 0.090 with SD 0.020. Their paired t-statistic is 3.4, supporting the authors’ reported separation under this block protocol. The adjacent comparisons involving human translators have substantially larger t-statistics, 24 and 11.
Where the evidence stops. This is a reliability diagnostic, not a component ablation or independent test corpus. The source states a 1.7 cutoff for 95% significance without specifying test sidedness. Block SD must not be relabeled as a full-corpus confidence interval.
Figure 3. English-speaking judges distinguish even the close machine pair. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Each horizontal category is a subtraction: a positive S3−S2 value means S3 received higher ratings. The blue markers show mean differences, with vertical spans for the reported 95% intervals. The numeric rows below give upper endpoint, lower endpoint and mean, in that order; the +95% and −95% labels are interval endpoints, not percentage improvements. These judges were ten native English speakers assessing readability and fluency on a 1–5 rating scale. Look first at S3−S2, whose interval stays just above zero, and then at H2−H1, whose larger positive difference contrasts with the bilingual result. e12e13e14
What it supports. The monolingual S3−S2 mean difference is 0.114 with interval [0.034, 0.194], supporting the small preference also reflected in BLEU. H2−H1 is 0.535 [0.400, 0.670]. Thus these judges separate both the close machine outputs and the two human translators under the reported tests.
Where the evidence stops. These are human rating differences, not BLEU units. Some ratings were skipped, and the source’s tests use the available judge/sentence comparisons. The figure does not establish that every individual judge or sentence follows the aggregate ordering.
Figure 4. The bilingual group exposes a boundary to a single human-quality ranking. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use the same subtraction convention as in Figure 3. Blue markers indicate the bilingual group’s mean differences, while the numeric rows list upper bound, lower bound and mean. These ten judges were native Chinese speakers who had lived in the United States for several years; none was a professional translator. Focus on H2−H1: its interval spans zero, unlike the monolingual interval. Then inspect S3−S2, which remains positive. The source interprets the human-pair contrast as suggesting greater attention to adequacy among bilingual judges, but the different groups also differ in language background and are different people. e12e13e14
What it supports. The bilingual S3−S2 difference is 0.140 with interval [0.042, 0.238], agreeing on the close machine ranking. H2−H1 is only 0.038 with interval [−0.069, 0.145], so the source reports no significant human-pair separation at 95% confidence. This qualifies claims of complete agreement with human judgment.
Where the evidence stops. The experiment does not independently manipulate adequacy versus fluency while holding judges fixed. Treat the proposed explanation as the authors’ interpretation. An interval crossing zero is insufficient evidence of a difference, not proof that the translations are equivalent.
7. Analysis & limitations
7.1 What the evidence leaves open
Sentence-level lexical alternatives can receive poor scores despite fluent expression. The paper explicitly relies on averaging errors over a corpus, and notes harsh behavior when any n-gram precision is zero. No smoothing variant is defined. e04e06
Five systems spanning machine and human quality provide limited evidence about discrimination among closely matched future systems. The separate multilingual study is cited, not reproduced here; summarization is proposed as an adaptation without an experiment. e05e15e17
Figures 5–6 plot human judgments near zero although Section 4 describes 1–5 ratings. The plot transformation is not explicitly defined there. Their correlations can be reported, but the vertical axes should not be treated as calibrated raw rating predictions. e12e15
7.2 Questions for discussion
- How much of the high system-level correlation survives when only closely matched machine outputs are compared?
- How sensitive are system rankings to reference style and to the human evaluation criterion?
8. Reproducibility audit
8.1 Requirements and known gaps
Recreating the published tables requires the exact corpus, professional references and five candidate outputs; the human comparison additionally requires the sampled sentence pairs and judge ratings. The paper gives corpus size and collection design, but not a named downloadable test set, system product identities, raw ratings or a released implementation. e05e09e10e12
An implementation should preserve per-sentence clipping, corpus pooling, closest-reference lengths, case folding and the original weights. Tokenization/punctuation rules, closest-length tie resolution, and empty-input behavior are not fully specified; the paper supplies no compute or software configuration. These conventions require explicit documentation before claiming exact replication. e03e04e07e08
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Check clipping and brevity separately before full-score replication
Reader-proposed check, not performed: implement the source rules and instrument clipped counts, p_n, c, r and BP. Use Example 2 to compare ordinary and clipped unigram precision, expecting 7/7 versus 2/7 with candidate/reference text fixed. Use Example 3 to confirm perfect unigram/bigram precision but BP below one; do not assign an invented four-gram score to this two-word candidate. On a corpus with nonzero precision at every order, duplicate every candidate/reference pair and verify unchanged BLEU. Failure of the first comparison identifies clipping errors; failure of duplication invariance identifies aggregation or length accounting errors. Document unresolved tokenization and closest-length ties. e03e04e06e07e08
Check 2: Test reference stability without relying on the human–machine gap
Reader-proposed check, not performed: obtain fixed candidate outputs, four references per story and fixed human judgments under a documented comparable protocol. Repeatedly choose one reference per story, matching Section 3’s stylistic-diversity design, and compare with two- and four-reference scoring. Keep sentences, outputs, normalization and scoring weights fixed. Report score shifts and how often S3 remains above S2, separately from correlation across all five systems. If the close machine ordering frequently reverses while overall correlation remains high, the broad human–machine quality range is insufficient evidence for reliable fine-grained comparison. If original artifacts are unavailable, label this a new validation experiment rather than numerical replication. e05e08e09e10e11e12e15
8.3 Reading coverage
Visual audit: All eight pages were rendered and visually inspected, including the title/author block, Examples 1–4, clipping and corpus-precision equations, BP/BLEU branch inequalities, Figures 1–7, Tables 1–2, protocol footnotes and the conclusion/references. Every final crop was inspected at its native rendered resolution; the Figure 5 crop was widened to retain the complete axes. Table column order, chart legends, confidence bounds and plotted regression markers were checked against the text. The Figure 5/6 rating-axis ambiguity is disclosed. No appendix is present; separate supplements and external cited works remain outside the supplied material.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8. Appendix coverage: not present.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract; Sections 1.1–1.2 (PDF p. 1)
- Section 2, including 2.1.1–2.1.3, 2.2.1–2.2.2 and 2.3; Examples 1–4 and all equations (PDF pp. 2–5)
- Section 3: BLEU evaluation (PDF pp. 5–6)
- Section 4, including 4.1–4.2: human evaluation (PDF pp. 6–7)
- Section 5: BLEU versus human evaluation (PDF pp. 7–8)
- Section 6, acknowledgments and references (PDF p. 8)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- The inspected title is BLEU: a Method for Automatic Evaluation of Machine Translation; the catalog differs only in the capitalization of Bleu. The four authors agree. The PDF identifies ACL proceedings, Philadelphia, July 2002, pp. 311–318; no separate revision identifier or revision history is supplied, so equivalence to other editions is not established.
- The supplied extraction does not reconstruct figure images. This limitation was addressed by visually inspecting all eight PDF pages, Figures 1–7, Tables 1–2 and all six final crops.
- Separate supplemental material availability has not been fully verified; no supplement was supplied.
- Code and external cited works were not inspected. No scoring implementation or experiments were run.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, proceedings header and title/author block
Observed title: BLEU: a Method for Automatic Evaluation of Machine Translation. Authors: Kishore Papineni, Salim Roukos, Todd Ward, Wei-Jing Zhu. Affiliation: IBM T. J. Watson Research Center, Yorktown Heights, NY 10598, USA. Header identifies ACL, Philadelphia, July 2002, pp. 311–318.
Go to primary source ↓e02PDF p. 1, Abstract and Sections 1.1–1.2
The paper motivates inexpensive repeatable evaluation through closeness to good human references, and calls BLEU an understudy to skilled judges.
Go to primary source ↓e03PDF p. 2, Section 2.1, Example 2 and footnote 2
Counts are capped by maximum occurrence in any single reference. Example 2 receives modified unigram precision 2/7 versus ordinary precision 7/7.
Go to primary source ↓e04PDF p. 3, Section 2.1.1, displayed corpus-precision formula and footnote 4
Matches are computed sentence by sentence, then clipped counts and candidate counts are pooled. The footnote warns that sentence scores can disagree with humans because legitimate phrases may differ from references.
Go to primary source ↓e05PDF pp. 3–4, Section 2.1.2 and Figures 1–2
Figure 1 compares a human and an MT system on 127 sentences with four references. Figure 2 compares three commercial MT systems and two humans against two references; the surrounding text reports H2 > H1 > S3 > S2 > S1.
Go to primary source ↓e06PDF p. 4, Section 2.1.3 and footnotes 5–6
Precisions decay with n-gram order. The baseline uses a uniform log average; order four gives best monolingual correlation, with three and five comparable. Geometric averaging slightly improves correlation over arithmetic averaging; zero precisions are harsh.
Go to primary source ↓e07PDF pp. 4–5, Sections 2.2–2.2.2 and Examples 3–4
Precision alone rewards a short matching phrase, while naive multi-reference recall rewards redundant alternatives. Effective reference length sums the closest sentence lengths; brevity is penalized at corpus level using target-language lengths.
Go to primary source ↓e08PDF p. 5, Section 2.3 and displayed BP/BLEU/log-BLEU equations
Only case folding is stated as normalization. BP=1 for c>r and exp(1−r/c) otherwise. BLEU multiplies BP by the weighted geometric mean of p_n; N=4 and w_n=1/N.
Go to primary source ↓e09PDF p. 5, Section 3; PDF p. 6, Section 3 final paragraph continuing into right column
BLEU ranges from 0 to 1. On about 500 sentences in 40 news stories, a human scores 0.3468 with four references versus 0.2571 with two. A story-level random single-reference selection retains the ordering; its numerical scores are omitted.
Go to primary source ↓e10PDF p. 6, Table 1, all columns; PDF p. 5, Section 3 protocol
Two-reference corpus scores are S1 0.0527, S2 0.0829, S3 0.0930, H1 0.1934 and H2 0.2571.
Go to primary source ↓e11PDF p. 6, Table 2 and Section 3 block-analysis paragraphs
Twenty blocks contain 25 sentences each. Means: 0.051/0.081/0.090/0.192/0.256; SDs: 0.017/0.025/0.020/0.030/0.039. Each t compares with the left neighbor: 6, 3.4, 24, 11. The text distinguishes block means from full-corpus scores.
Go to primary source ↓e12PDF p. 6, Section 4 and footnote 7
Each judge group contains ten people, none a professional translator: native English monolinguals and native Chinese bilinguals. The sampled evaluation has 250 source/translation pairs, a shared randomized display order and 1–5 ratings. Monolinguals rate readability/fluency; skipped ratings yield 350–470 degrees of freedom for the reported tests.
Go to primary source ↓e13PDF pp. 6–7, Section 4.1 and Figure 3, plotted points and numeric rows
Monolingual S2−S1 is 0.326 with 95% interval [0.252,0.400]; S3−S2 is 0.114 [0.034,0.194]; H1−S3 is 1.825 [1.705,1.945]; H2−H1 is 0.535 [0.400,0.670].
Go to primary source ↓e14PDF p. 7, Section 4.2 and Figure 4, plotted points and numeric rows
Bilingual S3−S2 is 0.140 [0.042,0.238]; H2−H1 is 0.038 [−0.069,0.145]. The authors report the human pair indistinguishable at 95% confidence and suggest greater attention to adequacy.
Go to primary source ↓e15PDF p. 7, Section 5 and Figures 5–6
Regressions across five systems using two-reference BLEU report correlations 0.99 for monolinguals and 0.96 for bilinguals. Blue predicted points/lines and red group points are shown. Plotted human values reach approximately zero, and the exact axis transformation is not stated in this passage.
Go to primary source ↓e16PDF p. 8, Section 5 continuation and Figure 7
Scores are normalized relative to the worst system and each measure’s observed range. Figure 7 compares BLEU, monolingual and bilingual normalized scores and shows their different H1–H2 gaps.
Go to primary source ↓e17PDF p. 8, Section 6 and References
The conclusion attributes strength to corpus averaging, cites a separate Arabic/Chinese/French/Spanish evaluation, and proposes possible adaptation to summarization; it presents no summarization experiment or appendix.
Go to primary source ↓8.5 Primary sources
BLEU: a Method for Automatic Evaluation of Machine Translation ↗
PDF · 4,628 extracted words
Source fingerprint
c0ee6aedcb674da26a640578ead33f66eb5bec87daa2656d4f845da37e096997