ROUGE: A Package for Automatic Evaluation of Summaries
1. Paper overview
In one sentence: ROUGE turns reference-summary overlap into inexpensive evaluation scores, but its agreement with human content judgments depends on the matching rule, reference policy and summarization task. e02e03e05e07e08e09e10e11e12e13e14
| At a glance | What to know |
|---|---|
| Research problem | Source description Frequent manual summarization evaluation is expensive. ROUGE asks whether inexpensive overlap with human references can track human judgments. The empirical target is content coverage; the introduction also names coherence, grammaticality and readability, but those are not the ground truth used in the correlation study. e02e10 |
| Core mechanism | |
| A key reported result | DUC 2003 10-word single-document summary evaluation: Reported ROUGE-SU4: 0.97; ROUGE-L: 0.97. Pearson correlation with human average content coverage; higher is better. 14 systems, 624 summaries judged per system; Table 2, CASE, one reference. ROUGE-S4: 0.88; ROUGE-2: 0.75. In STOP with one reference, SU4 is 0.98 and S4 is 0.95. The SU4/S4 comparison probes adding unigram credit. Useful configurations differ from the 100-word task; these are correlations, not summary-quality percentages. e09e10e12 |
| Reading caution |
Core contributions
Section 4, unnumbered weighted-LCS sequence example. Identical LCS coverage can conceal very different spacing between matches. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the first row as the reference and the next two as candidate summaries represented by tokens. The underlines select A, B, C and D in each row. Their order is unchanged, so ordinary LCS finds the same four-token subsequence for both candidates. What changes is the spacing: Y1 keeps the matches together, while Y2 inserts unrelated tokens between them. Section 4 introduces weighting precisely to distinguish these cases. The diagram isolates that scoring preference because all three sequences have the same length; differences cannot be attributed to a longer candidate or reference. e07e10
What it supports. With quadratic weighting, the paper reports ROUGE-W scores of 0.571 for Y1 and 0.286 for Y2. Ordinary ROUGE-L ties them. This supports the narrow mechanical claim that weighted runs distinguish contiguous from separated matches even when the matched subsequence length is unchanged.
Where the evidence stops. This is a constructed token example, not a human-judgment experiment. Quadratic weighting is used here, whereas the correlation experiments use exponent 1.2. The example does not establish that contiguous matches always preserve meaning better.
2. Motivation
2.1 The problem and the proposed response
Frequent manual summarization evaluation is expensive. ROUGE asks whether inexpensive overlap with human references can track human judgments. The empirical target is content coverage; the introduction also names coherence, grammaticality and readability, but those are not the ground truth used in the correlation study. e02e10
2.2 What this reading follows
A summary can repeat the right words while arranging them poorly, or express useful content without copying a long phrase. ROUGE explores that tension through several matching rules: n-grams, subsequences, weighted runs and ordered word pairs. This reading follows the original weighted-LCS example into its scoring algorithm, then examines all three validation tables. The central empirical question is how well average metric scores track average human content-coverage judgments across summarization systems. The answer changes with summary length, document setting, preprocessing and reference count. These tables therefore support a conditional evaluation protocol, rather than a single universally reliable score. e02e03e05e07e08e09e10e11e12e13e14
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Evaluation metrics |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
3.1 Evidence-based assessment
Supports the recorded classification
The evaluation-metrics classification is supported: ROUGE scores generated language and studies evaluation protocols. Architecture, prediction paradigm and quadrant are not applicable because its computations contain no world-state predictor or action-producing model. This is an applicability judgment, not an unverified One Model classification. e02e03e05e07e08e10
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Choose what a match should preserve
Begin with the reference as a collection of content that a candidate might cover. ROUGE-N asks how many reference n-grams reappear, making local phrase identity the matching unit. ROUGE-L instead follows an ordered subsequence, allowing gaps; the paper’s sentence examples show why word order can distinguish candidates that share a bigram. But LCS credits only a longest subsequence, leaving alternative matches uncounted. ROUGE-W adds a preference for consecutive runs, as the first two visuals illustrate. ROUGE-S takes another route by counting ordered word pairs, and ROUGE-SU also credits individual words. Each change answers a different failure case in the preceding definitions. None of these rules compares propositions or verifies whether a statement is true; interpreting lexical overlap as broader semantic quality would go beyond their defined inputs. e03e05e07e08
Section 4, unnumbered WLCS dynamic-programming procedure. The extra run-length table determines how much an additional matched token is worth. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow the equality test for the current reference and candidate tokens. On a match, the algorithm reads the preceding diagonal run length k from w, extends the accumulated score c by f(k+1)−f(k), and stores k+1. On a mismatch, the strict greater-than test chooses the upper score if it exceeds the left score; otherwise the left score is copied. Both mismatch branches reset the run length to zero. The final c(m,n) is the weighted sequence score. Equations (13)–(15) subsequently normalize it and combine recall with precision; this display alone does not produce the final ROUGE-W measure. e07
What it supports. The score increment depends on the run already matched. Together with the source condition f(x+y)>f(x)+f(y), this explains why uninterrupted runs receive extra credit. The algorithm is a deterministic sequence comparison; its table entries are intermediate computations, not learned representations or a model-training objective.
Where the evidence stops. The original initialization loops over i but assigns c(i,j) and w(i,j) without specifying j. The crop preserves that omission. A reproduction must explicitly resolve boundary initialization; the displayed recurrence and normalization should not be mistaken for a complete executable implementation.
5.2 Separate sentence matching from reference aggregation
There are two distinct aggregation questions. Within summary-level ROUGE-L, a reference sentence can find matching tokens across several candidate sentences; the paper takes their union rather than choosing only one candidate sentence. Across alternative human references, Section 2.1 describes choosing the highest pairwise score and averaging such scores over leave-one-reference-out sets. This latter procedure also enables a human reference to be evaluated against the others. Reader analysis: neither operation can be replaced casually by concatenating every sentence and reference, because that changes which matches and denominators are counted. The source is also imperfect: Equation (1) presents pooled reference counts, Section 2.1 prints argmax while discussing a maximum score, and the union example mixes a normalized fraction with count-based equations. These unresolved details belong in a historical replication specification. e03e04e06
5.3 Validate the evaluator at the level actually tested
The evaluation pipeline starts with human coverage judgments against one manual summary, even when additional references exist for automatic scoring. It then averages both human and ROUGE scores within each summarization system and computes correlations across systems. Thus a table cell measures agreement among system averages; it does not validate every individual summary. Compare metric rows only after fixing task, reference count and preprocessing. The headline SU4/S4 pair asks whether unigram credit helps under matched conditions; the multi-document CASE/STOP pair asks about preprocessing. Reader analysis: comparing those two tasks directly cannot isolate either mechanism, because length, systems and sample counts differ. Bootstrap uncertainty is described but not numerically tabulated, and the small-sample explanation is proposed rather than experimentally isolated. Strong point estimates justify further validation, not universal deployment claims. e08e09e10e11e12e13e15
5.4 Training and inference
During training
ROUGE has no learned parameters or optimization stage. Experiments use package v1.2.1 with original summaries (CASE), Porter-stemmed summaries (STEM), or stopwords removed (STOP). These are evaluation configurations, not training splits. e03e07e08e10
During inference
Scoring compares a candidate with references under the selected matching and aggregation rule. For DUC, the ROUGE-L and ROUGE-W descriptions use a very large beta so recall dominates. Evaluation averages scores per system before correlating them with human averages; ROUGE neither generates summaries nor executes actions. e04e05e07e10
5.5 Implementation flow
- Count reference n-gram coverage
ROUGE-N divides matching n-gram counts by reference-side counts. Equation (1) sums over references, giving repeated reference content additional weight. This is recall-oriented; it does not directly penalize every extra candidate word. e03
- Preserve order with LCS
Sentence-level ROUGE-L normalizes longest-common-subsequence length by reference and candidate lengths. Matches need not be consecutive. Summary-level ROUGE-L unions matches across candidate sentences for each reference sentence, whereas the compared normalized pairwise LCS takes the best sentence match. e05e06
- Reward consecutive runs
ROUGE-W stores an accumulated weighted score and the current matching-run length. Extending a run adds f(k+1)−f(k); a superadditive weighting function favors contiguous matches. The paper illustrates quadratic weighting and evaluates a polynomial exponent of 1.2. e07e10
- Count ordered pairs and recover unigram credit
ROUGE-S counts matching ordered word pairs, optionally limiting their gap and adjusting denominators to eligible pairs. ROUGE-SU also counts unigrams, allowing credit when a sentence shares words but no ordered pair. These remain lexical matching rules. e08
- Specify reference aggregation
Section 2.1 describes the highest pairwise score, then averaging best scores over leave-one-reference-out sets to support human comparison. This differs from Equation (1)’s pooled presentation. The displayed argmax denotes an index mathematically, although the prose requests a score; retain this ambiguity when implementing. e03e04
6. Experiments & results
ROUGE measures reference-summary overlap through n-grams, longest common subsequences, weighted subsequences and ordered word pairs. Lin validates these deterministic scores against human content-coverage judgments on DUC summarization tasks. Strong system-level correlations in single-document summarization weaken or change across headline and multi-document settings, making metric variant, preprocessing and reference policy essential parts of the evaluation.
This metric paper contains no numbered architecture diagram and no learned-model component ablation. Its original sequence example and dynamic-programming display serve as method visuals, while the three original correlation tables provide quantitative results and matched metric/preprocessing diagnostics. The source supplies no training curves, execution results or numerical confidence-interval endpoints; none are invented. e03e07e08e10e11e12e13e14
6.1 Read the original evidence
Table 1. Very high correlations in DUC 2002 describe system-average content agreement. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. First choose a year, then the reference-count block, then a preprocessing column: CASE uses original summaries, STEM applies Porter stemming, and STOP removes stopwords. Rows identify metric variants, with R-W-1.2 naming the weighted exponent. Every cell is a Pearson correlation between system-average metric scores and system-average human coverage scores, not an average ROUGE score. Section 6 describes green as best and gray as statistically equivalent to best; these are source markings, not independently recomputed tests. Compare rows within one column before comparing years, because the number of systems and the judged summaries per system change across years. e09e10e11
What it supports. In the 2002 one-reference CASE column, ROUGE-2, ROUGE-L and ROUGE-W-1.2 each report 0.99, versus ROUGE-1 at 0.98. In 2001 CASE, ROUGE-2 changes from 0.84 with one reference to 0.87 with three. Reference count and dataset setting therefore belong beside each quoted value.
Where the evidence stops. The 0.99 values are correlations across system averages, not 99% accurate summaries. Numeric confidence-interval endpoints are absent. The starred S/SU rows are preserved literally because the supplied text does not explicitly define their asterisk.
Table 2. Adding unigram credit is useful in the reported headline setting. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. This table groups columns by preprocessing first, unlike Table 1: within each CASE, STEM or STOP block, select one or four references. Start with the one-reference CASE column and compare R-S4 with R-SU4. Section 5.1 defines the latter as the skip-bigram measure extended with unigram credit, so these rows provide a mechanism-relevant comparison. Then move to the one-reference STOP column to see whether that separation persists after stopword removal. The task contains about ten words per summary, with 14 systems and 624 judged summaries per system; the correlations still operate on system averages, not 624 independent system-level observations. e08e09e10e12
What it supports. ROUGE-SU4 reports 0.97 versus ROUGE-S4 at 0.88 in one-reference CASE. In one-reference STOP, their values are 0.98 and 0.95. ROUGE-L also reaches 0.97 in CASE, while ROUGE-2 is 0.75. The table favors different matching choices than a blanket preference for longer n-grams would suggest.
Where the evidence stops. The pairwise differences are reported point estimates, not supplied confidence intervals for SU4 minus S4. Adding lexical credit need not improve factuality: Section 5.1 motivates it partly with a reversed-word sentence that shares vocabulary.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| DUC 2002 100-word single-document summary evaluation 14 systems, 295 summaries judged per system; Table 1, CASE, one reference. Correlations use system averages. | Reported r = 0.99 for ROUGE-2, ROUGE-L and ROUGE-W-1.2. Pearson correlation with human average content coverage; higher is better | ROUGE-1: 0.98 under the same setting. High agreement for these system averages; rounding and omitted interval endpoints prevent treating the 0.01 difference as a demonstrated advantage. e09e10e11 |
| DUC 2003 10-word single-document summary evaluation 14 systems, 624 summaries judged per system; Table 2, CASE, one reference. | Reported ROUGE-SU4: 0.97; ROUGE-L: 0.97. Pearson correlation with human average content coverage; higher is better | ROUGE-S4: 0.88; ROUGE-2: 0.75. In STOP with one reference, SU4 is 0.98 and S4 is 0.95. The SU4/S4 comparison probes adding unigram credit. Useful configurations differ from the 100-word task; these are correlations, not summary-quality percentages. e09e10e12 |
| DUC 2001 100-word multi-document preprocessing diagnostic 14 systems, 29 summaries judged per system; Table 3 A1, three references; CASE versus STOP. | Reported ROUGE-1: 0.53 (CASE), 0.87 (STOP). Pearson correlation with human average content coverage; higher is better | ROUGE-L: 0.56 in both; ROUGE-W-1.2: 0.60 in both. Stopword removal changes ROUGE-1 correlation by 0.34 correlation units here. This is a preprocessing diagnostic, not a trained-model ablation or a universal improvement. e09e10e13 |
| DUC 2002 200-word multi-document summary evaluation 10 systems, 59 summaries judged per system; Table 3 E2, multiple references, STOP. | Reported ROUGE-2: 0.94. Pearson correlation with human average content coverage; higher is better | ROUGE-1: 0.91; ROUGE-L: 0.88; ROUGE-W-1.2: 0.90. Longer multi-document summaries can show strong agreement. Keep this task separate from shorter-summary settings and their sample counts. e09e10e14 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 3. Preprocessing can change correlation substantially, while task length changes the comparison itself. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use the upper panels to keep summary length at 100 words. Select A1, three references, then compare CASE with STOP in the same row: this isolates the reported preprocessing change for that metric and setting. The lower panels instead cover different summary lengths and years; Section 6 says those results use multiple references. Do not pool their columns into one leaderboard. For a longer-summary example, read E2, STOP. Although Section 6 says green marks column-best values, A1 one-reference CASE highlights R-L at 0.49 while R-2 is 0.55. Follow the printed values and preserve this inconsistency. e09e10e13e14e15
What it supports. In A1 with three references, ROUGE-1 rises from 0.53 in CASE to 0.87 in STOP, whereas ROUGE-L stays at 0.56. In E2 STOP, ROUGE-2 reaches 0.94. These observations support sensitivity to preprocessing and task setting; they do not establish one configuration as best across multi-document summarization.
Where the evidence stops. This is a metric/preprocessing diagnostic, not a learned-component ablation. The author’s small-sample explanation for weaker multi-document agreement remains a hypothesis here. The source shading inconsistency also prevents treating green cells as a reliable numeric ranking.
7. Analysis & limitations
7.1 What the evidence leaves open
System-average content correlations do not establish reliable sentence-level factuality, grammaticality or semantic equivalence. The source itself shows LCS missing alternative subsequences and skip-bigrams awarding potentially spurious pairs. e02e05e08e10
The author proposes small multi-document sample counts as an explanation for unstable correlations and leaves improving multi-document agreement open. This paper does not establish that explanation through a controlled sample-size experiment. e15
Implementation ambiguities remain: pooled versus best-reference aggregation; a union-LCS example reporting 4/5 where the formula expects a count before normalization; and WLCS initialization leaving j unspecified. Tables use R-S* and R-SU*, while Section 6 lists skip distances 1, 4 and 9 without defining the asterisk. e03e04e06e07e10
Bootstrap 95% intervals are described, but only correlation values and shading are printed. Spearman and Kendall results are deferred. Exact intervals and additional significance claims cannot be recovered from those omissions. Table 3 also contains green shading inconsistent with the stated column-best rule; use numerical cells rather than color alone. e10e13
7.2 Questions for discussion
- How much of a metric’s apparent advantage survives changing the reference set while keeping candidate summaries fixed?
- Would the headline SU4 advantage persist for individual-summary judgments rather than system averages?
- How should lexical coverage be complemented when factual correctness and readable ordering matter?
8. Reproducibility audit
8.1 Requirements and known gaps
A historical replication requires DUC system outputs, manual references and human coverage judgments, plus package v1.2.1 and matching CASE/STEM/STOP settings. Preserve task lengths, reference counts and system-level averaging; use the reported exponent 1.2 for the weighted variant. e09e10
Complete tokenization, sentence segmentation, stopword-list, command-line and bootstrap-resampling settings are not supplied. Hardware and runtime costs are not reported. Resolve these and the mathematical ambiguities before asserting equivalence to the historical implementation. e04e06e07e10
Proposed checks: verify the weighted-LCS toy ranking at fixed token lengths, then test the SU4/S4 correlation difference on identical DUC headline outputs with paired resampling. These distinguish a scoring mechanism from an aggregate association. e07e09e10e12
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Does weighted LCS isolate contiguity at fixed coverage?
Reader-proposed check, not performed: use the exact X, Y1 and Y2 token sequences in Section 4, with no stemming or stopword removal, identical lengths and beta=1. Compute ordinary LCS and weighted LCS with f(k)=k², recording the c and w tables and an explicit boundary-initialization rule. The ordinary LCS scores should tie, while the final normalized weighted scores should match the reported 0.571 and 0.286 to rounding. A tie under quadratic weighting or a mismatch in these values would falsify implementation agreement on the paper’s central example. Repeat with exponent 1.2 as a separate configuration; do not expect the quadratic example’s numbers. e07e10
Check 2: Does unigram credit improve headline correlation under paired controls?
Reader-proposed check, not performed: use the same DUC 2003 headline outputs, human judgments and fixed single-reference selection for ROUGE-S4 and ROUGE-SU4. Hold tokenization, gap convention and system averaging constant. Run CASE and STOP separately, seeking the reported S4/SU4 pairs 0.88/0.97 and 0.95/0.98. Resample the same document indices across both metrics and all systems, recomputing system means and their correlations each time; report the resampling rule and a paired interval for the difference. Failure to recover the point estimates indicates a protocol mismatch. An interval including zero would weaken a claimed robust SU4 advantage even if rounded point estimates agree. e08e09e10e12
8.3 Reading coverage
Visual audit: All four text chunks were read individually. All eight PDF pages were rendered and visually inspected: title and identity on p. 1; reference aggregation and LCS equations on pp. 1–3; union-LCS and normalized pairwise comparison on pp. 3–4; WLCS example, algorithm branches and normalization on p. 4; skip-bigram definitions, SU and dataset counts on pp. 4–5; software, preprocessing, human judgments, statistics and Tables 1–2 on p. 6; full Table 3 and discussion on p. 7; concluding limits and references on p. 8. Every final crop was separately viewed. Crop boundaries retain all displayed algorithm steps, sequence underlines, table headers, rows and colors; captions and surrounding body prose are excluded. No table-specific footnote was removed. The prose color explanation is conveyed in the guides, with the Table 3 discrepancy disclosed. Only the supplied paper was reviewed; linked code, other studies and external supplements remain outside this pass.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8. Appendix coverage: not present.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- PDF p. 1: title, author, affiliation, Abstract and Section 1 Introduction
- PDF pp. 1–2: Section 2 ROUGE-N and Section 2.1 Multiple References
- PDF pp. 2–4: Sections 3.1–3.3, sentence-level LCS, summary-level LCS and normalized pairwise LCS
- PDF p. 4: Section 4 ROUGE-W, example, dynamic program and Equations (13)–(15)
- PDF pp. 4–5: Sections 5 and 5.1, ROUGE-S and ROUGE-SU
- PDF pp. 5–7: Section 6 Evaluations of ROUGE and Tables 1–3
- PDF pp. 7–8: Section 7 Conclusions, Section 8 Acknowledgements and References
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- The extraction limitation above was addressed by visually inspecting all eight supplied PDF pages, including equations, examples, algorithm and all three tables; five original crops were also inspected.
- No code, linked websites, external studies or separate supplements were inspected; no experiments were reproduced.
- The supplied title and sole author match the catalog. The PDF shows no explicit paper revision identifier or revision date, so no edition difference is established. This report identifies the supplied artifact by its primary hash. The v1.2.1 designation in Section 6 is the evaluated software version, not a paper revision.
- The paper reports Pearson tables but omits the computed Spearman and Kendall tables and numerical bootstrap confidence-interval endpoints. They are not reconstructed here.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title and author block; PDF p. 6, Section 6, software-version paragraph
The observed title is ROUGE: A Package for Automatic Evaluation of Summaries. The sole author is Chin-Yew Lin, Information Sciences Institute, University of Southern California. No explicit paper revision marker is shown; Section 6 specifies evaluation package v1.2.1.
Go to primary source ↓e02PDF p. 1, Abstract and Section 1
ROUGE compares candidate and human summaries using overlapping units. The motivation is the cost of human summary evaluation; several linguistic quality dimensions are introduced.
Go to primary source ↓e03PDF p. 1, Section 2, Equation (1); PDF p. 2, Section 2 continuation before Section 2.1
ROUGE-N uses reference-side n-gram counts in its denominator and sums matching counts across references. Repeated reference n-grams receive more weight.
Go to primary source ↓e04PDF p. 2, Section 2.1, displayed multi-reference rule and jackknifing paragraphs
The prose selects the maximum pairwise score; the display prints argmax. The package is described as averaging best scores over M sets of M−1 references, supporting leave-one-reference-out human comparisons. The procedure also applies to L, W and S.
Go to primary source ↓e05PDF p. 2, Section 3.1, Equations (2)–(4); PDF p. 3, Section 3.1, S1–S4 examples
Sentence-level LCS recall and precision divide matched sequence length by reference and candidate lengths. Large beta gives recall in DUC. S2 and S3 receive 0.75 and 0.5 with beta=1; alternative subsequences can be missed.
Go to primary source ↓e06PDF p. 3, Section 3.2, Equations (5)–(7) and union example; PDF pp. 3–4, Section 3.3
Summary-level LCS unions reference-token matches across candidate sentences, while the compared normalized pairwise approach chooses a best sentence score. The worked union covers four of five reference tokens but denotes its LCS-union score as 4/5, despite the equations dividing by total lengths.
Go to primary source ↓e07PDF p. 4, Section 4, X/Y1/Y2 example, dynamic-programming display, weighting condition and Equations (13)–(15)
Both candidates have the same LCS length, but quadratic ROUGE-W scores are 0.571 and 0.286. The algorithm tracks c and w, adds f(k+1)−f(k) on a match and resets w on a mismatch. Its initialization loop names i but uses an unspecified j. Weighting must satisfy f(x+y)>f(x)+f(y).
Go to primary source ↓e08PDF pp. 4–5, Section 5, Equations (16)–(18); PDF p. 5, Section 5.1
ROUGE-S counts ordered pairs with optional gap limits and eligible-pair normalization. Unrestricted examples yield 0.5, 0.167 and 0.333 for S2–S4. Reversed S5 shares words but no ordered pair; SU adds unigram credit.
Go to primary source ↓e09PDF p. 5, Section 6, three dataset bullets and footnote 2
Single-document 100-word tasks contain 12/14 systems and 149/295 summaries per system in 2001/2002. The 2003 headline task has 14 systems and 624 summaries per system. Multi-document evaluations have 29/59/30 summaries per system in 2001/2002/2003; task-specific system counts include 14 for 2001 100 words and 10 for 2002 200 words. Counts include baselines.
Go to primary source ↓e10PDF p. 6, Section 6, human coverage, averaging, configurations, uncertainty and table-shading paragraphs; footnote 4
Human judgments use one manual summary and average unit-level coverage. Correlations compare system averages, using one or multiple references and CASE/STEM/STOP; stemming uses Porter. Version 1.2.1 tests 17 variants and weighted exponent 1.2. The skip-distance list reads 1, 4 and 9, while tables label starred S/SU rows without defining the star. Bootstrap 95% intervals are estimated, but endpoints are absent; Spearman/Kendall tables are deferred. The prose says green marks best values and gray marks values statistically equivalent to the best; Table 3 has at least one shading inconsistency (see e13).
Go to primary source ↓e11PDF p. 6, Table 1, DUC 2002 100 WORDS SINGLE DOC / 1 REF / CASE, rows R-1, R-2, R-L and R-W-1.2; DUC 2001 / CASE / R-2
For 2002 one-reference CASE, R-1 is 0.98 and R-2, R-L and R-W-1.2 are 0.99. For 2001 CASE, R-2 is 0.84 with one reference and 0.87 with three.
Go to primary source ↓e12PDF p. 6, Table 2, CASE / 1 REF and STOP / 1 REF, rows R-2, R-L, R-S4 and R-SU4
CASE one-reference correlations are R-2 0.75, R-L 0.97, R-S4 0.88 and R-SU4 0.97. STOP one-reference values are R-S4 0.95 and R-SU4 0.98.
Go to primary source ↓e13PDF p. 7, Table 3 A1, 3 REFS / CASE and STOP, rows R-1, R-L and R-W-1.2; 1 REF / CASE, rows R-L and R-2; PDF p. 6, Section 6 shading explanation
R-1 changes from 0.53 CASE to 0.87 STOP, while R-L remains 0.56 and R-W-1.2 remains 0.60. In A1 one-reference CASE, R-L 0.49 is green while R-2 0.55 is larger, inconsistent with the prose description of green as the column-best value.
Go to primary source ↓e14PDF p. 7, Table 3 E2, STOP, rows R-1, R-2, R-L and R-W-1.2; Section 6 paragraph describing panels C–F
For DUC 2002 200-word multi-document evaluation with multiple references, STOP correlations are R-1 0.91, R-2 0.94, R-L 0.88 and R-W-1.2 0.90. The lower table also covers 10-, 50- and 400-word tasks.
Go to primary source ↓e15PDF p. 7, Section 6 discussion comparing Tables 1–3; PDF pp. 7–8, Section 7 Conclusions
The author suggests fewer multi-document samples may destabilize human system judgments and correlations. Higher agreement on multi-document summarization remains open. Sample-size stability and machine-translation studies are cited separately, not presented here.
Go to primary source ↓8.5 Primary sources
ROUGE: A Package for Automatic Evaluation of Summaries ↗
PDF · 6,679 extracted words
Source fingerprint
d8100065f4ed5134fcde1a3e8689be0c05c4624d9580128109c576ed1d8a922e