Language Models are Few-Shot Learners
1. Paper overview
In one sentence: Scaling text pretraining makes a fixed language model better at following demonstrations in its context, with substantial gains but uneven task coverage and costly inference. e-identitye-settingse-architecturee-lambadae-qae-supergluee-computee-contamination
| At a glance | What to know |
|---|---|
| Research problem | Source description Task-specific fine-tuning needs labeled datasets and can exploit narrow-distribution correlations. The paper asks how far one pretrained language model can go when a new task is communicated through instructions and a few demonstrations. Sample efficiency here concerns downstream task specification, not the volume of pretraining data. e-probleme-settingse-limitations |
| Core mechanism | Source description An eight-model scaling study separates zero-, one- and few-shot evaluation, spanning 125M to 175B parameters. It extends both language-model loss measurements and downstream evaluation without fine-tuning GPT-3. e-architecturee-scaling |
| A key reported result | SuperGLUE aggregate and task variation: Aggregate 71.8; COPA 92.0%; WiC 49.4%. SuperGLUE score; component accuracy (%). Official test results; 175B; 32 demonstrations per task; no gradient updates. Fine-tuned BERT-Large aggregate 69.0; listed fine-tuned SOTA 89.0. COPA SOTA 94.8%; WiC SOTA 76.1%. A 2.8-point aggregate improvement over BERT-Large coexists with a 17.2-point gap to SOTA and near-chance word-sense comparison. Figure 3.8 uses development scores and must remain separate. e-supergluee-context-scaling |
| Reading caution | Source description The filter failed on long documents, leaving benchmark overlap. Clean-subset evaluation cannot isolate memorization if filtering also changes difficulty. PIQA and Winograd remain flagged; near-total overlap led to omission of several language-model benchmarks and the Children's Book Test. e-contaminatione-contamination-method |
Core contributions
- Source description
An eight-model scaling study separates zero-, one- and few-shot evaluation, spanning 125M to 175B parameters. It extends both language-model loss measurements and downstream evaluation without fine-tuning GPT-3. e-architecturee-scaling
- Reader analysis
The evidence combines standard NLP benchmarks, synthetic arithmetic and word-manipulation probes, human assessment of generated news, and a post-hoc contamination audit. Together these expose capabilities and failure modes that an aggregate score hides. e-arithmetice-wordse-newse-contamination
Figure 2.1. Demonstrations change the context supplied to GPT-3 while its weights stay fixed. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the left column from top to bottom. Every panel ends with the same unfinished query, but the text before it changes: an instruction alone, one translated example, then several translated examples. The arrows on these panels label parts of a prompt; they do not represent parameter updates. The right column explicitly inserts yellow gradient-update blocks between examples and labels that procedure as unused for GPT-3's downstream evaluation. This matches Section 2's distinction between conditioning and fine-tuning. The drawing explains task delivery, not the internal transformer layers; those are specified in Section 2.1 and Table 2.1. e-settingse-architecturee-evaluatione-limitations
What it supports. The experiment tests whether one pretrained model can interpret a task through text and supply the missing completion. Increasing K adds demonstrations within a forward-pass context. It does not train a fresh downstream model. This distinction lets the paper measure task-specific example efficiency while reusing the same pretrained parameters.
Where the evidence stops. “No gradient updates” applies to task evaluation; pretraining still updates weights on a large corpus. The diagram neither proves a new rule is learned from scratch nor provides a controlled comparison against fine-tuning the same GPT-3 model.
2. Motivation
2.1 The problem and the proposed response
Task-specific fine-tuning needs labeled datasets and can exploit narrow-distribution correlations. The paper asks how far one pretrained language model can go when a new task is communicated through instructions and a few demonstrations. Sample efficiency here concerns downstream task specification, not the volume of pretraining data. e-probleme-settingse-limitations
2.2 What this reading follows
GPT-3 turns a pretrained language model into a task interface: write an instruction, show examples, and leave a query unfinished. The same model then produces or scores a continuation. The paper's contribution is a systematic test of how this behavior changes with scale, rather than a new task-specific training stage. Its strongest results make the approach useful, but the details are instructive: a prompt can teach an answer format, factual QA varies by dataset, and a respectable benchmark average can conceal chance-level performance. This reading follows the supplied July 2020 arXiv v4 through six original visuals, with training cost and contamination kept in view. e-identitye-settingse-architecturee-lambadae-qae-supergluee-computee-contamination
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Foundational work |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
This table preserves the labels recorded at reading time. The current major category is Components of WAMs. View the current classification.
3.1 Evidence-based assessment
Supports the recorded classification
The recorded foundational-language-backbone category is supported. GPT-3 predicts text and ranks textual answers; it does not jointly predict embodied futures and actions, implement inverse dynamics, or execute a control loop. Thus the WAM architecture, prediction-paradigm and quadrant labels are not applicable. A single language model serving many tasks does not establish the WAM taxonomy's “One Model” mechanism. e-settingse-architecturee-evaluatione-limitations
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Separate what is learned in weights from what is supplied in context
Start with pretraining. GPT-3 repeatedly predicts text using a corpus mixture whose quality filtering and sampling weights affect what knowledge and patterns can enter its parameters. Appendix B specifies the optimization and packing procedure; this is where gradient updates occur. At evaluation, an instruction and demonstrations are placed before a fresh query, and the transformer computes a continuation with its weights fixed. The paper calls this inner process in-context learning while deliberately leaving open whether it creates a new skill or recognizes an existing one. Reader interpretation: the reusable interface is the practical advance. It reduces the labeled examples needed to specify a downstream task, but does not erase the pretraining data bill or establish that the resulting system learns as a human does. e-datae-traininge-settingse-probleme-limitations
5.2 Treat prompt format and answer scoring as parts of the method
A task is not fully specified by naming a dataset. LAMBADA's ordinary continuation gives probability to many plausible endings; demonstrations in the fill-in-the-blank layout communicate that the output must be one word. That helps explain why its few-shot improvement cannot be attributed solely to additional examples. Multiple-choice tasks use another route: score supplied candidate completions, usually with length normalization and sometimes with the answer-context likelihood ratio given in Section 2.4. Free-form benchmark completion instead uses beam search, followed by the dataset's metric. Reader interpretation: the model is shared, but the evaluation interface contains task-specific choices. A reproduction must preserve those choices and their development-set selection before interpreting score differences as evidence about model capacity. e-lambadae-promptse-evaluation
5.3 Read capability gains alongside the evidence that could confound them
The SuperGLUE context sweep helps establish that demonstrations matter for a fixed pretrained model, whereas the test table shows how unevenly that benefit transfers across tasks. The contamination audit asks a different question: could the evaluation examples already have entered pretraining? Appendix C defines an overlap rule and compares clean-only scores with all-example scores. This comparison is informative, but clean examples need not have the same difficulty as removed examples; some matches contain only background passages. Reader interpretation: near-zero score changes weaken a simple memorization account without proving uncontaminated generalization. Likewise, strong textual commonsense or fluent generated news does not demonstrate grounded physical interaction. The paper itself identifies lack of such grounding and uncertainty about new-task learning as limits of the approach. e-supergluee-context-scalinge-contaminatione-contamination-methode-limitationse-news
5.4 Training and inference
During training
All eight models train on 300B tokens. Adam uses beta1=0.9, beta2=0.95 and epsilon=10^-8, gradient-norm clipping at 1.0 and weight decay 0.1. The largest model's peak learning rate is 6×10^-5 and final batch size 3.2M tokens. Warmup spans 375M tokens; cosine decay reaches 10% of peak over 260B tokens. e-architecturee-training
Documents are packed into full-length sequences separated by end-of-text tokens, without special inter-document masking. Training uses V100 GPUs with both intra-matrix and inter-layer model parallelism; GPU count is not specified. e-traininge-data
During inference
Demonstrations are randomly sampled per evaluation query, with documented dataset exceptions. When separate development and test sets exist, K is selected on development data. Multiple-choice evaluation usually length-normalizes likelihood; ARC, OpenBookQA and RACE instead benefit from answer-context normalization. Free-form benchmark completion uses beam width 4 and length penalty 0.6. e-evaluation
Zero-shot means no demonstrations; one-shot means one. Neither means absence of task-related knowledge in pretraining. LAMBADA additionally changes from ordinary continuation to a demonstrated fill-in-the-blank format, so its few-shot gain includes better task specification. e-settingse-lambadae-limitations
5.5 Implementation flow
- Build a weighted text corpus
Quality-filter Common Crawl using a classifier trained against curated text, apply fuzzy document deduplication, and mix it with WebText2, Books1, Books2 and Wikipedia. Curated sources are oversampled relative to their size; the training distribution is therefore not simply the raw web. e-data
- Scale one autoregressive transformer
GPT-3 inherits GPT-2's pre-normalization, initialization and reversible tokenization, with alternating dense and locally banded sparse attention. The largest model has 96 layers, model width 12,288, 96 attention heads and a 2,048-token context. Model parallelism partitions width and depth. e-architecture
- Express a task as continuation
Place complete demonstrations before an unfinished query. K denotes the number of demonstrations, usually constrained to 10–100 by the context window. The model's activations condition on this sequence; weights remain fixed. There is no separate task-specific optimizer or action-execution loop. e-settingse-evaluation
6. Experiments & results
GPT-3 tests whether scaling autoregressive text pretraining makes a fixed model more useful when tasks are specified by examples in its context. The 175-billion-parameter model achieves strong completion and factual-QA results without downstream gradient updates, but gains vary sharply by task. Prompt format, evaluation split, pretraining exposure and substantial compute are central to interpreting the evidence.
6.1 Read the original evidence
Figure 3.2. LAMBADA shows both a large few-shot gain and sensitivity to task formatting. Original paper, p. 12 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow each color across the model sizes, then compare the three endpoints at 175B. The orange line uses K=15 demonstrations. Its uneven behavior for small models matters: adding demonstrations does not produce a uniform benefit across the series. The adjacent Table 3.2 supplies exact endpoint values, while the text explains a change hidden by the simple legend. Zero-shot uses ordinary continuation, whereas one- and few-shot examples frame the target as a blank requiring one word. Figure G.21 supplies that prompt layout. Thus both model capacity and the ability to infer the requested output format contribute to the observed behavior. e-lambadae-evaluatione-promptse-contaminatione-contamination-table
What it supports. Table 3.2 reports 86.4% few-shot accuracy, versus 76.2% zero-shot and 72.5% one-shot. Against the listed 68.0% prior accuracy, the few-shot improvement is 18.4 percentage points. The strong result supports useful contextual adaptation, while the one-shot decline shows that more context alone is not sufficient.
Where the evidence stops. The zero-/few-shot comparison changes format as well as demonstration count. LAMBADA also has genuine pretraining overlap; Table C.1's 86.0% clean score is reassuring but does not eliminate distribution-shift or memorization concerns.
Table 3.3. Closed-book QA improves unevenly: TriviaQA is the standout result. Original paper, p. 13 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the setting names in the first column. RAG is a fine-tuned retrieval system; the two T5 rows are fine-tuned closed-book systems; GPT-3 is closed-book without downstream gradient updates. Compare one dataset column at a time. NaturalQS and WebQS are the paper's short labels for Natural Questions and WebQuestions. The few-shot setup uses K=64. Table H.1 identifies their test splits and distinguishes TriviaQA's development evaluations from the wiki-split test-server result in this table's caption. The header's broad QA terminology does not mean GPT-3 retrieves supporting passages. Figure G.34 also shows that TriviaQA accepts multiple answer variants. e-qae-hparamse-prompts
What it supports. GPT-3 few-shot scores 71.2 on TriviaQA, exceeding the listed 60.5 for T5-11B+SSM and 68.0 for RAG. Its 29.9 Natural Questions and 41.5 WebQuestions scores remain below T5-11B+SSM's 36.6 and 44.7. The evidence supports a dataset-dependent capability, not uniform factual-QA superiority.
Where the evidence stops. The rows differ in model size, pretraining, downstream supervision and access to retrieval. TriviaQA's zero/one-shot development scores and few-shot test-server score are not a paired same-split improvement estimate, even though Table H.1 also reports 71.2 few-shot on development.
Table 3.8. A competitive aggregate hides a sharp failure on word-sense comparison. Original paper, p. 19 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the aggregate score at the upper left, then move across both halves of the table to inspect the task profile. GPT-3 uses 32 demonstrations per task and no task-specific weight updates. All values here are test results. The upper and lower blocks continue the same three model rows rather than describing different experimental runs. Pay attention to metric labels: CB and ReCoRD each have separate accuracy and F1 columns, and MultiRC has two metrics as well. COPA asks about plausible causal alternatives, while WiC compares a word's meaning across sentences. Their different outcomes make a single headline score incomplete. e-supergluee-prompts
What it supports. GPT-3's aggregate score is 71.8, ahead of BERT-Large's 69.0 but below the listed SOTA of 89.0. COPA reaches 92.0% versus 94.8% SOTA, while WiC is 49.4%, described as chance-level in the text. Success at some tasks therefore coexists with a major sentence-comparison weakness.
Where the evidence stops. The SOTA and BERT rows are fine-tuned baselines with different resources. This table does not show GPT-3 fine-tuning. Component scores also use different metrics, so their raw numerical gaps should not be treated as one common accuracy scale.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| LAMBADA last-word prediction Test set; 175B model; 15 development-set demonstrations for few-shot; fill-in-the-blank format. | Few-shot: 86.4% and 1.92; zero-shot: 76.2% and 3.00; one-shot: 72.5% and 3.35. Accuracy (%) and perplexity | Prior reported accuracy: 68.0%. The few-shot difference is 18.4 percentage points. Strong contextual adaptation, but one example hurts and the prompt format changes. Genuine pretraining overlap remains; the contamination audit is reassuring rather than definitive. e-evaluatione-lambadae-contaminatione-contamination-table |
| Closed-book factual question answering 175B; K=64. TriviaQA few-shot uses the wiki-split test server; its zero/one-shot scores use development data. Natural Questions and WebQuestions use test sets. | Few-shot: TriviaQA 71.2, Natural Questions 29.9, WebQuestions 41.5. Reported answer accuracy (%) | Fine-tuned closed-book T5-11B+SSM: 60.5, 36.6 and 44.7 respectively. Retrieval-based RAG: 68.0, 44.5 and 45.5. GPT-3 leads the listed TriviaQA comparisons but trails on the other two. RAG has external retrieval; these are different resource regimes, not a controlled architecture comparison. e-qae-hparams |
| SuperGLUE aggregate and task variation Official test results; 175B; 32 demonstrations per task; no gradient updates. | Aggregate 71.8; COPA 92.0%; WiC 49.4%. SuperGLUE score; component accuracy (%) | Fine-tuned BERT-Large aggregate 69.0; listed fine-tuned SOTA 89.0. COPA SOTA 94.8%; WiC SOTA 76.1%. A 2.8-point aggregate improvement over BERT-Large coexists with a 17.2-point gap to SOTA and near-chance word-sense comparison. Figure 3.8 uses development scores and must remain separate. e-supergluee-context-scaling |
| Few-shot English–Romanian translation WMT16 test; 175B; 64 paired demonstrations in context. | Romanian→English 39.5; English→Romanian 21.0. BLEU from multi-bleu.perl with XLM tokenization | MASS: 33.1 and 35.2; listed supervised SOTA: 39.9 and 38.5. Direction matters. Paired demonstrations prevent strict equivalence to unsupervised NMT, and these scores must not be mixed with Appendix H's SacreBLEU values. e-translation |
| Synthetic arithmetic 2,000 randomly generated instances per operation; 175B; K=50 few-shot; exact answers. | Two-digit addition 100.0; three-digit addition 80.4; five-digit addition 9.3; two-digit multiplication 29.2. Accuracy (%) | Zero-shot: 76.9, 34.2, 0.7 and 19.8 respectively. Success degrades with computational difficulty. Table 3.9 and H.1 report 80.4 for three-digit addition, while p. 22 prose says 80.2; the table value is retained with this discrepancy disclosed. e-arithmetice-hparams |
| Human detection of short synthetic news Approximately 200-word articles; real titles/subtitles and three conditioning articles; US-based participants; automatic length-based output selection. | GPT-3 175B: 52%, 95% CI 49–54%. Mean correct human/machine assignments among non-neutral judgments | Deliberately poor control: 86%, 95% CI 83–90%; chance is 50%. This measures distinguishability, not factual correctness. Five candidates were generated and the closest-length one selected; it is not a single unselected sample per article. e-newse-news-protocol |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 3.8. Varying model size and context separately reveals where adaptation gains arise. Original paper, p. 20 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the panels as two different sweeps. On the left, the horizontal axis is model size, and the three colors are zero-, one- and 32-shot settings. On the right, the model stays at 175B and the horizontal axis becomes K, the number of examples in context. The first demonstration gives a conspicuous jump; later examples give smaller additional gains. These are changes in inference-time conditioning, not training epochs. The figure caption is essential: GPT-3's curves use development data, while the dashed external reference lines are not directly comparable. Table 3.8, rather than a curve crossing, supplies the official test comparison. e-context-scalinge-hparamse-architecturee-computee-evaluation
What it supports. The figure supports two observations: larger pretrained models generally score better, and the largest model can use extra demonstrations to improve its development score. Table H.1 reports 73.2 for its 32-shot development aggregate, distinct from the 71.8 official test score. The visible small-model irregularities caution against describing every increase as monotonic.
Where the evidence stops. This is a scaling and context diagnostic, not an isolated attention-module ablation. Model size changes compute and architectural dimensions together. The right panel does not control demonstration correctness or formatting, and the paper supplies no uncertainty bands for these curves.
Figure 4.2. Overlap screening tests robustness of benchmark scores but cannot fully isolate memorization. Original paper, p. 32 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Each dot summarizes a benchmark after filtering examples with detected overlap against the available pretraining corpus. The horizontal axis gives the percentage retained as clean under that rule, not the percentage of all possible contamination conclusively ruled out. The vertical axis is a relative percent change in its own task metric. The green upward arrow means clean-only evaluation scored better; the red downward arrow means all-data evaluation scored better. Those directions agree with Appendix C and Table C.1. Most dots sit near zero, while some extreme points arise from small or distribution-shifted clean subsets. The table is needed to inspect counts and exact scores. e-contaminatione-contamination-methode-contamination-table
What it supports. PIQA's development score falls from 82.3 to 79.3 on its clean subset: 3.0 percentage points, reported as roughly a 4% relative decline. LAMBADA changes from 86.4 to 86.0. These checks limit a simple contamination explanation for many gains, while leaving the flagged benchmarks and alternative explanations unresolved.
Where the evidence stops. An overlap can be background prose rather than a leaked answer, and cleaning can change difficulty. Table C.1 uses a different demonstration seed. Its Winograd clean proportion also differs from Section 4's prose; the source provides no reconciliation.
7. Analysis & limitations
7.1 What the evidence leaves open
The filter failed on long documents, leaving benchmark overlap. Clean-subset evaluation cannot isolate memorization if filtering also changes difficulty. PIQA and Winograd remain flagged; near-total overlap led to omission of several language-model benchmarks and the Children's Book Test. e-contaminatione-contamination-method
Weaknesses include sentence comparison, long-form coherence and physical commonsense. CoQA reaches 85.0 F1 while QuAC reaches 44.3 F1 on development data. No bidirectional-architecture or denoising-objective ablation establishes why these gaps occur. e-comprehensione-limitations
The paper leaves unresolved whether contextual improvement learns a new rule or recognizes a pretrained task. It also reports expensive inference, poor pretraining sample efficiency and missing grounding in physical interaction. e-limitations
Preliminary gender, race and religion probes expose stereotyped associations; their prompts and co-occurrence metrics are limited measures of harm. The authors also discuss misuse and energy costs, without presenting a comprehensive mitigation evaluation. e-impact
7.2 Questions for discussion
- How much of LAMBADA's gain survives when task format is held fixed across demonstration counts?
- Would correct, incorrect and unrelated demonstrations separate task recognition from learning a new mapping?
- Can difficulty-matched contamination analysis distinguish memorization from selection effects?
8. Reproducibility audit
8.1 Requirements and known gaps
Rebuilding training needs the original corpus mixture, tokenizer, filtering, deduplication and parallel implementation. Appendix A specifies logistic regression, HashingTF, Pareto selection with alpha=9 and MinHashLSH with 10 hashes, but omits enough corpus and configuration detail to prevent an exact reconstruction from the PDF alone. e-data
Appendix D estimates 3.14×10^23 training FLOPs, or 3.64×10^3 PF-days, for GPT-3 175B, excluding attention operations. This is an approximate calculation, not measured runtime. A full reproduction also needs undisclosed GPU counts, software versions and random seeds. e-computee-traininge-data
A smaller, meaningful replication should freeze weights, preserve Appendix G's prompt layouts and the named splits, and repeat demonstration draws. Test whether correct demonstrations outperform equally long corrupted demonstrations; separately audit overlap while matching clean and dirty examples for difficulty. e-evaluatione-promptse-contamination-methode-contamination-table
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Does a correct demonstration mapping matter beyond format?
Reader-proposed check, not performed: evaluate the symbol-insertion task on disjoint held-out words using one frozen checkpoint. Sweep K over 0, 1, 4, 16 and 32 and compare correct input/output pairs, randomly permuted output pairs, and unrelated examples matched approximately in token length. Use the same instruction, query set and decoding rule; verify every prompt fits the context window. Repeat across multiple demonstration draws and report paired accuracy differences with uncertainty. A gain unique to correct pairs supports use of the demonstrated mapping. Similar gains from corrupted pairs would weaken that explanation and point toward formatting or task recognition. e-wordse-promptse-evaluatione-limitations
Check 2: Does an overlap advantage survive matching for difficulty?
Reader-proposed check, not performed: given the exact checkpoint and indexed pretraining corpus, reproduce Appendix C's PIQA overlap labels, then compare clean and dirty examples matched by text length, answer length and a fixed smaller model's score. Manually distinguish answer-bearing matches from background-only matches. Keep demonstration samples and choice scoring fixed across comparisons, bootstrap matched pairs, and report absolute percentage-point gaps separately from relative percent changes. A persistent gap concentrated in answer-bearing overlap would strengthen the memorization explanation; a disappearing gap after matching would support selection effects. Report corpus-access gaps explicitly rather than calling a different corpus a full replication. e-contaminatione-contamination-methode-contamination-tablee-evaluation
8.3 Reading coverage
Visual audit: The title/author page, architecture and data tables, evaluation rules, results, scaling diagnostics, contamination analysis, limitations, impact discussion, training/compute appendices and cited prompt examples were visually inspected. Six final original crops were individually viewed, with legends, axes and table headers retained. Figure 2.1's update arrows were checked against Section 2; Figure 4.2's positive/negative directions and relative-percent convention were checked against Appendix C and Table C.1. Figure 3.8 development scores remain separate from Table 3.8 test scores. Relevant source inconsistencies include arithmetic prose versus tables, PIQA zero-shot values, Winograd overlap percentages, CoQA's mislabeled plot axis, and news-study prose/table counts. All 21 text chunks were read, including Appendices A–H; visual coverage is limited to the listed pages, and no separate supplement or code was inspected.
PDF pages inspected for this edition: 1, 3, 4, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 31, 32, 33, 34, 35, 36, 37, 38, 39, 43, 44, 45, 46, 47, 49, 52, 56, 57, 59, 63. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract and contents (pp. 1–2)
- 1 Introduction (pp. 3–6)
- 2 Approach: settings, architecture, data, training and evaluation (pp. 6–10)
- 3 Results: language modeling, completion, QA, translation, Winograd, commonsense, comprehension, SuperGLUE, NLI, arithmetic, word manipulation and qualitative generation (pp. 10–30)
- 4 Benchmark memorization and contamination (pp. 29–33)
- 5 Limitations (pp. 33–34)
- 6 Broader Impacts (pp. 34–39)
- 7 Related Work; 8 Conclusion; acknowledgements and contributions (pp. 39–42)
- Appendices A–D: filtering, training, contamination and compute (pp. 43–46)
- Appendices E–F: human assessment and additional samples (pp. 46–49)
- Appendix G: all task phrasings (pp. 50–62)
- Appendix H: complete results table and extracted figure captions (pp. 63–67)
- References (pp. 68–75)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- Identity/version: the inspected title page is arXiv:2005.14165v4 [cs.CL], 22 July 2020. The catalog names the NeurIPS 2020 publication; this report concerns the supplied 75-page arXiv revision. The venue edition was not supplied or compared.
- All 31 catalog authors correspond to the title-page list. Observed forms include Tom B. Brown, Jared Kaplan, Daniel M. Ziegler and Christopher Hesse; the catalog uses Tom Brown, Jared D Kaplan, Daniel Ziegler and Chris Hesse. Metadata preserves the observed forms.
- All 21 supplied text chunks were read individually. Visual inspection covered the pages declared in the illustrated edition; remaining images, including Appendix H's plots on pp. 64–67, were not visually inspected.
- External code, checkpoints, datasets, linked supplements and referenced works were not inspected; no experiments were reproduced.
- The primary source itself omits the two conditioning poems in Figure F.1. Their text has not been reconstructed. The language inventory is deferred to a separate supplement, and Books1/Books2 are not identified sufficiently to reconstruct those corpora.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e-identityPDF p. 1, title, author block, affiliation footnote and arXiv margin
The exact title and 31 authors identify arXiv:2005.14165v4, 22 July 2020. OpenAI is the main affiliation; Jared Kaplan's footnote adds Johns Hopkins University.
Go to primary source ↓e-problemPDF pp. 3–4, Section 1 and Figure 1.1
The motivation concerns task-specific supervision and narrow-distribution fine-tuning. The diagram distinguishes weight learning across pretraining from adaptation within a sequence.
Go to primary source ↓e-settingsPDF pp. 6–7, Section 2 and Figure 2.1
Defines fine-tuning, few-shot, one-shot and zero-shot. GPT-3's evaluated settings use forward passes without task-specific weight updates; K counts demonstrations.
Go to primary source ↓e-architecturePDF p. 8, Section 2.1 and Table 2.1, GPT-3 175B row
Eight model sizes share a 2,048-token context and GPT-2-derived architecture with alternating dense and locally banded sparse attention. Table gives 96 layers, width 12,288, 96 heads, batch 3.2M and learning rate 0.6×10^-4.
Go to primary source ↓e-dataPDF pp. 8–9, Sections 2.2–2.3, Table 2.2; p. 43, Appendix A
Describes corpus mixture, oversampling, document deduplication, classifier filtering and V100 model-parallel training. Appendix A specifies logistic regression/HashingTF, Pareto alpha=9 and MinHashLSH with 10 hashes. Full book-corpus identities and exact system configuration are absent.
Go to primary source ↓e-trainingPDF p. 43, Appendix B; p. 8, Table 2.1 caption
All models train for 300B tokens. Appendix B gives Adam settings, clipping, weight decay, warmup/decay, batch ramp and document packing without special masking between documents.
Go to primary source ↓e-evaluationPDF p. 10, Section 2.4
Specifies demonstration sampling and dataset exceptions, development selection of K, likelihood normalization, answer-context ratio, beam width 4, length penalty 0.6, and test/development reporting. Its reference to “200B” conflicts with the 175B model specification.
Go to primary source ↓e-scalingPDF p. 11, Figure 3.1 and Section 3.1.1; p. 31, Figure 4.1
Validation loss approximately follows the stated compute-scaling trend; train/validation curves show a persistent gap. These are language-model losses, not downstream or embodied success measures.
Go to primary source ↓e-lambadaPDF p. 12, Table 3.2, Figure 3.2 and Section 3.1.2; p. 13, contamination note
Table gives 76.2/72.5/86.4 accuracy and 3.00/3.35/1.92 perplexity for zero/one/few-shot, with prior accuracy 68.0. Figure specifies K=15; text explains changed cloze formatting and notes contamination.
Go to primary source ↓e-qaPDF pp. 13–14, Section 3.2, Table 3.3 and Figure 3.3
Table lists GPT-3 and T5/RAG results for Natural Questions, WebQuestions and TriviaQA, and identifies the TriviaQA wiki test-server result. GPT-3 has no retrieval or QA fine-tuning.
Go to primary source ↓e-translationPDF pp. 14–16, Section 3.3, Table 3.4 and Figure 3.4
Specifies paired demonstrations, 93% English pretraining by word count, direction-dependent results and multi-bleu.perl/XLM tokenization. Romanian→English scores 39.5 versus MASS 33.1; reverse direction 21.0 versus 35.2. Language inventory is deferred to a supplement.
Go to primary source ↓e-reasoningPDF pp. 16–18, Sections 3.4–3.5, Tables 3.5–3.6
Winograd and PIQA carry contamination asterisks. Commonsense-task gains are mixed. PIQA zero-shot is 80.5 in Table 3.6 but 81.0 in the prose; these are not silently reconciled.
Go to primary source ↓e-comprehensionPDF p. 18, Section 3.6 and Table 3.7; p. 19, Figure 3.7; p. 63, Table H.1, CoQA/QuAC rows
Few-shot CoQA is 85.0 F1 and QuAC 44.3 F1, both on development data. RACE uses accuracy rather than F1. Figure 3.7 on p. 19 labels its axis Accuracy although the caption and tables specify CoQA F1.
Go to primary source ↓e-supergluePDF pp. 18–20, Section 3.7 and Table 3.8
The official test table reports 71.8 aggregate for GPT-3, 69.0 for BERT-Large and 89.0 SOTA, using 32 examples per task. COPA is 92.0 and WiC 49.4. WSC/MultiRC reuse a fixed demonstration sample.
Go to primary source ↓e-context-scalingPDF p. 20, Figure 3.8 and caption
Left panel varies model size and setting; right varies K for 175B. GPT-3 curves are development scores and the dotted baseline references are not directly comparable. More examples cease to fit reliably beyond K=32.
Go to primary source ↓e-arithmeticPDF pp. 21–23, Section 3.9.1, Figure 3.10 and Table 3.9
Defines uniformly sampled integer tasks with 2,000 examples each and exact scoring. Table reports the retained arithmetic values. The three-digit-addition prose gives 80.2 versus table 80.4. Limited exact-string searches find few arithmetic overlaps.
Go to primary source ↓e-wordsPDF p. 4, Figure 1.2; pp. 23–24, Section 3.9.2, Table 3.10 and Figure 3.11
Word tasks use frequent words distorted by character operations. Random-insertion accuracy improves with model size, demonstrations and an instruction. Table 3.10 differs slightly from prose for several tasks; the mechanism of new-task learning remains uncertain.
Go to primary source ↓e-newsPDF pp. 25–26, Section 3.9.4 and Table 3.11, control and GPT-3 175B rows
Short-news detection accuracy is 52% (49–54% CI) for 175B versus 86% (83–90%) for the poor control, excluding neutral judgments. Conditioning uses three news articles; factual inaccuracies remain.
Go to primary source ↓e-news-protocolPDF pp. 46–47, Appendix E, procedure/design and Table E.1
Five outputs per title were generated and the closest in length selected. Quizzes mix human and generated articles. The 175B row lists 80 recruited and 9 excluded. Prose model/experiment counts and some table totals are inconsistent.
Go to primary source ↓e-contaminationPDF pp. 31–33, Section 4 and Figure 4.2
Describes failed overlap removal, clean-subset diagnostics, false positives, PIQA/Winograd flags and omitted benchmarks. LAMBADA has genuine overlap but a small clean-score difference. Distribution shift prevents a definitive memorization estimate.
Go to primary source ↓e-contamination-methodPDF pp. 43–44, Appendix C, filtering and overlap methodology
Initial removal uses normalized 13-grams and surrounding text; analysis uses a dataset-dependent N, generally 8–13 outside synthetic tasks. Comparisons use the full available corpus, including material not necessarily sampled during training.
Go to primary source ↓e-contamination-tablePDF p. 45, Table C.1, LAMBADA/PIQA/Winograd rows and caption
LAMBADA scores 86.4 overall and 86.0 clean with 57% clean; PIQA development scores 82.3 and 79.3 with 71% clean. Winograd has 40% clean here, unlike the p. 32 prose's 45% flagged. Caption defines relative changes and discloses a different demonstration seed.
Go to primary source ↓e-limitationsPDF pp. 33–34, Section 5
Acknowledges unreliable coherence/physics, sentence-comparison failures, untested bidirectional objectives, ambiguity between new learning and recognition, costly inference, pretraining inefficiency, and lack of video/physical grounding.
Go to primary source ↓e-impactPDF pp. 35–39, Sections 6.1–6.3, Tables 6.1–6.2 and Figure 6.1
Discusses misuse, preliminary gender/race/religion biases, prompt/co-occurrence limitations and compute/energy concerns. The authors caution that this is not exhaustive bias characterization.
Go to primary source ↓e-computePDF p. 46, Appendix D and Table D.1, GPT-3 175B row
Approximate compute is 3.14×10^23 FLOPs or 3.64×10^3 PF-days; calculations omit attention operations and are not runtime measurements.
Go to primary source ↓e-promptsPDF p. 52, Figures G.4–G.5; p. 56, Figure G.21; p. 57, Figure G.26; p. 59, Figures G.32 and G.34
Shows PIQA physical-commonsense choices, COPA causal alternatives, LAMBADA fill-in-the-blank formatting, symbol-insertion instruction, WiC comparison phrasing and TriviaQA's multiple accepted answer variants.
Go to primary source ↓e-hparamsPDF p. 63, Table H.1, LAMBADA/QA/arithmetic/SuperGLUE rows and split/K columns
Confirms LAMBADA K=15, QA K=64, arithmetic K=50, QA split labels and the distinction between SuperGLUE development 73.2 and test-server 71.8.
Go to primary source ↓e-omitted-poemsPDF p. 49, Figure F.1, conditioning block
The two conditioning poems are explicitly replaced by “[Poem text omitted]” in the source itself.
Go to primary source ↓8.5 Primary sources
Language Models are Few-Shot Learners ↗
PDF · 38,669 extracted words
Source fingerprint
97fd272f1fdfc18677462d0292f5fbf26ca86b4d1b485c2dba03269b643a0e83