LLaMA: Open and Efficient Foundation Language Models
1. Paper overview
In one sentence: LLaMA spends more training tokens on compact causal language models to improve capability at a chosen inference size, while leaving the contribution of individual architectural choices unresolved. e01e02e03e04e08e14e20
| At a glance | What to know |
|---|---|
| Research problem | Source description Training-compute optimality can favor a model that is expensive to serve. LLaMA instead targets performance at different inference budgets, spending more pretraining tokens on relatively small networks. Its evidence is benchmark quality versus parameter count; deployment-wide cost savings remain an author motivation rather than a measured serving study. e02e04e08 |
| Core mechanism | |
| A key reported result | Zero-shot common-sense reasoning: LLaMA-13B: HellaSwag 79.2, BoolQ 78.1; LLaMA-65B: HellaSwag 84.2, BoolQ 85.3. Accuracy (%). Eight benchmarks in Table 3; zero-shot multiple-choice evaluation. Benchmark-specific split names are not supplied here. GPT-3-175B: 78.9 and 60.5; Chinchilla-70B: 80.8 and 83.7, respectively. Supports competitive smaller models. Section 3.1 says BoolQ is an exception against Chinchilla, but Table 3 shows 85.3 > 83.7. This report preserves the table values and flags the conflict. e08 |
| Reading caution | Reader analysis No controlled RMSNorm, SwiGLU, RoPE or data-mixture ablation establishes each component's contribution. The explanation that fewer books cause lower MMLU performance is the authors' hypothesis, not an isolated experiment; comparisons also use baseline numbers from other papers. e04e08e13e14 |
Core contributions
- Reader analysis
Tracking checkpoints exposes task-dependent scaling: most monitored benchmarks improve, while SIQA fluctuates and the two largest models track closely on WinoGrande. These diagnostics do not isolate any architectural component. e13
Table 1. The training recipe is a weighted mixture, with different reuse rates across sources. Original paper, p. 2 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with Sampling prop., which describes how frequently each source enters training, rather than its fraction of stored bytes. CommonCrawl contributes 67% and C4 15%; the remaining rows add code, encyclopedic text, books, scientific papers and question-answer discussions. Then read Epochs alongside those proportions. The caption specifies that epoch counts describe the 1.4T-token runs, while the 1T-token runs retain the same sampling proportions. Wikipedia and books are revisited more than twice, whereas GitHub remains below one epoch. Finally, use Disk size only as a corpus-size descriptor: it is not a token count or a measure of downstream usefulness. e03
What it supports. Most sampled tokens come from web corpora, while smaller specialized collections provide additional domains. Wikipedia's 2.45 epochs and books' 2.23 epochs make clear that equal sampling shares need not imply equal reuse. The table makes the broad training composition inspectable, but does not identify which source caused a particular benchmark gain.
Where the evidence stops. Section 2.1 gives selected filtering procedures and thresholds, not a complete immutable corpus specification. Sampling weights cannot substitute for the exact filtered documents, tokenizer and deduplication decisions needed to reproduce the mixture.
2. Motivation
2.1 The problem and the proposed response
Training-compute optimality can favor a model that is expensive to serve. LLaMA instead targets performance at different inference budgets, spending more pretraining tokens on relatively small networks. Its evidence is benchmark quality versus parameter count; deployment-wide cost savings remain an author motivation rather than a measured serving study. e02e04e08
2.2 What this reading follows
LLaMA asks what happens when the model that will be served, rather than the one cheapest to train, becomes the design constraint. Its answer combines a broad public-data mixture, familiar transformer refinements and long pretraining runs. Read the evidence in two passes: first trace how data and model size determine the training experiment, then examine which benchmarks actually improve. The paper is persuasive about competitive language capability at modest parameter counts, but it does not establish universal superiority, a measured deployment cost advantage or an action-executing world model. This edition follows the verified February 2023 v1 PDF and preserves its numerical inconsistencies. e01e02e03e04e08e14e20
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Foundational work |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
This table preserves the labels recorded at reading time. The current major category is Components of WAMs. View the current classification.
3.1 Evidence-based assessment
Supports the recorded classification
The recorded foundational-language-backbone classification is supported. Causal token prediction supplies language capability, but the paper specifies neither joint future/action prediction nor inverse dynamics or executed control. Its single language network does not establish the world-action taxonomy's One Model category; the recorded Not applicable fields are appropriate. e04e07e20
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 1. Turn public text into a particular training distribution
The data pipeline is part of LLaMA's method, not just background material. Section 2.1 describes separate preprocessing choices for web pages, code, books and scientific documents before the tokenizer turns them into a common prediction stream. CommonCrawl goes through CCNet and an additional classifier based on Wikipedia reference pages; GitHub receives license selection, heuristics and exact file deduplication; books use a content-overlap threshold. Table 1 then controls how often each processed collection contributes tokens. This distinction matters for reproduction: locating the named datasets does not recreate the training distribution. SentencePiece BPE further changes the representation by splitting numbers into individual digits and using byte fallback. Table 2 finally sets how much of this mixture each model consumes, separating a named corpus from the actual exposure used in a run. e03e04
Table 2. Model size and training exposure change together across the family. Original paper, p. 3 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each row as a complete training configuration. The parameter counts are more precise than the shorthand names used elsewhere: 6.7B, 13.0B, 32.5B and 65.2B. Dimension, number of heads and number of layers grow together, so moving down the table changes several aspects of capacity. The final columns are equally important: every row uses a 4M-token batch, but the two largest models receive 1.4T tokens and a lower peak learning rate. Section 2.2 supplies the shared architecture omitted from this table: RMSNorm before sublayers, SwiGLU activations and rotary positional embeddings at each layer. e04e05
What it supports. The largest row combines dimension 8192, 64 attention heads and 80 layers, while the smallest uses 4096, 32 and 32. Their final scores therefore compare complete model-and-training configurations. The table helps reconstruct these settings, but it cannot assign the observed gains separately to depth, width or training duration.
Where the evidence stops. This is the source's architecture specification table, not a computational graph. No diagram or controlled removal study isolates RMSNorm, SwiGLU or RoPE; their causal contribution remains unmeasured in this paper.
5.2 2. Separate predicting text from choosing an answer
LLaMA's shared computation is causal language modeling. The transformer sees preceding tokens, with pre-normalized sublayers, SwiGLU and rotary positions; efficient attention omits scores that the causal mask would exclude. Task behavior then depends on how that predictor is queried. A free-form question may be answered by generation, while multiple-choice evaluation scores candidate continuations. Most such tasks use character normalization, but BoolQ and OpenBookQA use the paper's likelihood ratio against the literal Answer: context. Appendix A adds another layer for QA: a fixed prefix, greedy decoding, stopping rules and text normalization determine exact match. Thus the evaluator is part of the reported result. Few-shot demonstrations enter the prompt, whereas LLaMA-I requires a separate instruction-finetuning stage. None of these mechanisms specifies an environment action controller. e04e06e07e18e14e20
5.3 3. Test the inference-budget argument against its evidence
The reader's interpretation is that LLaMA makes a strong capability-per-parameter case, while leaving a complete deployment-economics case open. Figure 1 shows continuing training-loss reduction and Figure 2 shows broad downstream gains, giving a reason to spend more training on a fixed-size network. Table 3 then demonstrates several competitive scores against much larger models. But Table 9 prevents a universal reading: base 65B still trails Chinchilla and PaLM on MMLU. The instruction experiment changes that comparison, illustrating why training stages must remain explicit. Similarly, the GSM8k advantage over Minerva-62B appears with majority voting, which spends additional inference computation. A useful reproduction should therefore report model size, training exposure, prompting and decoding together; parameter count alone cannot explain either benchmark quality or total cost. e02e04e08e11e13e14
5.4 Training and inference
During training
AdamW uses beta values 0.9 and 0.95, weight decay 0.1, clipping at 1.0 and 2,000 warmup steps. Cosine decay ends at 10% of the peak learning rate; peaks are 0.0003 for 7B/13B and 0.00015 for 33B/65B. e05e04
Memory-efficient causal attention, selective activation recomputation, manual backward functions, model/sequence parallelism and communication overlap enable the large runs. The authors report approximately 380 tokens/second/GPU on 2,048 A100-80GB GPUs for 65B, implying roughly 21 days for 1.4T tokens. e06
LLaMA-I is a separate 65B instruction-finetuned model following Chung et al.'s protocol and dataset. The supplied paper does not enumerate its complete finetuning hyperparameters, update count or frozen-module configuration. e14e19
During inference
Closed-book QA uses greedy generation without retrieved evidence. Appendix A stops answers at the first line break, final dot or comma; exact match lowercases answers and removes articles, punctuation and duplicate whitespace. e09e18
Code evaluation uses temperature 0.1 for pass@1 and 0.8 for larger pass@k metrics. HumanEval is zero-shot and MBPP is three-shot. Majority voting in mathematics generates several solutions per question, adding inference work distinct from changing model parameters. e11e12
5.5 Implementation flow
- Curate and tokenize
CommonCrawl and C4 dominate the mixture; GitHub, Wikipedia, books, arXiv and StackExchange provide other domains. Filtering differs by source: CCNet line deduplication and language/quality filtering, exact GitHub file deduplication, and removal of books with over 90% content overlap. SentencePiece BPE splits numbers into digits and falls back to bytes for unknown UTF-8 characters. e03
- Predict causally
A transformer processes preceding tokens through causal multi-head attention. Each sublayer receives RMSNorm-normalized input; SwiGLU replaces ReLU, and rotary positional embeddings replace absolute embeddings at every layer. This supports next-token language modeling; no observation-to-action branch or environment feedback loop is specified. e04e06e20
- Scale the training configuration
The nominal 7B/13B/33B/65B models have 32/40/60/80 layers and dimensions 4096/5120/6656/8192. The smaller pair sees 1.0T tokens and the larger pair 1.4T, all with 4M-token batches. Consequently, final model comparisons change both size and, across these pairs, training exposure. e04
- Apply task-specific readout
Evaluation either generates a free-form answer or ranks completions. Most multiple-choice scores normalize likelihood by completion character count; BoolQ and OpenBookQA instead use a context-versus-Answer: likelihood ratio. Prompted examples condition inference without constituting the separate instruction-finetuning experiment. e07e14
6. Experiments & results
LLaMA trains causal language models longer to improve capability at a chosen inference size. Public-data mixtures and established transformer refinements yield strong language benchmarks, with uneven gains across tasks. The study concerns text prediction and prompting; it does not introduce a world-action model.
The source has no architecture block diagram and no controlled component-removal ablation. Section 2.2 and Table 2 provide the architecture specification, so Table 2 serves as the method visual. Figures 1–2 provide training diagnostics in the ablation section; they cannot identify the isolated effects of RMSNorm, SwiGLU, RoPE or mixture choices. No diagram or experiment has been reconstructed to fill those gaps. e04e13
6.1 Read the original evidence
Table 3. Parameter efficiency is visible on several tasks, with explicit exceptions. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Choose one benchmark column and compare models vertically before making a family-wide judgment. The upper block contains baselines and the lower block contains LLaMA sizes; dashes indicate missing values rather than failures. For example, HellaSwag places 13B at 79.2 beside GPT-3-175B at 78.9, and 65B at 84.2 beside Chinchilla-70B at 80.8. BoolQ offers another useful check because it is one of the tasks using the special Answer: likelihood normalization described in Section 3. Read the source prose together with the cells: a claim about exceptions must agree with the actual numbers. e07e08
What it supports. LLaMA-13B is competitive with a much larger GPT-3 on HellaSwag and exceeds it on BoolQ, while 65B exceeds Chinchilla in both displayed comparisons. These examples substantiate the smaller-model argument. They do not establish a uniform improvement across all tasks or a controlled comparison at equal training compute.
Where the evidence stops. Section 3.1 calls BoolQ an exception against Chinchilla, yet the table gives LLaMA-65B 85.3 versus 83.7. The crop is unchanged; this edition follows those cells. Baseline values were taken from other papers.
Table 9. MMLU exposes a limit to the base model's across-benchmark competitiveness. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Begin with Average, then move left through the subject groups to see where a headline score comes from. The LLaMA block improves with model size, reaching 63.4 for 65B. That remains below the table's Chinchilla-70B value of 67.5 and PaLM-540B value of 69.3. The 13B model reaches 46.9 versus GPT-3's 43.9, illustrating that the parameter-efficiency story can coexist with a gap at the high end. These are base-model, five-shot results. Instruction-finetuned LLaMA-I appears separately in Table 10; its score must not be silently substituted into this table's comparison. e14e21
What it supports. The base 65B model does not match the strongest reported MMLU comparators, despite strong common-sense results elsewhere. The separate instruction experiment raises its score from 63.4 to 68.9, a gain of 5.5 percentage points. Task choice and training stage therefore materially change the comparison.
Where the evidence stops. The authors propose limited book data as an explanation, without isolating that factor. Appendix Table 16 prints Chinchilla's aggregate as 67.6 rather than Table 9's 67.5; this unresolved discrepancy is preserved.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Zero-shot common-sense reasoning Eight benchmarks in Table 3; zero-shot multiple-choice evaluation. Benchmark-specific split names are not supplied here. | LLaMA-13B: HellaSwag 79.2, BoolQ 78.1; LLaMA-65B: HellaSwag 84.2, BoolQ 85.3. Accuracy (%) | GPT-3-175B: 78.9 and 60.5; Chinchilla-70B: 80.8 and 83.7, respectively. Supports competitive smaller models. Section 3.1 says BoolQ is an exception against Chinchilla, but Table 3 shows 85.3 > 83.7. This report preserves the table values and flags the conflict. e08 |
| Closed-book question answering NaturalQuestions: 3,610-question open-domain test split, 64-shot. TriviaQA: filtered development set, zero-shot. | LLaMA-65B: NaturalQuestions 39.9; TriviaQA 68.2. Normalized exact match (%) | NaturalQuestions: PaLM-540B 39.6. TriviaQA: Chinchilla-70B 55.4. The protocols are separate. Appendix A warns GPT-3/PaLM used unfiltered TriviaQA test data, preventing a direct merged comparison. e09e18 |
| RACE reading comprehension Zero-shot RACE-middle and RACE-high; Section 3.3 follows Brown et al.'s setup without further split detail. | LLaMA-65B: 67.9 middle, 51.6 high. Accuracy (%) | PaLM-540B: 68.1 middle, 49.1 high. Competitiveness depends on the subset; LLaMA does not win both. e10 |
| GSM8k mathematical reasoning Table 7 standard evaluation and majority voting over 100 samples per problem; split and exact prompts are not specified locally. | LLaMA-65B: 50.9 standard; 69.7 with majority voting. Reported problem-solving score (%) | Minerva-62B: 52.4 standard; 68.5 with majority voting. The stated advantage over Minerva-62B holds only with voting. LLaMA has no math-specific finetuning, but its pretraining does include arXiv. e11e03 |
| Code generation HumanEval zero-shot and MBPP three-shot; temperature 0.1 for pass@1, with split identifiers not restated. | LLaMA-65B: HumanEval 23.7; MBPP 37.7. pass@1 (%) | PaLM-540B: 26.2 and 36.8, respectively. The general model wins MBPP but trails on HumanEval. These are code-test metrics, not evidence of physical action execution. e12 |
| MMLU knowledge evaluation Five-shot evaluation using benchmark demonstrations; Appendix B identifies subject test sets. | LLaMA-13B 46.9; LLaMA-65B 63.4; LLaMA-I-65B 68.9. Accuracy (%) | GPT-3-175B 43.9; Chinchilla-70B 67.5 in Table 9; PaLM-540B 69.3. Base 65B trails the strongest comparators; instruction finetuning adds 5.5 percentage points. Table 16 prints Chinchilla's aggregate as 67.6, a source discrepancy left unresolved. e14e21 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 1. Longer training continues to reduce the plotted language-model training loss. Original paper, p. 3 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow one colored line from left to right before comparing models. The horizontal axis counts billions of training tokens; the vertical axis is labeled Training loss, with lower values indicating a smaller training objective. The blue and orange runs terminate at 1,000 billion tokens, and the green and red runs extend to 1,400 billion. The legend links these to 7B, 13B, 33B and 65B. At matched token counts the larger models generally have lower loss, while every line still slopes downward late in training. The sharp visible spikes are part of the source graphic; their individual causes are not identified here. e13e02e04
What it supports. The training curves support continued optimization over the reported token budgets and explain why the authors investigate training smaller networks for longer. They do not, by themselves, show how much useful task performance is purchased by each extra token. Figure 2 supplies the complementary downstream view.
Where the evidence stops. The figure reports training loss, with no held-out-loss series or uncertainty bands. It is a training diagnostic rather than a controlled ablation or direct measurement of inference latency, financial cost or generalization.
Figure 2. Downstream improvements are broad, but smooth loss reduction does not imply smooth gains everywhere. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use the shared color legend in the lower-right panel to identify model sizes across all six tasks. Each panel has its own vertical scale, so apparent steepness should not be compared directly between tasks. Solid lines are LLaMA checkpoint trajectories; the dashed purple line is a Chinchilla reference score, not a Chinchilla learning curve. Inspect HellaSwag and the question-answering panels for sustained improvement, then contrast SIQA's fluctuations and the close green/red WinoGrande traces. Section 3.7 explicitly draws attention to these exceptions. This figure is most informative when read alongside Figure 1's training loss rather than treated as another final leaderboard. e13e04e05
What it supports. Additional training generally improves the monitored benchmarks, supporting the central long-training strategy within these runs. SIQA and WinoGrande qualify that conclusion: smaller changes or closely tracking curves can make a larger model's lower training loss less informative about its advantage on a particular task.
Where the evidence stops. These checkpoint traces do not isolate training duration from the rest of the optimization schedule. The caption does not specify the shot count for every panel, and no uncertainty bands are shown; avoid extracting exact final scores or significance from the curves.
7. Analysis & limitations
7.1 What the evidence leaves open
No controlled RMSNorm, SwiGLU, RoPE or data-mixture ablation establishes each component's contribution. The explanation that fewer books cause lower MMLU performance is the authors' hypothesis, not an isolated experiment; comparisons also use baseline numbers from other papers. e04e08e13e14
Risk evaluations remain narrow. RealToxicityPrompts comparisons depend on decoding and an external scoring service; Section 5.1 names PerspectiveAPI while Table 11's caption says PerplexityAPI. CrowS-Pairs and WinoGender reveal biases. TruthfulQA gives 65B only 0.57 truthful and 0.53 truthful-and-informative fractions under model-based scoring. e15e16e17
Carbon estimates are standardized scenarios, not measured local emissions: Table 15 assigns 65B 449 MWh and 173 tonnes CO2-equivalent using 400 W/GPU, PUE 1.1 and US-average carbon intensity. Development-wide estimates include much more than the final run. e22
7.2 Questions for discussion
- At what serving volume would additional pretraining repay its cost?
- Does longer training improve stable held-out capability when prompts and learning-rate schedules are controlled?
- Would a controlled increase in books improve MMLU without reducing other capabilities?
8. Reproducibility audit
8.1 Requirements and known gaps
A faithful reconstruction needs the filtered corpus snapshots, tokenizer, architecture, optimizer and distributed implementation. The paper provides mixture weights and some thresholds, but not every filtering cutoff, software version or exact instruction-finetuning setting; public data descriptions alone do not recreate the training corpus. e03e04e05e06e14
Proposed checks should first fix QA formatting, answer extraction and scoring, then compare checkpoints under identical prompts. Testing longer training requires controlling model size, data order and schedule rather than treating final cross-size scores as a token-budget ablation. e07e13e18
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Check whether scoring choices explain apparent benchmark gains
Reader-proposed check; not performed. With an independently obtained fixed LLaMA checkpoint, evaluate the same BoolQ and OpenBookQA examples under the paper's Answer: likelihood ratio and under character-normalized likelihood. Keep prompts, candidate order, tokenizer and precision fixed, and save both per-candidate scores and selected answers. Compare paired accuracy and disagreement counts, with uncertainty over examples. The falsifiable observation is whether rankings or model comparisons materially change solely because of scoring. As a separate harness control, verify Appendix A's QA stopping and normalization on fixed generated strings before attempting its NaturalQuestions result. Record the precise split and demonstrations so a discrepancy cannot be hidden by changing the evaluation protocol. e07e08e18
Check 2: Test late-training gains without changing the model
Reader-proposed check; not performed. Evaluate earlier and later checkpoints from one fixed-size training run on HellaSwag, SIQA and WinoGrande with identical prompts, scoring and held-out examples. Report paired changes and uncertainty alongside training loss and token count. If extending a run, compare against a same-size control trained to the shorter budget with its own properly completed cosine schedule; keep the corpus mixture and data order fixed. The prediction motivated by Figure 2 is a steadier improvement on HellaSwag than on SIQA, with weaker separation on WinoGrande. If loss decreases while fixed-protocol task scores stagnate, that would limit the proposed benefit of extra training on those tasks. This is a mechanism check, not a claim to reproduce the full 65B run. e03e04e05e07e13
8.3 Reading coverage
Visual audit: Inspected the title, complete byline and v1 identifier on p. 1; data, architecture, optimization and implementation on pp. 2–4; all sixteen quantitative/configuration tables across pp. 2–11 and 18; Figures 1–2 on pp. 3 and 8; QA protocol and Figure 3 on p. 17; and representative instruction-generation pages 22 and 27. All six final crops were viewed individually with complete headers, axes and legends. Table/prose conflicts concerning BoolQ, GSM8k and the MMLU aggregate were checked against their pages. Appendix examples on pp. 19–21 and 23–26 and references on pp. 12–16 were read as text only; no claim about their image details is made. All pages supporting the edition's method, numerical and proposed-check details are included above.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 17, 18, 22, 27. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract and Section 1, Introduction (p. 1)
- Sections 2.1–2.4: data, architecture, optimizer and efficient implementation (pp. 2–4)
- Sections 3.1–3.7: all benchmark results and training evolution (pp. 4–8)
- Section 4, Instruction Finetuning (p. 7)
- Sections 5.1–5.4: toxicity, bias and truthfulness (pp. 7–10)
- Sections 6–8: carbon footprint, related work and conclusion (pp. 10–11)
- Acknowledgements and References (pp. 12–16)
- Appendix A, Question Answering (p. 17)
- Appendix B, MMLU (p. 18)
- Appendix C, Generations from LLaMA-65B (pp. 19–21)
- Appendix D, Generations from LLaMA-I (pp. 22–27)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout. This supplied extraction caveat was addressed by inspecting all three numbered figures and all sixteen tables in the PDF.
- Separate supplemental material availability has not been fully verified.
- Identity/version note: the title page identifies arXiv:2302.13971v1, 27 February 2023. The title and all 14 authors match the catalog; the PDF prints Timothee Lacroix without the catalog accent in Timothée. No author additions were observed.
- The manifest records an earlier LaTeX artifact at https://arxiv.org/src/2302.13971, SHA-256 a2295f429b5e15cf53a16ab6dfe240a96c96a2ed7ca66e6f0785c196761c5b28. That archive was not re-read; complete equivalence with this PDF is unverified. This report grounds its claims in the supplied PDF, not a later edition.
- All eight supplied text chunks were read. Appendix generation examples were read as text throughout; their page images were inspected on pp. 22 and 27 only. Reference pages were read as text.
- Code, model weights, external references and repositories were not inspected; no experiments were reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title, byline, affiliation and arXiv margin
Title matches the catalog; fourteen authors are credited under Meta AI. The margin identifies 2302.13971v1, 27 Feb 2023; Timothee is printed without an accent.
Go to primary source ↓e02PDF p. 1, Abstract and Section 1
Motivates smaller models trained longer for inference budgets, introduces the 7B–65B family and describes using publicly available data.
Go to primary source ↓e03PDF p. 2, Section 2.1 and Table 1, all rows and caption
Mixture: CommonCrawl 67%, C4 15%, GitHub/Wikipedia/books 4.5% each, arXiv 2.5%, StackExchange 2%. Epoch counts refer to 1.4T tokens; 1T runs share proportions. Describes filtering, book-overlap threshold, multilingual Wikipedia and digit/byte BPE.
Go to primary source ↓e04PDF p. 3, Section 2.2 and Table 2
RMSNorm pre-normalization, SwiGLU and per-layer RoPE modify the transformer. Table gives parameter counts, widths, heads, layers, peak learning rates, 4M-token batches and 1T/1.4T budgets.
Go to primary source ↓e05PDF p. 3, Section 2.3
AdamW beta1=0.9, beta2=0.95, weight decay 0.1, clipping 1.0, 2,000 warmup steps and cosine decay to 10% of maximum learning rate.
Go to primary source ↓e06PDF pp. 3–4, Section 2.4
Describes causal attention without storing attention weights or computing masked scores, selective checkpointing, manual backward, model/sequence parallelism and communication overlap; reports 380 tokens/s/GPU on 2,048 A100-80GB GPUs and approximately 21 days for 65B.
Go to primary source ↓e07PDF p. 4, Section 3, zero/few-shot definitions and unnumbered completion-scoring expression
Generation or completion ranking handles evaluation. Typical ranking normalizes by characters; BoolQ/OpenBookQA divide conditional completion likelihood by likelihood after Answer:. Few-shot examples are provided in the input.
Go to primary source ↓e08PDF p. 4, Table 3, GPT-3/Chinchilla/LLaMA rows; p. 5, Section 3.1
13B HellaSwag/BoolQ are 79.2/78.1 versus GPT-3 78.9/60.5. 65B gives 84.2/85.3 versus Chinchilla 80.8/83.7. The prose's BoolQ exception against Chinchilla conflicts with the table. Baseline numbers come from corresponding papers.
Go to primary source ↓e09PDF p. 4, Table 4, 64-shot column; p. 5, Section 3.2 and Table 5, zero-shot column
NaturalQuestions: 65B 39.9 versus PaLM-540B 39.6 at 64-shot. TriviaQA filtered-dev zero-shot: 65B 68.2 versus Chinchilla 55.4. Closed-book models receive no evidence documents.
Go to primary source ↓e10PDF p. 5, Section 3.3 and Table 6
Zero-shot RACE-middle/high: 65B 67.9/51.6 and PaLM-540B 68.1/49.1; setup follows Brown et al.
Go to primary source ↓e11PDF p. 5, Section 3.4; p. 6, Table 7, GSM8k columns and caption
65B standard/voted GSM8k scores are 50.9/69.7 versus Minerva-62B 52.4/68.5. Voting uses 100 samples for GSM8k and 256 for MATH, with different counts noted for Minerva-540B. LLaMA has no math-specific finetuning.
Go to primary source ↓e12PDF pp. 5–6, Section 3.5; p. 6, Table 8 and caption
HumanEval zero-shot and MBPP three-shot pass@1: 65B 23.7/37.7 versus PaLM-540B 26.2/36.8. Pass@1 temperature is 0.1; larger pass@k uses 0.8. Starred comparator cells were read from another paper's figures.
Go to primary source ↓e13PDF p. 3, Figure 1 and caption; pp. 6–7, Section 3.7; p. 8, Figure 2
Training-loss curves extend to 1T for 7B/13B and 1.4T for 33B/65B. Six benchmark trajectories mostly improve; SIQA fluctuates and larger-model WinoGrande curves are close. These are checkpoint diagnostics, not component ablations.
Go to primary source ↓e14PDF p. 6, Section 3.6; p. 7, Tables 9–10 and Section 4
Five-shot MMLU: 13B 46.9, 65B 63.4, LLaMA-I 68.9, GPT-3 43.9, Chinchilla 67.5, PaLM-540B 69.3. One instruction experiment follows Chung et al.; the books-based explanation of MMLU differences is tentative.
Go to primary source ↓e15PDF p. 7, Section 5 opening; p. 8, Section 5.1 and Table 11 caption
Authors caution that selected risk benchmarks are insufficient. RealToxicityPrompts uses greedy generation on 100k prompts; API timing and methodology complicate comparisons. Prose says PerspectiveAPI; caption says PerplexityAPI.
Go to primary source ↓e16PDF p. 9, Sections 5.2–5.3 and Table 12; p. 10, Table 13
CrowS-Pairs measures stereotypical sentence preference. WinoGender evaluates pronoun-dependent coreference and gotcha cases inconsistent with occupational gender stereotypes; results indicate persistent bias.
Go to primary source ↓e17PDF pp. 9–10, Section 5.4; p. 10, Table 14 and caption
65B truthful and truthful-and-informative fractions are 0.57 and 0.53; trained models score answers via the OpenAI API. Authors highlight remaining incorrect answers.
Go to primary source ↓e18PDF p. 17, Appendix A and Figure 3
NaturalQuestions uses a 3,610-question open-domain test split; TriviaQA uses filtered dev, unlike GPT-3/PaLM unfiltered test. Specifies the QA prefix, greedy decoding, answer stopping and exact-match normalization.
Go to primary source ↓e19PDF p. 22, Appendix D opening; p. 27, terminal-dialogue generation example
LLaMA-I is described as 65B finetuned with Chung et al.'s instruction dataset/protocol. Appendix dialogue illustrates generated terminal-like text, not documented execution.
Go to primary source ↓e20PDF p. 10, Section 7 opening; pp. 3–4, Sections 2.2–3
Language modeling concerns token-sequence probabilities and next-token prediction. The method and evaluations specify text processing, generation and ranking rather than action-conditioned environment prediction or control.
Go to primary source ↓e21PDF p. 18, Appendix B, Table 16, caption and aggregate rows
Lists 57 MMLU subject test-set results. LLaMA-65B and LLaMA-I aggregates are 63.4/68.9. Chinchilla All is 67.6, while Table 9 on p. 7 prints 67.5.
Go to primary source ↓e22PDF p. 10, Section 6; p. 11, Table 15, LLaMA-65B row and caption
65B estimate: 449 MWh and 173 tCO2eq. Assumptions are A100-80GB at 400 W, PUE 1.1 and 0.385 kg CO2eq/kWh. Full development is separately estimated at roughly five months on 2,048 GPUs.
Go to primary source ↓8.5 Primary sources
LLaMA: Open and Efficient Foundation Language Models ↗
PDF · 14,524 extracted words
Source fingerprint
2e663675ae36ad12adb2f5a05281bac2747ecf8d23d92bedd9f937a89fee7136