LoRA: Low-Rank Adaptation of Large Language Models
1. Paper overview
In one sentence: LoRA makes task adaptation small by learning low-rank weight updates that can be merged for inference, while useful rank and update placement remain task-dependent. ev-identityev-objectiveev-updateev-targetsev-allocationev-rank
| At a glance | What to know |
|---|---|
| Research problem | Source description Full fine-tuning creates another full-size parameter set for every task and requires optimizer state for all updated weights. LoRA asks whether downstream adaptation can instead be encoded by small matrices while retaining the pretrained model's inference structure. The motivating problem is supervised language adaptation, not environment prediction or action execution. ev-objectiveev-update |
| Core mechanism | Source description A linear, low-rank residual parameterization separates shared pretrained weights from task-specific updates and permits merging at deployment. ev-updateev-targets |
| A key reported result | GPT-3 MNLI-matched natural language inference: 91.7 Validation accuracy (%). 175B model; full MNLI matched validation set, best validation performance. Table 15 identifies the 4.7M configuration as value-only rᵥ=2. Full fine-tuning: 89.5; 40.1M-parameter AdapterH: 91.5. A 2.2 percentage-point gain over full tuning. Typical seed variation is about ±0.1 percentage points, not a per-row confidence interval. ev-gpt3-resultev-gpt3-detailsev-gpt3-protocol |
| Reading caution | Source description Merged weights complicate mixed-task batching. Attention-only experiments do not establish the best placement for other architectures; the authors explicitly warn that small ranks may fail on different tasks or languages. ev-targetsev-rank |
Core contributions
- Source description
A linear, low-rank residual parameterization separates shared pretrained weights from task-specific updates and permits merging at deployment. ev-updateev-targets
- Author claim
The authors report competitive adaptation across RoBERTa, DeBERTa, GPT-2 and GPT-3, and interpret rank and subspace experiments as evidence of low intrinsic adaptation rank. ev-glueev-gpt2-resultev-gpt3-resultev-rankev-subspace
- Reader analysis
GPT-3 training memory is reported as 1.2 TB versus 350 GB; query/value rank-4 task checkpoints are about 35 MB versus 350 GB. The base model remains necessary. The prose's 25% training-speedup claim and footnote's 32.5 versus 43.1 tokens/s/V100 use unreconciled percentage descriptions. ev-efficiency
Figure 1. A small trainable residual runs alongside a frozen dense projection. Original paper, p. 1 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read upward from x. The left arrow enters the pretrained weight W, shown as a square d-by-d matrix. The right branch first passes through A, narrows to dimension r, and then passes through B. Both contributions meet at the plus sign before h. The caption says only A and B are trained. The initialization labels describe the starting values, not constraints that hold throughout training. Cross-check Section 4.1 for the general rectangular shapes and the factor α/r, which is not drawn. There is no nonlinear activation between A and B in this formulation; that linear structure makes merging possible. ev-updateev-targets
What it supports. The update changes the mapping without replacing the pretrained weight. Zero initialization of B initially suppresses the residual, while the trained product BA can later be added to the original matrix. The resulting dense layer has the original input/output dimensions and needs no separate adaptation branch after merging.
Where the evidence stops. The diagram depicts the unmerged computation. Its arrows agree with Equation (3), but neither displays the α/r scale specified in the following prose. Include that scale when implementing the update or merge.
2. Motivation
2.1 The problem and the proposed response
Full fine-tuning creates another full-size parameter set for every task and requires optimizer state for all updated weights. LoRA asks whether downstream adaptation can instead be encoded by small matrices while retaining the pretrained model's inference structure. The motivating problem is supervised language adaptation, not environment prediction or action execution. ev-objectiveev-update
2.2 What this reading follows
A large pretrained language model can be useful for many tasks, but a separate fully tuned copy for each task is expensive to train and store. LoRA keeps the shared weights fixed and learns a small additive correction. The key is that this correction is linear: its two matrices can be multiplied and absorbed into the original layer before inference. Read the mechanism first, then separate the evidence about deployment cost from the evidence about prediction quality. Finally, use the two ablations to distinguish where an update is placed from how much rank it receives. This edition covers the supplied October 2021 arXiv v2 draft, not an asserted copy of the ICLR 2022 publication. ev-identityev-objectiveev-updateev-targetsev-allocationev-rank
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Foundational work |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
3.1 Evidence-based assessment
Supports the recorded classification
The foundational training-optimization placement is supported. LoRA changes how language-model weights are adapted; it does not define a world model, action predictor or control loop. Architecture, prediction paradigm and quadrant are therefore not applicable. This paper does not establish a distillation method merely because the catalog subcategory also mentions distillation. ev-objectiveev-updateev-targets
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Follow one hidden vector through the adaptation
Start with a hidden input vector x at one dense projection. The pretrained matrix maps it directly to the original output space. LoRA sends the same vector through A into an r-dimensional intermediate space, then through B back to the output dimension; the two outputs are added. Only A and B change during adaptation. Gaussian A and zero B make the initial residual zero, so the starting layer implements its pretrained function. The text additionally scales the residual by α/r, which is absent from the schematic. Training still propagates the task objective through the network to learn the small branch. At deployment, multiplication and addition can be folded into one dense weight. This explains why a small task checkpoint saves storage while the large base model remains necessary. ev-updateev-objectiveev-efficiency
5.2 Separate a parameter budget from a deployment budget
Reader analysis: fewer trainable parameters answer only one resource question. Table 4 shows that small updates can preserve strong predictions, while Table 1 asks whether an adaptation mechanism adds work to each forward pass. A sequential adapter may have few weights yet introduce operations that matter at small batch sizes. LoRA's merged form avoids that additional path, but the base model is still evaluated. Likewise, storing a 35 MB task update is different from deploying a 35 MB language model: the draft explicitly retains the 350 GB GPT-3 base. Compare accuracy within each task before comparing resource costs across methods. Also retain the operating condition: one merged task is straightforward, whereas batches mixing different task updates require a different implementation and lose the simple single-weight interpretation. ev-gpt3-resultev-latencyev-efficiencyev-targets
5.3 Read low rank as an empirical design clue
Reader analysis: the two ablations ask different questions. Table 5 holds the total update budget approximately fixed and distributes it across attention projections; Table 6 keeps a projection choice and increases rank, thereby increasing trainable capacity. The first supports spreading updates beyond query-only adaptation, while the second shows that a very small rank can already be competitive. Neither proves that every downstream task has one useful adaptation direction. The separate GPT-2 experiment makes the boundary concrete: BLEU peaks at rank 4, and its caption admits that some hyperparameters were tuned at that rank. The singular-vector studies offer a possible explanation through shared leading directions, but correlation is not a causal test of feature usefulness. A defensible reproduction must control optimization and evaluation while varying rank or placement. ev-allocationev-rankev-gpt2-rankev-subspaceev-feature
5.4 Training and inference
During training
Optimize the task objective through the small parameter set Θ. The appendices specify AdamW and linear learning-rate schedules; the frozen network still supplies activations needed to learn A and B. ev-objectiveev-updateev-glue-protocolev-gpt2-protocolev-gpt3-protocol
RoBERTa and DeBERTa results use median performance over five seeds with best-epoch selection. Standard MRPC/RTE/STS-B runs initialize LoRA from the best MNLI adaptation; dagger-marked RoBERTa comparisons instead start from pretrained weights. ev-glue-protocol
GPT-3 uses two epochs, batch size 128 and weight decay 0.1; sequence lengths are 384/768/2048 for WikiSQL/MNLI/SAMSum. Table 12 lists LoRA learning rate 2×10⁻⁴ and 250,000 warmup tokens. Best validation performance is reported. ev-gpt3-protocol
During inference
Merge the scaled low-rank update into W₀ before ordinary inference. Task switching replaces the update. The zero-extra-latency claim concerns this merged computation, not dynamic execution of multiple task-specific branches. ev-updateev-targets
GPT-2 generation uses beam size 10, no repeated four-grams, and length penalties 0.9 for E2E or 0.8 for WebNLG/DART. These decoding choices are part of the evaluated system. ev-gpt2-protocol
5.5 Implementation flow
- Freeze the shared model
For a pretrained matrix W₀, constrain the trainable update to BA. A reduces the input to rank r; B maps it to the output dimension. The pretrained path still runs, but its weights receive no updates. ev-update
- Choose attention projections
Most experiments adapt query and value projections. The study freezes Transformer MLP weights and leaves adaptation of MLPs, LayerNorms and biases for future investigation. Placement is a design choice tested separately from rank. ev-targetsev-allocation
- Initialize and scale
Initialize A with Gaussian noise and B to zero, making the initial update zero. Section 4.1 scales the residual by α/r; Figure 1 and displayed Equation (3) omit that scale. ev-update
6. Experiments & results
LoRA adapts a frozen language model by learning low-rank additive weight updates. These updates can be merged into the original dense layers for inference. This v2 draft demonstrates competitive language understanding and generation with far fewer trainable parameters, while its ablations show that update placement and task-specific rank matter.
6.1 Read the original evidence
Table 1. Small-batch inference exposes the cost of sequential adapter layers. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Each column is a distinct operating point: read batch size and sequence length before reading the latency below. The |Θ| header counts trainable adapter parameters, so it is not a count of the entire GPT-2 model or of a particular LoRA configuration. Values are milliseconds from one forward pass on a Quadro RTX8000, averaged over 100 trials. Fine-Tune/LoRA shares a row because the merged LoRA model uses the original dense structure. The percentages in parentheses describe adapter overhead relative to that baseline. Appendix B's heatmaps expand this comparison across batch sizes, sequence lengths and adapter bottlenecks. ev-latencyev-updateev-targets
What it supports. At batch 1 and length 128, the shared Fine-Tune/LoRA row is 19.8 ± 2.7 ms, versus 23.9 ± 2.1 for AdapterL and 25.8 ± 2.2 for AdapterH. The reported overheads are 20.7% and 30.3%, illustrating why a low parameter count alone does not ensure low latency.
Where the evidence stops. These are hardware- and shape-specific forward-pass measurements. They do not measure full autoregressive response time, task-switching overhead, or mixed-task serving. The zero-added-latency argument assumes a correctly merged LoRA update.
Table 3. Small task updates preserve strong restaurant data-to-text performance. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Separate the GPT-2 medium block above the divider from the large-model block below it. Within each block, compare trainable parameters before moving across the five quality metrics; all are higher-is-better here. E2E converts restaurant slot-value inputs into natural-language descriptions. Asterisks mark values taken from prior work, including the full-tuning and prefix-layer baselines, rather than reruns with uncertainty reported in this draft. Appendix D.3 reports means over three random seeds for the authors' GPT-2 experiments, selecting the best epoch per run. The metric labeled MET is METEOR. Table 11 supplies the training and beam-search settings needed to interpret these scores. ev-gpt2-resultev-dataev-gpt2-protocol
What it supports. For GPT-2 medium, LoRA reaches 70.4 ± 0.1 BLEU with 0.35M trainable parameters, compared with 68.2 for full tuning of 354.92M parameters and 69.7 for similarly sized prefix-layer tuning. This supports efficient generation adaptation; it does not make every metric or model-size comparison a strict win.
Where the evidence stops. The caption calls the ± values confidence intervals without specifying a confidence level. Imported baselines have no corresponding intervals here. Table 3 does not explicitly label its evaluation split, and the large-model CIDEr column does not favor LoRA over every adapter.
Table 4. Large-model adaptation remains competitive across three distinct tasks. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each task column independently: WikiSQL measures logical-form validation accuracy, MNLI-m measures matched validation accuracy, and SAMSum lists ROUGE-1/2/L together. The parameter column describes trainable task parameters, not total model size. Two LoRA budget rows appear at the bottom, and they need not represent one common projection configuration across tasks. In particular, Table 15 links the 4.7M MNLI score of 91.7 to value-only rank-2 adaptation; query/value rank-1 adaptation has the same parameter count but reaches 91.3 there. The caption provides typical task-level seed fluctuations instead of uncertainty for each individual row. ev-gpt3-resultev-gpt3-detailsev-gpt3-protocolev-data
What it supports. At 4.7M trainable parameters, LoRA reports 91.7% MNLI-m accuracy versus 89.5% for full tuning, and SAMSum ROUGE 53.8/29.8/45.9 versus 52.0/28.0/44.5. WikiSQL gives a more restrained comparison: 73.4 at the smaller LoRA budget versus 73.8 for full tuning, with the larger LoRA budget reaching 74.0.
Where the evidence stops. WikiSQL differences are within the stated typical ±0.5-point fluctuation. SAMSum's split description is ambiguous: Appendix C lists training/test counts, while D.4 says best validation performance. The source does not resolve their relationship in this table.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| GLUE language understanding Table 2 standard, non-dagger runs; LoRA medians over five seeds and best epochs. Table 2 does not explicitly label the evaluation split. | RoBERTa-base: 87.2 with 0.3M trainable parameters; DeBERTa-XXL: 91.3 with 4.7M. Reported eight-task average; higher is better | Published full-tuning baselines: 86.4 with 125M and 91.1 with 1500M, respectively. Competitive aggregate results; this average combines different metrics and does not imply improvement on every task or identical tuning budgets. ev-glueev-glue-protocol |
| E2E restaurant data-to-text generation GPT-2 medium; E2E benchmark following the cited baseline setup, mean over three seeds with best-epoch selection. Table 3 does not explicitly name the evaluation split. | 70.4 ± 0.1 with 0.35M trainable parameters. BLEU; higher is better | Full fine-tuning: 68.2 with 354.92M; prefix-layer tuning: 69.7 with 0.35M. Both comparison values are imported from prior work. A 2.2 BLEU-point gain over the listed full-tuning baseline. The caption calls ± values confidence intervals but does not give their confidence level. ev-gpt2-resultev-dataev-gpt2-protocol |
| GPT-3 MNLI-matched natural language inference 175B model; full MNLI matched validation set, best validation performance. Table 15 identifies the 4.7M configuration as value-only rᵥ=2. | 91.7 Validation accuracy (%) | Full fine-tuning: 89.5; 40.1M-parameter AdapterH: 91.5. A 2.2 percentage-point gain over full tuning. Typical seed variation is about ±0.1 percentage points, not a per-row confidence interval. ev-gpt3-resultev-gpt3-detailsev-gpt3-protocol |
| GPT-3 WikiSQL semantic parsing 175B model; 56,355 training and 8,421 validation examples; logical-form evaluation. | 73.4 at 4.7M; 74.0 at 37.7M trainable parameters. Validation accuracy (%) | Full fine-tuning: 73.8. The smaller LoRA model is numerically lower, and the larger is only 0.2 percentage points higher; typical variation is ±0.5. This supports competitiveness, not an assured gain. ev-gpt3-resultev-dataev-gpt3-details |
| GPT-3 SAMSum dialogue summarization 175B model; Appendix C lists 14,732 training/819 test examples, while D.4 says best validation performance. Their relationship is not resolved in Table 4. | 53.8 / 29.8 / 45.9 at 4.7M trainable parameters. ROUGE-1 / ROUGE-2 / ROUGE-L | Full fine-tuning: 52.0 / 28.0 / 44.5. Higher reported overlap scores; typical fluctuations are ±0.2/±0.2/±0.1. The split/selection ambiguity prevents describing this as an independently verified held-out test result. ev-gpt3-resultev-dataev-gpt3-protocol |
| GPT-2 medium inference latency One forward pass; Quadro RTX8000; batch 1, sequence 128; 100 trials; adapter budget 11M. | Fine-Tune/LoRA: 19.8 ± 2.7. Milliseconds; lower is better | AdapterL: 23.9 ± 2.1; AdapterH: 25.8 ± 2.2. The reported adapter overheads are 20.7% and 30.3%. This is a particular forward-pass benchmark, not end-to-end autoregressive serving latency. ev-latency |
| GPT-3 update-placement ablation Table 5; fixed approximately 18M trainable parameters across 96 layers. | Query+value, rank 4: 73.7 / 91.3. WikiSQL / MultiNLI validation accuracy (%) | Query-only, rank 8: 70.4 / 91.0; all four attention projections, rank 2: 73.7 / 91.7. Spreading capacity improves over query-only. Contrary to the caption's blanket preference, all-four ties query/value on WikiSQL and exceeds it on MultiNLI. ev-allocation |
| GPT-3 rank ablation Table 6; query/value updates; parameter count grows with rank. | r=1: 73.4 / 91.3; r=8: 73.8 / 91.6; r=64: 73.5 / 91.4. WikiSQL / MultiNLI validation accuracy (%) | Query-only r=1: 68.8 / 90.7. Very small ranks are competitive on these tasks; increasing rank is not monotonically beneficial, and placement remains important. ev-rank |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 5. Where the update is placed matters at a fixed parameter budget. Original paper, p. 10 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read a column as a complete allocation policy across GPT-3's 96 layers. Wq, Wk, Wv and Wo denote query, key, value and attention-output projections. One selected projection uses rank 8, two use rank 4, and all four use rank 2, keeping the total update budget approximately fixed. The table prints 18M, while Table 15 gives 18.8M for corresponding configurations. Compare columns within a task row; the uncertainty labels describe typical seed variation for that task. This design tests where to spend adaptation capacity. It is different from increasing rank while leaving the number of adapted projections unchanged. ev-allocationev-gpt3-details
What it supports. Query/value rank 4 improves WikiSQL over query-only rank 8 from 70.4% to 73.7%. However, the all-four rank-2 configuration also reaches 73.7% and improves MultiNLI from query/value's 91.3% to 91.7%. The cells therefore support distributing updates while qualifying the paper's stated preference for query/value.
Where the evidence stops. The caption and following paragraph call query/value best overall, but the all-four column ties or exceeds it on both displayed tasks. Preserve this discrepancy. The table also changes placement and per-matrix rank together, so it does not isolate either alone.
Table 6. Very low rank is competitive, but increasing rank does not produce a universal gain. Original paper, p. 10 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Choose a task block, then hold the weight-type row fixed as you move from rank 1 to rank 64. This increases the trainable parameter count, unlike the preceding fixed-budget allocation table. Next compare rows at one rank, remembering that more projection types also mean more parameters. Query-only adaptation is visibly weaker on WikiSQL than adaptation including value projections. The ± labels are task-level seed variations, not confidence intervals attached to every cell. The rank-1 query/value MNLI entry is 91.3; it should not be confused with Table 4's 91.7 result from a different configuration with the same total budget. ev-rankev-gpt3-detailsev-gpt2-rank
What it supports. Query/value rank 1 reaches 73.4% WikiSQL and 91.3% MultiNLI, compared with 73.8% and 91.6% at rank 8. Rank 64 gives 73.5% and 91.4%, so adding capacity is not monotonically helpful. These observations support low-rank adaptation on the evaluated GPT-3 tasks without identifying one universally optimal rank.
Where the evidence stops. Table 15 reports 73.6 rather than 73.5 for query/value rank-64 WikiSQL; the draft does not reconcile that difference. Its separate GPT-2 study peaks at rank 4 for BLEU and acknowledges rank-specific hyperparameter tuning.
7. Analysis & limitations
7.1 What the evidence leaves open
Merged weights complicate mixed-task batching. Attention-only experiments do not establish the best placement for other architectures; the authors explicitly warn that small ranks may fail on different tasks or languages. ev-targetsev-rank
Subspace overlap and projected-weight norms motivate the authors' feature-amplification interpretation, but do not prove that omitted singular directions are useless. Appendix G also conflicts with Section 7.2 on left/right singular-vector terminology and prints an inconsistent projection-distance equality. ev-subspaceev-featureev-subspace-notation
GPT-2's separate 26,000-step rank study peaks at rank 4 for BLEU; several ranks tie the displayed minimum validation loss. Its hyperparameters were partly tuned at rank 4, so it does not identify a task-independent optimal rank. ev-gpt2-rank
7.2 Questions for discussion
- How much of the benefit comes from low rank versus distributing updates across attention projections?
- Would rank preferences persist under equally budgeted hyperparameter searches and independently held-out evaluation?
8. Reproducibility audit
8.1 Requirements and known gaps
A practical starting point is GPT-2 medium on E2E: AdamW, five epochs, batch 8, learning rate 0.0002, 500 warmup steps, query/value rank 4, α=32, dropout/label smoothing 0.1 and weight decay 0.01. Preserve Table 11 decoding and three-seed aggregation. ev-gpt2-protocol
Dagger comparisons control initialization and sequence length, but Table 9 lists LoRA batch 4 versus adapter batch 32; do not assume identical optimization budgets. Resolve D.4's association of all-four rank 2 with 37.7M: Table 15 gives 18.8M and uses rank 4 for 37.7M. The query/value rank-64 WikiSQL score also differs between Tables 6 and 15. ev-glue-protocolev-gpt3-protocolev-gpt3-detailsev-rank
The source announces a PyTorch package and selected model checkpoints. Exact code revision, package versions, GPU count and full search budgets are not established here; V100 training and the separate RTX8000 latency setup are reported. Proposed controlled checks appear in the illustrated edition. ev-identityev-efficiencyev-latencyev-gpt2-protocolev-gpt3-protocol
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Verify the merge, then measure its cost
Reader-proposed check, not performed: adapt GPT-2 medium on E2E using Table 11, retaining the pretrained weights and both low-rank matrices. With dropout disabled and identical inputs, compare logits from the explicit scaled branch against logits from the merged dense weights, first in a common high-precision reference and then in the chosen deployment precision. Use the unchanged pretrained model as an initialization control: the zero-B model should match it before training. Measure synchronized forward-pass latency over 100 trials at batch 1/length 128 and batch 32/length 512, holding device and precision fixed, and report merge time separately. A persistent logit discrepancy beyond the declared numerical tolerance falsifies the implementation's equivalence; extra steady-state latency after merging challenges the claimed deployment behavior in that implementation. ev-updateev-latencyev-gpt2-protocol
Check 2: Disentangle rank, placement and tuning advantage
Reader-proposed check, not performed: on GPT-2 medium/E2E, compare query-only rank 8, query/value rank 4, and all-four attention projections rank 2, verifying equal trainable parameter counts. Separately sweep query/value ranks 1, 4 and 16. Use the same data partitions, training-token budget, seed set, decoding settings and learning-rate search budget for every configuration; state the α/r scaling policy. Select configurations using validation data and evaluate the untouched test split once. Report BLEU, validation loss, parameter count and seed dispersion. If placement gains disappear under matched tuning, the apparent advantage may be optimization-dependent. If rank 1 remains substantially worse across equally tuned runs, the GPT-3 small-rank result does not transfer to this task/model. This is a smaller mechanism test, not a reproduction of GPT-3 accuracy. ev-allocationev-rankev-gpt2-rankev-dataev-gpt2-protocol
8.3 Reading coverage
Visual audit: Visually inspected the title and author block; all main and appendix figures (1–8) and tables (1–18); objective, initialization, scaling, merge and projection-selection text; all training/evaluation configurations on pp. 18–24; and the subspace definitions and diagnostics on pp. 11–12 and 22–26. Every final crop was separately inspected. Pages 14–15 contain references and were read in text only. Figure 1's branch directions agree with Equation (3), but both omit the α/r scale stated in Section 4.1. Table 5's cells qualify its caption's claim. For uncropped diagnostics, Figure 3's displayed argument order differs from Equation (4), and Appendix G's singular-vector terminology and projection-distance equality are inconsistent; these are not silently repaired. Separate supplements, code and other editions remain outside the supplied reading.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract and Sections 1–3: motivation, objective and competing adaptation strategies (pp. 1–4)
- Section 4: low-rank parameterization, deployment and limitations (pp. 4–5)
- Section 5: baselines and RoBERTa, DeBERTa, GPT-2 and GPT-3 experiments (pp. 5–8)
- Sections 6–8: related work, update analysis and conclusions (pp. 8–13)
- References (pp. 13–16)
- Appendices A–C: few-shot comparison, adapter latency and datasets (pp. 16–18)
- Appendix D.1–D.4: all model hyperparameters (pp. 18–21)
- Appendices E–F.3: combinations, additional generation tasks and low-data experiments (pp. 20–24)
- Appendices G–H.4: subspace metric, cross-layer diagnostics, GPT-2 rank and amplification (pp. 22–26)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- Recovered-edition limitation: arXiv v2 preprint, explicitly labelled Version 2 on page 1. Catalog cites ICLR 2022; this draft is not asserted to be identical to the blocked ICLR publication PDF. All eight authors match; PDF uses Edward Hu where arXiv metadata and catalog use Edward J. Hu.
- All eight supplied text chunks were read individually. Figures 1–8 and Tables 1–18 were also inspected in the PDF. The extraction limitation above was addressed by this visual pass; it remains part of the acquisition record.
- This report reviews arXiv:2106.09685v2, dated 16 October 2021, rather than claiming to review the catalog's ICLR 2022 publication edition. Page 1 identifies Version 2 and says it adds better baselines, GLUE experiments and adapter-latency analysis relative to V1. Other editions were not supplied or compared.
- The title is visually consistent with LoRA: Low-Rank Adaptation of Large Language Models. The supplied observedTitle retains extraction spacing and line-break artifacts. The title page credits Edward Hu, without the catalog's middle initial J.; all eight authors otherwise agree.
- Code, linked checkpoints and external supplements were not inspected; no experiments were reproduced. Bibliography-only pages 14–15 were read as text but not rendered.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
ev-identityPDF p. 1, title/author block, arXiv margin, Version 2 label and revision footnote
LoRA title; Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang and Weizhu Chen; Microsoft Corporation. arXiv:2106.09685v2, 16 Oct 2021. Hu/Shen share equal contribution. Revision note adds baselines, GLUE and latency analysis. Abstract announces package and selected checkpoints.
Go to primary source ↓ev-objectivePDF pp. 2–3, Section 2, Equations (1)–(2)
Task data are context-target token sequences. Full tuning changes Φ; parameter-efficient tuning encodes ΔΦ with smaller Θ and optimizes conditional log likelihood.
Go to primary source ↓ev-updatePDF p. 1, Figure 1 and caption; p. 4, Section 4.1, Equation (3) and deployment paragraph
Frozen W₀ plus BA; both branches share x and outputs are summed. A is Gaussian-initialized, B zero-initialized. Text scales residual by α/r although figure/equation omit the scale. Merging preserves the original dense computation.
Go to primary source ↓ev-targetsPDF p. 5, Section 4.2; p. 6, final LoRA baseline paragraph
Study adapts attention weights, freezes MLPs, generally targets query/value, and leaves MLP/LayerNorm/bias adaptation for future work. Merged task updates complicate batching different tasks.
Go to primary source ↓ev-efficiencyPDF p. 5, Practical Benefits and Limitations; footnotes 4–5; Section 5 hardware statement
GPT-3 memory: 1.2TB to 350GB; query/value r=4 checkpoint: 350GB to 35MB, with base model still required. Prose says 25% speedup; footnote gives 32.5 vs 43.1 tokens/s/V100 using equal numbers of weight shards. Section 5 specifies Tesla V100.
Go to primary source ↓ev-latencyPDF p. 4, Table 1 and caption; p. 17, Appendix B and Figure 5
Single forward pass on Quadro RTX8000, averaged over 100 trials. Batch1/length128: Fine-Tune/LoRA 19.8±2.7 ms; AdapterL 23.9±2.1 (+20.7%); AdapterH 25.8±2.2 (+30.3%). Table's |Θ| counts adapter parameters. Figure 5 varies batch, length and bottleneck.
Go to primary source ↓ev-gluePDF p. 6, Table 2, RoBERTa-base and DeBERTa-XXL rows and caption
RoBERTa-base averages: FT 86.4 (125M), LoRA 87.2 (0.3M). DeBERTa-XXL: FT 91.1 (1500M), LoRA 91.3 (4.7M). Metrics are MNLI overall accuracy, CoLA Matthews correlation, STS-B Pearson correlation and other-task accuracy. Asterisks indicate imported baselines.
Go to primary source ↓ev-glue-protocolPDF p. 7, Sections 5.2–5.3; p. 18, D.1–D.2; p. 19, Table 9; p. 20, Table 10
AdamW, linear decay, best-epoch selection, five-seed medians. Standard MRPC/RTE/STS-B initialize LoRA from MNLI; dagger RoBERTa starts from pretrained model and length128. Table9 lists dagger LoRA batch4 versus adapter batch32. Tables9–10 specify query/value rank8 and model/task-specific hyperparameters.
Go to primary source ↓ev-gpt2-resultPDF p. 7, Table 3, GPT-2 M and L rows and caption; Section 5.4
GPT-2 M LoRA: 0.35M, BLEU70.4±0.1, NIST8.85±0.02, MET46.8±0.2, ROUGE-L71.8±0.1, CIDEr2.53±0.02. FT BLEU68.2 at354.92M; PreLayer69.7 at0.35M, both imported. GPT-2 L LoRA BLEU70.4±0.1 at0.77M; FT68.5. Caption calls intervals confidence intervals without level.
Go to primary source ↓ev-dataPDF pp. 17–18, Appendix C, GLUE/WikiSQL/SAMSum/E2E/DART/WebNLG paragraphs
WikiSQL uses schema/question→SQL and 56,355/8,421 train/validation examples. SAMSum lists14,732/819 train/test conversations/summaries. E2E restaurant slot-value generation has roughly42,000/4,600/4,600 train/validation/test examples. WebNLG distinguishes seen and unseen categories.
Go to primary source ↓ev-gpt2-protocolPDF p. 19, Appendix D.3; p. 20, Table 11
GPT-2: mean of3 seeds, best epoch, AdamW,5epochs,batch8,500warmup steps,linear schedule,LR0.0002,rq=rv=4,α32. E2E weight decay0.01,dropout0.1,label smoothing0.1. Beam10,length penalty0.9(E2E)/0.8(WebNLG,DART),no-repeat4-gram.
Go to primary source ↓ev-gpt3-resultPDF p. 8, Table 4 and caption; Section 5.5
GPT-3 FT WikiSQL73.8/MNLI89.5/SAMSum52.0,28.0,44.5; LoRA4.7M73.4/91.7/53.8,29.8,45.9; LoRA37.7M74.0/91.6/53.4,29.2,45.1. AdapterH40.1M MNLI91.5. Task-level typical seed variation, not per-entry uncertainty:0.5/0.1 percentage points and ROUGE0.2,0.2,0.1.
Go to primary source ↓ev-gpt3-protocolPDF pp. 19–20, Appendix D.4; p. 21, Table 12
GPT-3 AdamW,2epochs,batch128,weight decay0.1; lengths384/768/2048; best validation performance. Table12:250,000warmup tokens,linear LR schedule,LoRA LR2e-4. D.4 describes two parameter budgets but associates all-four rank2 with37.7M, unlike Table15.
Go to primary source ↓ev-allocationPDF p. 10, Section 7.1, Table 5, its caption and following paragraph
Across96layers with approximately18M parameters, query r8=70.4/91.0 WikiSQL/MultiNLI; query+value r4=73.7/91.3; all-four r2=73.7/91.7. Caption/text call query+value best overall, which is not the literal ordering of the reported cells.
Go to primary source ↓ev-rankPDF p. 10, Section 7.2, Table 6 and footnote 6
Query+value ranks1/8/64: WikiSQL73.4/73.8/73.5; MultiNLI91.3/91.6/91.4. Query-only rank1=68.8/90.7. All-four rank4 WikiSQL74.0. Authors warn small ranks may not suffice for every task, including a different downstream language.
Go to primary source ↓ev-gpt3-detailsPDF p. 23, Appendix F.2, Table 15, LoRA rows
Value-only rv2:4.7M,73.4/91.7; query/value r1:4.7M,73.4/91.3. Query/value r8:37.7M,73.8/91.6; all-four r4:37.7M,74.0/91.7. All-four r2:18.8M,73.7/91.7. Query/value r64 WikiSQL73.6, differing from Table6's73.5.
Go to primary source ↓ev-subspacePDF p. 11, Section 7.2, Equation (4), Figure 3; p. 12, Figure 4; p. 25, Figure 6; p. 26, Figure 7
Normalized overlap compares leading singular-vector subspaces across ranks8/64 and seeds; Gaussian matrices provide a comparison. Main plots use layer48; appendix covers layers1,32,64,96. Authors interpret leading overlap as useful shared directions. Figure3 argument order differs from Eq4 while its axes span i≤8 and j≤64.
Go to primary source ↓ev-featurePDF p. 12, Section 7.3 and Table 7; p. 24, H.3; p. 25, H.4; p. 26, Figure 8
Projected-weight norms compare update, pretrained and random directions. Authors interpret updates as amplifying directions not dominant in pretrained weights; these are matrix diagnostics and an interpretation, not direct semantic feature interventions.
Go to primary source ↓ev-subspace-notationPDF p. 11, Equation (4) and right-singular-vector definition; p. 22, Appendix G; pp. 23–24, projection metric and equality
Section7.2 uses right singular vectors of A; AppendixG says left singular matrices. G defines d²=p−sum(σ²) but prints sum(σ²)/p=(1−d²)/p, algebraically inconsistent for p≠1. Equation4's normalized overlap is independently specified.
Go to primary source ↓ev-gpt2-rankPDF p. 24, Appendix H.2; p. 26, Table 18 and caption
Separate GPT-2 medium E2E rank study after26,000steps: rank1 BLEU68.72/loss1.23; rank4 BLEU70.38/loss1.18; rank16 loss1.16, tied at displayed precision by ranks32–512. Caption acknowledges hyperparameters partly tuned at rank4.
Go to primary source ↓ev-appendix-experimentsPDF pp. 20–21, Appendix E; pp. 21–24, F.1–F.3, Tables 13–17
Additional DART/WebNLG comparisons, LoRA combined with prefix methods, and MNLI subsampling experiments are supplied. Table14 separates unseen/seen/all categories. Table16 uses full validation set after training on subsets; Table17 supplies separate schedules.
Go to primary source ↓ev-futurePDF pp. 12–13, Section 8
Authors identify combinations, adaptation mechanism, principled placement and possible pretrained-weight rank deficiency as open directions.
Go to primary source ↓8.5 Primary sources
LORA: L OW-R ANK ADAPTATION OF LARGE LAN- GUAGE MODELS ↗
PDF · 13,747 extracted words
Source fingerprint
e9a0d3128767db616085dc0f4e6e455e672e89af823e8ed1282793682787395a