Gemma 3 Technical Report
1. Paper overview
In one sentence: Gemma 3 expands visual and language capabilities by compressing image representations and restricting most attention to local windows, while long-context task reliability and full training reproducibility remain limited. architecturevisionpretrainingposttrainingcacheit-benchmarkspan-and-scanlong-results
| At a glance | What to know |
|---|---|
| Research problem | Source description The design problem is to add image understanding, multilingual coverage and longer context to consumer-oriented open models without letting the inference KV cache dominate memory. The family spans 1B–27B; the 1B model is text-only and has the shorter context limit. architecture |
| Core mechanism | Source description A 5:1 local/global attention pattern and short local windows reduce cached history, supported by text-only perplexity probes and a computed memory comparison. attention-ratiocache |
| A key reported result | Zero-shot MATH with instruction-tuned models: Gemma 3 27B IT: 89.0 Accuracy (%). Table 6 IT evaluation; sampled zero-shot answers, no COT mark or normalization in Table 21. Dataset split is not identified. Gemma 2 27B: 55.6; Gemini 1.5 Pro: 86.5; Gemma 3 4B: 75.6. A 33.4-percentage-point gain over Gemma 2 27B, calculated from reported scores. This does not isolate post-training from architecture/data changes; uncertainty is unreported. it-benchmarksit-protocol |
| Reading caution | Reader analysis The authors acknowledge residual probe contamination risk. Final-model comparisons change multiple ingredients; small text-only attention ablations cannot establish multimodal long-context quality. Appendix transfer evaluations also use different protocols from IT evaluation. attention-ratioit-benchmarksappendix-comparisonsprobe-contamination |
Core contributions
- Source description
A 5:1 local/global attention pattern and short local windows reduce cached history, supported by text-only perplexity probes and a computed memory comparison. attention-ratiocache
- Source description
A shared vision encoder compresses images into soft tokens; optional Pan & Scan restores local detail at additional inference cost. visionpan-and-scan
- Reader analysis
Distillation and an improved post-training recipe yield broad IT gains. The authors' general capability claim should be assessed task by task rather than treated as uniform superiority. pretrainingposttrainingit-benchmarks
Figure 6. Restricting most layers to local history slows growth of the KV cache. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the legend: L:G names the local-to-global layer ratio, and sw is the local sliding-window size in tokens. Both curves describe 2B configurations. The red curve uses five local layers for each global layer and sw=1024; the yellow comparator uses global attention throughout. Read the vertical axis as KV-cache memory alone, in MB, and follow the context labels across the horizontal axis. Section 2 supplies the architectural explanation: local layers need only their bounded window, while global layers retain access to long context. Section 5.2 describes this plot as a computed memory comparison, so it should be read separately from accuracy and runtime measurements. architecturecacheattention-ratiowindow-labels
What it supports. The widening gap supports the intended cache-saving effect of the attention pattern. It gives a concrete reason for retaining relatively few global layers when extending context. This diagnostic complements Figure 3's perplexity comparison; it does not by itself establish that long-range information is used successfully.
Where the evidence stops. This is a 2B architecture diagnostic, not a released 2B Gemma 3 checkpoint. The y-axis excludes model weights. Its explicit 5:1 legend is consistent with Section 2; adjacent Figures 4–5 use inconsistent 3:1/1:3 notation.
2. Motivation
2.1 The problem and the proposed response
The design problem is to add image understanding, multilingual coverage and longer context to consumer-oriented open models without letting the inference KV cache dominate memory. The family spans 1B–27B; the 1B model is text-only and has the shorter context limit. architecture
2.2 What this reading follows
A compact multimodal model has two distinct memory problems: representing images economically and retaining an expanding conversation. Gemma 3 addresses the first with a frozen SigLIP encoder and pooled image tokens, and the second with five local attention layers for each global layer. Distillation and instruction tuning then build the model's answering capabilities. Read the six source excerpts as a sequence of separate tests: memory scaling, attention quality, quantization, final-model accuracy, image-detail recovery and long-context reliability. The strongest results do not settle every question. In particular, a supported context length is different from consistently using its contents, and one vision ablation has conflicting checkpoint labels. architecturevisionpretrainingposttrainingcacheit-benchmarkspan-and-scanlong-results
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Foundational work |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
This table preserves the labels recorded at reading time. The current major category is Components of WAMs. View the current classification.
3.1 Evidence-based assessment
Supports the recorded classification
The recorded foundational Language & VLM backbone classification fits the architecture and evaluations. WAM architecture, prediction paradigm and quadrant are not applicable: multimodal text generation and training-time RL do not establish joint future/action prediction, inverse dynamics or executed control. A unified decoder is insufficient evidence for the WAM One Model category. architectureposttrainingvideo
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
5. Method in detail
5.1 Turn image detail into language-model inputs
Begin with the encoder's fixed input size. Gemma 3 resizes an image to 896×896, extracts SigLIP representations and applies average pooling so the decoder receives 256 soft image tokens. The encoder is shared across the 4B, 12B and 27B models and stays frozen while the language model trains; precomputing its outputs avoids running it in each language-model training step. This compression is economical, but Section 2.1 explains that resizing can erase small objects or make text unreadable. Pan & Scan changes the inference input by dividing the image into equal-sized, non-overlapping crops covering the whole image, each resized for encoding. It can be disabled for speed. Table 8 evaluates its benefit, but its conflicting PT/IT descriptions leave the exact checkpoint unresolved. The numerical activation thresholds and maximum crop count are not specified. visioncomputepan-and-scan
5.2 Separate stored context from usable context
The attention design allocates long-range access selectively. Five local layers use a short window for every global layer that can attend across the context. Figure 6 illustrates the corresponding reduction in stored keys and values, and Figure 3 asks whether making global layers rarer damages text validation perplexity. Extending the positional range is another operation: the larger models start at 32K and move to 128K near the end of pre-training with RoPE rescaling. Reader interpretation: these are three different tests—memory cost, language-model fit and task-level use of distant information. Passing the first two does not imply passing the third. Table 15 makes that distinction concrete through the 27B IT RULER drop from 91.1 to 66.0. An application relying on distant details must therefore test its task, beyond checking the advertised context limit. architecturecacheattention-ratiolong-traininglong-results
5.3 Trace the supervision before attributing the gains
During pre-training, the student learns from a teacher distribution restricted to 256 sampled logits per token, with unsampled targets zeroed and the remaining probabilities renormalized for cross-entropy. Post-training then introduces a large IT teacher and RL feedback from several sources, including human preferences, executable code and math answers. These are training signals; the report's inference examples and benchmarks generate text. Reader interpretation: the final MATH improvement is strong evidence about the complete IT system, but cannot identify how much each ingredient contributed. Figure 8 reinforces the need to specify training budget: its preference for smaller or larger teachers reverses with horizon. The undisclosed teacher identities and detailed schedules prevent exact replication from the report alone. Keep PT probes, final IT evaluation and supervised transfer results in separate comparisons. pretrainingposttrainingit-benchmarksteacherappendix-comparisons
5.4 Training and inference
During training
Pre-training consumes 2T, 4T, 12T and 14T tokens for 1B, 4B, 12B and 27B. The mixture adds multilingual monolingual/parallel data, with filtering, decontamination and quality reweighting. Exact corpus proportions are absent. pretraining
All sizes use knowledge distillation: sample 256 logits per token using teacher probabilities, set unsampled target probabilities to zero, renormalize, then train the student with cross-entropy. The teacher-size probe reverses its preference as the training horizon grows. pretrainingteacher
Instruction tuning combines distillation from a large IT teacher with RL based on improved BOND, WARM and WARP. Rewards draw on human feedback, code execution and math answers. QAT separately uses non-quantized checkpoint probabilities, typically for 5000 steps. posttrainingquantization
During inference
Supply the actual BOS token and correct conversation delimiters, then generate text autoregressively. PT ends with eos; IT ends with end_of_turn. Pan & Scan optionally covers an image with non-overlapping equal-sized crops, resizing each for the frozen encoder; disabling it reduces processing. formatvision
RL and code-execution feedback belong to training. The evaluated inference interface answers questions from text/images or sampled video frames; it does not establish a feedback controller, action-conditioned future model or robot execution. posttrainingarchitecturevideo
5.5 Implementation flow
- Encode observations
The shared SigLIP variant is adapted on visual-assistant tasks, then frozen during language-model training. It accepts 896×896 images and uses 4×4 average pooling to produce 256 image tokens. These enter the language decoder; the report does not detail the connecting projection. visionarchitecture
- Process mixed context
The decoder uses grouped-query attention, RMSNorm and QK-norm. Five local layers precede each global layer, starting locally. Local windows span 1024 tokens; global layers attend across the longer context. architecture
- Extend positional coverage
Pre-training begins at 32K. Near its end, 4B/12B/27B extend to 128K with RoPE rescaling factor 8. Global RoPE base frequency is 1M, versus 10k locally. The 1B limit remains 32K. long-trainingarchitecture
6. Experiments & results
Gemma 3 combines a distilled language decoder, pooled SigLIP image tokens and mostly local attention to expand visual and long-context capabilities within a smaller inference-memory budget. Strong instruction-tuned math results coexist with substantial long-context accuracy loss and incomplete training disclosure (architecture, vision, it-benchmarks, long-results).
The supplied report has no architecture block diagram or encoder-to-decoder schematic. Its method is described in Sections 2–3 and diagnosed through experiments. The mechanism visual therefore uses the original Figure 6 memory comparison, complemented by Figure 3's attention ablation. All six excerpts are source figures or quantitative tables; none reconstructs an absent diagram. visual-scopearchitecturecacheattention-ratio
6.1 Read the original evidence
Table 3. Quantizing weights leaves the context cache as a substantial memory cost. Original paper, p. 3 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Choose a model size before comparing columns. Its first row gives the weight footprint; the following +KV row gives the total including the cache. The caption fixes the cache at 32768 context tokens and 8-bit quantization, so these are not 128K totals. The columns distinguish bf16, per-channel Int4, block-32 Int4 and switched fp8. For the 4B example, follow bf16 from 8.0 GB to 12.7 GB with the cache, then Int4 from 2.6 GB to 7.3 GB. Section 2.3 explains that the quantized checkpoints are produced by QAT against probabilities from their non-quantized counterparts. quantizationarchitecturecache
What it supports. The 4B Int4 weights occupy 2.6 GB, but the corresponding +KV total is 7.3 GB. This shows why a weights-only deployment estimate can be misleading even after quantization. The local-attention design and weight quantization address different contributors to the reported memory footprint.
Where the evidence stops. The table supplies component memory totals, not peak runtime allocation, speed, or task-quality changes after quantization. Its fixed 8-bit cache setting must be preserved in comparisons; the listed formats do not prove compatibility or capacity on a particular device.
Table 6. Strong math gains coexist with task-specific gaps to comparison models. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the grouped column headings before following a row: these are final instruction-tuned models, unlike the pre-training radar plot elsewhere on the same page. The MATH row gives Gemma 3 27B 89.0, Gemma 2 27B 55.6 and Gemini 1.5 Pro 86.5; Gemma 3 4B reaches 75.6. Now move to GPQA Diamond, where Gemma 3 27B has 42.4 versus Gemini 1.5 Pro's 59.1. The caption identifies zero-shot evaluation, and Table 21 identifies MATH accuracy with sampled answers. Treat each row as a distinct benchmark and protocol, without averaging unrelated scales into an invented overall score. it-benchmarksit-protocoltable-discrepanciesposttraining
What it supports. The MATH result demonstrates a large improvement over the same-size Gemma 2 model, and even the smaller 4B model exceeds that baseline in this row. The GPQA comparison limits the broader claim of parity with Gemini 1.5 Pro: competitiveness depends on the ability being measured.
Where the evidence stops. Table 6 and appendix Table 18 disagree on 27B IT LiveCodeBench (29.7 versus 39.0) and HiddenMath (60.3 versus 56.0), without explanation. Those values are not merged. This table reports no uncertainty or controlled attribution of gains to individual training changes.
Table 15. Longer supported context still reduces performance on the reported long-context tasks. Original paper, p. 22 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. First separate the PT columns, meaning pre-trained, from the IT columns, meaning instruction-tuned. Within a model column, compare the 32K and 128K rows for the same benchmark. RULER uses zero-shot sampled accuracy according to Table 19; MRCR uses its own score and few-shot sampling. For 27B IT, RULER falls from 91.1 to 66.0, whereas MRCR moves from 63.2 to 59.3. Compare PT and IT only after fixing context length: at 128K the 27B PT RULER value is 72.9, above the IT value. Neither a larger model nor instruction tuning produces a uniform ranking throughout this table. long-resultslong-protocollong-training
What it supports. The 25.1-percentage-point RULER drop for 27B IT shows a substantial difference between accepting 128K tokens and exploiting them reliably. The smaller MRCR change also shows that context sensitivity depends on the task. Cache efficiency and long-context answer quality therefore need separate evaluation.
Where the evidence stops. No uncertainty, detailed sample counts or exact MRCR shot count are supplied. These rows compare context conditions, not a controlled ablation of local attention alone; they cannot identify which architectural or training component caused the loss.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Zero-shot MATH with instruction-tuned models Table 6 IT evaluation; sampled zero-shot answers, no COT mark or normalization in Table 21. Dataset split is not identified. | Gemma 3 27B IT: 89.0 Accuracy (%) | Gemma 2 27B: 55.6; Gemini 1.5 Pro: 86.5; Gemma 3 4B: 75.6. A 33.4-percentage-point gain over Gemma 2 27B, calculated from reported scores. This does not isolate post-training from architecture/data changes; uncertainty is unreported. it-benchmarksit-protocol |
| Human chat preference Blind side-by-side Chatbot Arena; preliminary results received March 8, 2025; excludes vision. | Gemma 3 27B IT: 1338 (+8/−9) Elo with reported 95% CI | Gemma 2 27B IT: 1220 (+3/−2). Stronger preference in this historical evaluation pool, not a current leaderboard or visual-ability score. arena |
| Pan & Scan on InfographicVQA Table 8 four-shot validation; caption says PT checkpoint, nearby prose says 27B IT. Checkpoint status remains inconsistent. | 27B with P&S: 76.4 ANLS score, reported 0–100 scale | Without P&S: 59.4; reported gain: 17.0 score points. Supports the usefulness of image detail in this comparison. Do not conflate it with final IT test-set results or silently resolve the checkpoint conflict. pan-and-scanvision-protocolpt-visionit-vision |
| RULER across context lengths Table 15; zero-shot sampled evaluation per Table 19, at 32K and 128K. Benchmark sample counts are unspecified. | 27B IT: 91.1 at 32K; 66.0 at 128K Accuracy (%) | 27B PT: 85.9 at 32K; 72.9 at 128K. IT loses 25.1 percentage points with the longer context; nominal context support does not establish uniform retrieval quality. long-resultslong-protocol |
| 4B quantized inference memory Table 3 footprint accounting at 32768 context tokens with an 8-bit KV cache. | Int4: 2.6 for weights; 7.3 including KV cache Memory (GB) | bf16: 8.0 for weights; 12.7 including KV cache. Quantized weights reduce the footprint, but the cache remains material. These are reported component totals, not measured throughput or guaranteed device capacity. quantization |
| Perception Test video question answering Table 17: zero-shot MCVQA, 16 linearly spaced frames; split not identified. | 27B IT: 58.1 Top-1 accuracy (%) | 4B IT: 50.6; 12B IT: 54.9. Evidence for answering questions about observed video, not predicting video or executing actions. No uncertainty is provided. video |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 3. The local/global ratio changes little in these text-only perplexity probes. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the x-axis literally as Local:Global. The four settings progressively reduce the fraction of layers that see the full context, while the red and yellow curves track the 2B and 9B text-only experiments. The vertical axis is a change in perplexity, not an absolute perplexity score or downstream task accuracy; the dashed line marks zero change. The values remain close to that reference across the tested ratios. Section 5.2 identifies 1:1 with Gemma 2 and 5:1 with Gemma 3. This is the quality-side check to pair with the memory-side evidence in Figure 6, rather than an independent evaluation of all final models. attention-ratiocachelong-results
What it supports. The plotted comparison supports the authors' claim that increasing the local/global ratio has little effect on this validation objective. In particular, the selected 5:1 setting does not show a large perplexity penalty in either probe. This makes the memory-saving choice plausible within the tested text-only setting.
Where the evidence stops. No error bars, seed variation, absolute perplexities or detailed validation-set description accompany this plot. Small average perplexity changes cannot establish equivalent visual reasoning, retrieval across 128K tokens or robustness to specific long-range dependencies.
Table 8. Extra image crops improve the reported document and infographic scores. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each model's two rows as a paired comparison, then use its blue delta row to see the gain. P&S means Pan & Scan: Section 2.1 describes an inference-only operation that covers an image with equal-sized non-overlapping crops before resizing them for the encoder. The table caption specifies four-shot validation. Table 20 names ANLS for DocVQA and InfographicVQA and accuracy for TextVQA, so their deltas are not interchangeable measures. Checkpoint identity needs special care: this caption calls the checkpoint pre-trained, while the adjacent paragraph calls the 27B model IT. The baseline values also match the pre-trained, no-P&S Table 11. pan-and-scanvisionvision-protocolpt-visionit-vision
What it supports. For 27B, InfoVQA rises from 59.4 to 76.4, a gain of 17.0 score points; DocVQA gains 4.8 and TextVQA gains 1.6. The direction supports using additional image detail for these tasks, with especially large gains on the infographic benchmark in this reported comparison.
Where the evidence stops. The PT/IT disagreement is unresolved in the supplied version. Preserve the table's numbers with that qualification, and do not compare them directly with final IT test-set scores. No crop-count, latency or compute-matched control is reported here.
7. Analysis & limitations
7.1 What the evidence leaves open
The authors acknowledge residual probe contamination risk. Final-model comparisons change multiple ingredients; small text-only attention ablations cannot establish multimodal long-context quality. Appendix transfer evaluations also use different protocols from IT evaluation. attention-ratioit-benchmarksappendix-comparisonsprobe-contamination
Internal inconsistencies remain: Table 8 PT versus IT; Figure 4's 3:1 legend versus 1:3 caption; vocabulary 256k versus 262k; differing Table 6/18 LiveCodeBench and HiddenMath scores. Nearby Arena prose also disagrees with Table 5's Llama-405B value. Conflicting results are not merged. pan-and-scanwindow-labelsvocabulary-discrepancytable-discrepanciesarena
Privacy testing uses 50-token prefixes/suffixes and exact or approximate matching. No detected personal information among memorized outputs is detection-bounded. Safety evaluations describe low violations qualitatively without numerical rates, precluding a quantitative assurance comparison. privacysafety
7.2 Questions for discussion
- Which long-context tasks reveal losses that validation perplexity fails to expose?
- How much of Pan & Scan's benefit survives a latency-matched comparison, and which checkpoint generated Table 8?
8. Reproducibility audit
8.1 Requirements and known gaps
Full training requires the data mixture, teacher distributions, encoder and large TPU infrastructure. Table 2 specifies 512/2048 TPUv5e chips for 1B/4B, 6144 TPUv4 for 12B and 6144 TPUv5p for 27B. Precomputed image embeddings remove encoder execution from language-model training steps. computepretrainingvision
Teacher identities, full optimizer/schedule settings, reward weights, corpus proportions, software versions and numerical Pan & Scan thresholds/crop limits are unspecified. Jax/Pathways/GSPMD/XLA names alone do not supply a runnable recipe. computepretrainingposttrainingvision
Proposed smaller checks are to compare attention ratios under matched training and long-context evaluation, and to toggle Pan & Scan on one fixed checkpoint while measuring ANLS and latency by image shape. These require explicit checkpoint and prompting controls. attention-ratiolong-resultspan-and-scanvision-protocol
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Does the perplexity result predict long-context reliability?
Reader-proposed experiment, not run: train matched text-only 2B students with L:G=1:1 versus 5:1, both using a 1024-token local window. Hold data order, teacher targets, token budget, RoPE settings and optimizer schedule fixed, and use multiple seeds. Compare validation perplexity, directly allocated KV memory, and RULER accuracy at 32K and 128K with identical prompts and examples. Pre-register an acceptable accuracy difference. If perplexity remains similar and memory falls but long-context accuracy breaches that tolerance, the Figure 3 proxy would not establish task equivalence. Changing attention masks only at inference would be a different experiment and should not stand in for this training comparison. attention-ratioarchitecturecachelong-traininglong-resultslong-protocol
Check 2: Is Pan & Scan buying detail or simply more compute?
Reader-proposed experiment, not run: choose one verified 4B checkpoint and explicitly record PT or IT status before comparing with Table 8. On fixed DocVQA and InfographicVQA validation examples, hold the four-shot prompts and decoding constant and compare P&S off, P&S on, and a control that repeats the globally resized image to match the visual-token count. Record crop count, ANLS, peak memory and latency, stratified by aspect ratio and text size. Gains concentrated on small-text or non-square images and exceeding the repeated-image control would support the detail-recovery explanation. Similar gains from repetition, or no advantage under a fixed latency budget, would weaken that interpretation. visionpan-and-scanvision-protocolpt-vision
8.3 Reading coverage
Visual audit: Visually inspected the title/byline/version, contributor credits, all original Figures 1–9 and Tables 1–21, and the surrounding method, training, evaluation and appendix protocol pages. All six final original crops were separately viewed with readable axes, legends and complete table bodies. Figure 6's L:G=5:1 and sw=1024 match the architecture text; Figure 4's ratio-label conflict, Table 8's PT/IT conflict, vocabulary notation and cross-table result discrepancies remain disclosed. Reference-only pages 12–16 were read in the complete text but were not part of the image pass. Separate supplements, code and checkpoints were not inspected.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 17, 18, 19, 20, 21, 22, 23, 24, 25. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Title and abstract; Sections 1–2.4 (problem, architecture, vision, pre-training, QAT, infrastructure)
- Section 3 (instruction tuning and formatting); Sections 4.1–4.2 (evaluation)
- Sections 5.1–5.5 (all ablations); Section 6 (memorization/privacy)
- Sections 7.1–7.4 (responsibility, safety/security); Section 8 (conclusion)
- References, pp. 11–16; contributor credits, pp. 17–19
- Complete appendix, pp. 20–25: PT, transfer, multilingual, long-context, IT, video and evaluation protocols
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Identity notes: exact title and identifier match. The inspected artifact is arXiv:2503.19786v1, dated 25 March 2025 in the margin, with a 12 March 2025 report header. No later revision was supplied or compared.
- The title-page group author is Gemma Team, affiliated with Google DeepMind. The catalog flattens the credited roles and splits the single name Dmitry (Dima) Lepikhin into two entries. Metadata retains the verified group byline; the catalog remains unchanged.
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout. This acquisition omission was addressed by visually inspecting all scientific figure/table pages and every final crop.
- Separate supplemental material availability has not been fully verified.
- All six supplied text chunks were read individually. Reference-only pages 12–16 were read as text; visual inspection covers pages 1–11 and 17–25.
- Code, released checkpoints and external documentation were not inspected; no experiments were reproduced.
- The report supplies no formal method equations. No mathematical objective or undocumented implementation details have been reconstructed.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
identityPDF p. 1, title, byline, header and arXiv margin; pp. 17–19, contributor credits
The title is Gemma 3 Technical Report, credited to Gemma Team, Google DeepMind. The header date is 2025-03-12; the margin identifies arXiv:2503.19786v1, 25 Mar 2025. Credits include Dmitry (Dima) Lepikhin as one technical advisor.
Go to primary source ↓architecturePDF pp. 1–2, Introduction and Section 2; p. 2, Table 1
Decoder-only transformer with GQA, RMSNorm and QK-norm; five local layers per global layer and a 1024-token local window. Context is 128K except 32K for the text-only 1B model. Table 1 lists a 417M vision encoder for 4B, 12B and 27B.
Go to primary source ↓visionPDF p. 2, Section 2.1; p. 8, Section 5.5, Impact of image resolution
The shared SigLIP variant is adapted on visual-assistant tasks and frozen during language-model training. It processes 896×896 images. Average pooling, 4×4 at this resolution, reduces each encoded image to 256 soft tokens. Inference-only Pan & Scan uses equal-sized, non-overlapping crops covering the image; its trigger and crop limit are not numerically specified.
Go to primary source ↓pretrainingPDF p. 3, Section 2.2, Training data, Tokenizer, Filtering and Distillation
Token budgets are 2T/4T/12T/14T for 1B/4B/12B/27B. Multilingual monolingual and parallel data are added. Distillation samples 256 logits per token according to teacher probabilities, zeros other targets and renormalizes for cross-entropy. Data filtering, decontamination and quality reweighting are described.
Go to primary source ↓posttrainingPDF p. 4, Section 3, Techniques, Reinforcement learning objectives and Data filtering
Post-training uses a large IT teacher and RL based on improved BOND, WARM and WARP. Feedback includes human-trained weight-averaged reward models, code execution and math ground truth. Detailed teacher identities, reward weights and schedules are not given.
Go to primary source ↓formatPDF p. 4, Table 4 and Section 3, BOS token and PT versus IT Formatting
Both model types require the BOS token ID. IT uses start_of_turn, user/model roles and end_of_turn; PT ends with eos. The literal text [BOS] does not produce the BOS token.
Go to primary source ↓computePDF p. 3, Table 2 and Section 2.4; p. 4, continuation of Section 2.4
Training uses 512 TPUv5e chips for 1B, 2048 TPUv5e for 4B, 6144 TPUv4 for 12B and 6144 TPUv5p for 27B. Image embeddings are precomputed. The stack includes ZeRO-3-style sharding, Jax, Pathways, GSPMD and MegaScale XLA; versions are not specified.
Go to primary source ↓quantizationPDF p. 3, Section 2.3 and Table 3, 4B and +KV rows
QAT typically runs 5000 steps using non-quantized checkpoint probabilities. For 4B, bf16/int4 weights occupy 8.0/2.6 GB; with a 32768-token, 8-bit KV cache the totals are 12.7/7.3 GB. Other formats are block-int4 and SFP8. This is footprint accounting, not a latency benchmark.
Go to primary source ↓arenaPDF p. 5, Table 5, Gemma-3-27B-IT and Gemma-2-27B-it rows and caption; Section 4.1 continuation
Blind human comparisons give preliminary March 8, 2025 Elo scores of 1338 (+8/−9, 95% CI) and 1220 (+3/−2). Vision is excluded. The table lists Llama-3.1-405B as 1269 while nearby prose assigns 1257 to LLaMA 3 405B.
Go to primary source ↓it-benchmarksPDF p. 6, Table 6, MATH, MMLU-Pro, GPQA Diamond, SimpleQA and MMMU rows
Zero-shot IT evaluation reports Gemma 3 27B MATH 89.0, versus Gemma 2 27B 55.6 and Gemini 1.5 Pro 86.5. Gemma 3 4B MATH is 75.6. Gemma 3 27B MMLU-Pro/GPQA Diamond are 67.5/42.4 versus Gemini 1.5 Pro 75.8/59.1; MMMU validation is 64.9 versus 65.9.
Go to primary source ↓it-protocolPDF p. 25, Table 21, MATH, HumanEval and LiveCodeBench rows and caption
IT MATH uses accuracy, sampling and zero-shot prompting, with no COT mark and no normalization. HumanEval is zero-shot pass@1. LiveCodeBench is zero-shot with COT and an average over eight samples.
Go to primary source ↓attention-ratioPDF p. 6, Figure 3 and Section 5.2, Local:Global ratio
Text-only 2B and 9B ablations vary L:G from 1:1 through 3:1, 5:1 and 7:1; plotted changes in validation perplexity remain small. The plot provides no error bars or absolute perplexities.
Go to primary source ↓window-labelsPDF p. 7, Figures 4–5 and Section 5.2 continuation
Figure 4's legend labels 2B L:G=1:1 and L:G=3:1, but its caption says 1:1 and 1:3 local-to-global ratios. Figure 5 also labels configurations 1:3. Their ratio notation is not consistent with Figure 4's legend.
Go to primary source ↓cachePDF p. 7, Figure 6, caption and Section 5.2, Impact on KV cache memory
The computed KV-cache footprint of a 2B architecture with L:G=5:1 and sw=1024 grows much more slowly across 1K–128K context than the 2B global-only comparator. The graph measures KV memory in MB, excluding weight memory.
Go to primary source ↓long-trainingPDF p. 7, Section 5.3 and Figure 7
Training starts at 32K and extends the 4B/12B/27B models to 128K near the end of pre-training. RoPE rescaling factor 8 is reported; global/local base frequencies are 1M/10k. The authors warn of degradation when continuing to scale.
Go to primary source ↓teacherPDF p. 8, Figure 8 and Section 5.4 continuation
In the teacher-size ablation, the smaller teacher performs better at short training horizons; longer training reverses that trend. Negative plotted relative perplexity differences favor the larger teacher. Teacher sizes and identities are not disclosed.
Go to primary source ↓resolutionPDF p. 8, Table 7, all rows and caption
A short-schedule 2B probe compares encoder resolutions 256, 448 and 896 with 256 image tokens. DocVQA scores rise from 31.9 to 45.4 to 59.8; the other displayed tasks also improve. This is separate from final-model evaluation.
Go to primary source ↓pan-and-scanPDF p. 8, Table 8, all rows and caption; Section 5.5, Pan & Scan paragraph
Four-shot validation comparison: 4B DocVQA/InfoVQA/TextVQA goes from 72.8/44.1/58.9 to 81.0/57.0/60.8. For 27B it goes from 85.6/59.4/68.6 to 90.4/76.4/70.2. The caption says pre-trained checkpoint, while nearby prose calls the 27B model IT.
Go to primary source ↓vision-protocolPDF p. 25, Table 20, DocVQA, InfographicVQA and TextVQA rows and caption
The vision protocol uses four-shot sampled outputs: ANLS for DocVQA and InfographicVQA, accuracy for TextVQA; no COT or normalization.
Go to primary source ↓pt-visionPDF p. 20, Table 11, DocVQA, InfoVQA and TextVQA rows and caption
Pre-trained validation scores without P&S match the no-P&S rows in Table 8. This supports the caption's PT interpretation but does not resolve its conflict with the prose.
Go to primary source ↓long-resultsPDF p. 22, Table 15, RULER and MRCR rows, all model columns
27B IT RULER is 91.1 at 32K and 66.0 at 128K; 27B PT is 85.9 and 72.9. MRCR for 27B IT is 63.2 and 59.3. At 128K, 12B PT RULER is 80.7, above 27B PT's 72.9.
Go to primary source ↓long-protocolPDF p. 24, Table 19, RULER and MRCR rows
RULER uses accuracy with zero-shot sampling; MRCR uses MRCR score with few-shot sampling. The report does not give an exact MRCR shot count or detailed benchmark sample counts.
Go to primary source ↓videoPDF p. 22, Table 17 and caption; Section 8.2
Video evaluation is zero-shot with 16 linearly spaced frames. Perception Test MCVQA uses top-1 accuracy, scoring 50.6/54.9/58.1 for 4B/12B/27B. ActivityNet-QA uses standard GPT evaluation, with scores 46.3/50.4/52.8.
Go to primary source ↓appendix-comparisonsPDF pp. 20–21, Tables 9–14 and surrounding appendix text
The appendix separates pre-training, supervised multimodal transfer, and multilingual evaluations. Table 12 fine-tunes PT checkpoints without P&S and compares PaliGemma 2 at task-dependent resolutions; this is not the final IT evaluation.
Go to primary source ↓table-discrepanciesPDF p. 6, Table 6; p. 23, Table 18, LiveCodeBench and HiddenMath rows
For Gemma 3 27B IT, Table 6 reports LiveCodeBench 29.7 and HiddenMath 60.3, while Table 18 reports 39.0 and 56.0. The supplied source does not explain these differences.
Go to primary source ↓privacyPDF pp. 8–9, Section 6 and Figure 9
Discoverable extraction uses 50-token prefixes and 50-token suffixes; exact matching and approximate matching within 10% edit distance are distinguished. The authors report reduced memorization and no detected personal information among memorized outputs, explicitly bounded by detection thresholds.
Go to primary source ↓safetyPDF pp. 9–10, Sections 7.1–7.4
The report describes safety filtering, SFT/RLHF and synthetic adversarial queries labeled by human raters, plus closed-ended CBRN knowledge tests. Claims of low violations and low domain knowledge are qualitative, without numerical rates or uncertainty.
Go to primary source ↓visual-scopePDF p. 2, Figure 1; p. 6, Figures 2–3; p. 7, Figures 4–7; p. 8, Figure 8; p. 9, Figure 9
The source illustrates an interaction and empirical diagnostics; it provides no encoder-to-decoder architecture block diagram. Figure 6 is an original architecture-related memory diagnostic.
Go to primary source ↓vocabulary-discrepancyPDF p. 2, Table 1 caption; p. 3, Section 2.2, Tokenizer
The Table 1 caption says 256k vocabulary entries while the tokenizer paragraph says 262k. The supplied PDF does not reconcile the notation.
Go to primary source ↓probe-contaminationPDF p. 6, Section 5.1 continuation below Figure 2
The authors caution that pre-training probes may remain contaminated despite decontamination, limiting definitive conclusions.
Go to primary source ↓it-visionPDF p. 22, Table 16 and caption; Section 8.2, Additional multimodal evaluations
Final IT multimodal evaluations use P&S. Except where marked, Table 16 uses final test sets; MMMU and VQAv2 are validation, and MathVista is testmini. These settings differ from the Table 8 four-shot validation comparison.
Go to primary source ↓8.5 Primary sources
Gemma 3 Technical Report ↗
PDF · 11,124 extracted words
Source fingerprint
b2793d288d6672f8d6a03f65370bb4a48d607f54937a22fc7d9f727435795305