PaliGemma: A versatile 3B VLM for transfer
1. Paper overview
In one sentence: PaliGemma turns pretrained vision and language components into a compact, adaptable VLM through joint input attention and broad training, with task-specific fine-tuning and resolution costs determining the final behavior. e-architecturee-maske-stage1e-transfere-results
| At a glance | What to know |
|---|---|
| Research problem | Source description How can a model below three billion parameters acquire visual skills broad enough to transfer to captioning, question answering, segmentation and short-video understanding? The target is an adaptable base model. Benchmark-friendly behavior is obtained during subsequent transfer, so base-model capacity and a particular specialist's score must be distinguished. e-identitye-architecturee-transfer |
| Core mechanism | Source description A simple image-encoder/linear-projector/language-decoder system supports natural-language answers and serialized spatial outputs through one sequence interface. e-architecturee-structured |
| A key reported result | Document question answering (DocVQA): 43.7 ± 0.5 at 224px; 78.0 ± 0.3 at 448px; 84.8 at 896px. ANLS, higher is better, reported on a 0–100 scale. 44,812 train-plus-validation examples; resolution-specific fine-tuning; ANLS evaluated on the test server. The 896px checkpoint exceeds the 224px result by 41.1 ANLS points. A large resolution-associated gain, with both extra Stage2 training and more image tokens. The central scores are single runs; server-task deviations come from validation, and no 896px deviation is supplied. e-resultse-docvqae-stage2 |
| Reading caution | Reader analysis Resolution gains combine visual information, sequence length and continued training. Figure 9 controls information by downsampling before upsampling, supporting contributions from both detail and additional token computation. Its mixed-metric average should not be read as one task's accuracy. e-stage2e-resolution |
Core contributions
- Source description
A simple image-encoder/linear-projector/language-decoder system supports natural-language answers and serialized spatial outputs through one sequence interface. e-architecturee-structured
- Source description
Systematic ablations examine masking, supervision, frozen components, connector complexity and resolution, while diverse transfers test the usefulness of the resulting checkpoints. e-objectivee-freezinge-connectore-resolutione-results
Figure 1. Two pretrained components meet at a simple token interface. Original paper, p. 2 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the two paths on the left. The photograph enters the green SigLIP encoder and then the linear projection; the question enters the purple SentencePiece tokenizer. Their outputs join in the central token column before entering Gemma's transformer decoder. The answer at the right is a generated text string. The 'contrastive' label describes SigLIP's pretraining origin, not the VLM's suffix-prediction objective. Section 3.1 supplies details omitted from the drawing: the projection starts at zero, image tokens precede text, and the decoder uses a special attention mask. Prompt-dependent changes to image representations happen inside the decoder, after SigLIP encoding. e-architecturee-maske-connector
What it supports. The architecture makes many tasks accessible through the same image-plus-prefix interface. Its connector is deliberately simple: Section 5.5 reports average scores of 77.2 for linear and 77.1 for the tested MLP when all weights are tuned. Extra connector depth is therefore not supported by that comparison.
Where the evidence stops. The example answer illustrates information flow, not a scored evaluation. The diagram contains no future-state rollout, action head or execution loop, so it does not establish a world–action architecture.
2. Motivation
2.1 The problem and the proposed response
How can a model below three billion parameters acquire visual skills broad enough to transfer to captioning, question answering, segmentation and short-video understanding? The target is an adaptable base model. Benchmark-friendly behavior is obtained during subsequent transfer, so base-model capacity and a particular specialist's score must be distinguished. e-identitye-architecturee-transfer
2.2 What this reading follows
PaliGemma asks what a small vision-language model should learn before it becomes a specialist. A SigLIP encoder supplies image tokens, a linear projection connects them to Gemma, and a shared decoder processes the image together with a task prompt. The important design choice is how those tokens interact: the input can attend bidirectionally, while the answer is predicted causally. Broad pretraining then builds skills that later fine-tuning selects and formats. Read the figures in that order—connection, attention, transfer evidence and controlled ablations. The reported capabilities concern understanding and structured perception; their relevance to embodied systems is as a possible backbone, without demonstrated action control. e-architecturee-maske-stage1e-transfere-results
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Foundational work |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
This table preserves the labels recorded at reading time. The current major category is Components of WAMs. View the current classification.
3.1 Evidence-based assessment
Supports the recorded classification
The foundational Language & VLM backbones classification is supported. PaliGemma learns image-conditioned text and structured perception outputs. Its joined vision/language architecture does not establish a world–action One Model category: neither action prediction nor action-conditioned future dynamics is demonstrated. The recorded Not applicable architecture/paradigm/quadrant fields are therefore appropriate for this taxonomy. e-architecturee-structurede-transfere-results
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Follow the question into the image-token block
Begin with a question about an image. SigLIP first encodes the image without seeing that question, and the projection converts its output to Gemma's embedding width. The question then joins those projected tokens inside the decoder. At this point, the attention mask makes a material difference: image positions can read the complete prefix and revise their decoder representations in light of the requested task. Answer positions receive the input context and preceding answer information, but not future targets. The loss supervises the suffix alone. For segmentation, that suffix contains coordinate and mask codes rather than ordinary words. Thus the common interface is sequence prediction, with different interpretations of the resulting tokens; it does not imply that the decoder predicts actions or simulates the environment. e-architecturee-maske-structured
Figure 2. The image–prompt block is bidirectional; answer targets remain hidden until their turn. Original paper, p. 3 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each row as the position asking for context and each column as a position it may inspect, following the caption's row-to-column convention. The columns follow the same token order as the rows. Green checks cover the complete image/prefix block, including BOS and the separator. Thus an image position can attend to the question even though the question follows it in the sequence. Pink prohibitions prevent these input positions from reading answer targets. In the suffix/target rows, permitted answer context grows one position at a time. Section 5.2 adds the separate learning rule: only suffix prediction contributes to the next-token loss. e-maske-objectivee-architecture
What it supports. A decoder-only language backbone need not impose a causal ordering on everything it receives. PaliGemma preserves causal answer generation while allowing its entire input block to interact. This is the specific information-flow hypothesis tested by the masking ablation, rather than evidence that a larger connector is needed.
Where the evidence stops. Attention permission and loss placement are different choices. The drawing shows target alignment, including blocked current targets; it should not be copied as an implementation mask without checking how decoder inputs and prediction labels are shifted.
5.2 Build reusable skills before choosing a specialist
Stage1 is more than a short alignment step. It exposes the entire model to a broad mixture of captioning, OCR, questions and spatial tasks, with a slow vision-encoder warm-up protecting the initial representation. Stage2 then increases the spatial input budget and target density. Only afterward does Stage3 choose a benchmark's output syntax and training recipe. This sequence explains why task indicators can improve pretraining predictability without noticeably changing the average score after transfer: adaptation can compensate for interface differences. It also explains the main table's scope. Most rows measure separately trained specialists, while designated cross-task evaluations reuse a specialist trained elsewhere. A model transferred on VQAv2 and evaluated on MMVP is zero-shot with respect to MMVP, although it has already undergone substantial supervised adaptation. e-stage1e-stage2e-mixturee-objectivee-transfere-resultse-mmvp
5.3 Separate the resolution observation from its explanation
The main table makes the practical observation clear: DocVQA benefits greatly from higher-resolution checkpoints, while some other tasks improve much less. That comparison alone cannot explain the gain, because resolution changes image information, token count and the model's training history together. Figure 9 supplies a more discriminating comparison by retaining the long token sequence while first removing fine image detail. Its intermediate result supports contributions from both information and computation. Reader interpretation: resolution is a task-dependent resource allocation decision, and the right test should report each task's metric alongside cost. The controlled ablation strengthens the mechanism argument but does not turn its mixed-task average into a universal scaling law. Appendix-level variation and the shorter ablation pretraining budget remain part of the conclusion. e-resultse-docvqae-resolutione-stage2e-objective
5.4 Training and inference
During training
Stage0 reuses the pretrained unimodal components. Stage1 updates the whole VLM for one billion examples at 224 pixels and text length 128; a slow SigLIP warm-up limits initially misaligned gradients. Supervision predicts only suffix tokens. e-architecturee-stage1e-mask
Pretraining combines multilingual captioning, OCR, generated VQA/VQG, object questions, detection, segmentation and grounded captioning. The authors exclude transfer datasets and near-duplicate images. Some pretraining data remain private, limiting independent reconstruction of this mixture. e-mixturee-compute
Stage2 continues for 50 million examples at 448 pixels and another ten million at 896, increasing text length to 512 and emphasizing tasks needing fine detail. Stage3 tunes task-specific behavior using validation or minival selection. Full tuning is the default; Appendix J explicitly freezes ViT for some transfers despite the broader main-text wording. e-stage2e-transfere-settings
Training uses big_vision with JAX/GSPMD on TPUv5e and a continuous reciprocal-square-root schedule; transfer uses cosine cooldown. The authors report slightly under three days for Stage1 on TPUv5e-256 and 15 hours for each Stage2. e-compute
During inference
Encode the image and prefix, then generate the suffix autoregressively. Appendix J uses greedy decoding for most tasks and beam search for selected captioning tasks, including beam size two for COCOcap. The output remains a prediction requiring task-specific interpretation. e-architecturee-maske-settingse-coco
Serving cost depends on precision, batch and sharding. Table 5's bfloat16 batch-one measurements give 17 ms prefill and 5.3 ms token extension with Megatron sharding, versus 101 ms and 8.2 ms with FSDP, on eight TPUv3 devices with 512 prefill tokens and a 640-token cache. These are configuration-specific measurements. e-inference
5.5 Implementation flow
- Encode and join modalities
SigLIP ViT-So400m produces image tokens. A zero-initialized linear layer maps them into Gemma-2B v1.0's embedding dimension; SentencePiece tokenizes the text. Fixed square inputs of 224, 448 or 896 pixels produce 256, 1024 or 4096 image tokens, respectively. e-architecture
- Let the input attend jointly
Place image tokens before BOS, prefix and a separately tokenized newline separator. Image and prefix positions attend throughout this input block, allowing decoder-side image representations to depend on the question. Suffix prediction remains causal; the model cannot inspect future answer targets. e-mask
- Represent spatial predictions as tokens
The vocabulary includes 1024 coordinate bins and 128 VQ-VAE mask-code entries. Referring-expression segmentation serializes a box and mask codes. These describe an observed object; no action decoder, environment transition rollout or feedback controller is specified. e-structurede-architecture
- Adapt the interface to the task
Fine-tune the appropriate resolution checkpoint. Multi-image transfer concatenates independently encoded images without special separators; 16 video frames at 224 pixels supply 4096 image tokens. This extends the input context for understanding observed video. e-transfere-video
6. Experiments & results
PaliGemma connects a SigLIP vision encoder to Gemma-2B, then trains the entire VLM to become a useful starting point for task-specific adaptation. Its distinctive choices are bidirectional image–prompt attention, suffix-only prediction loss, broad multimodal pretraining and separate resolution checkpoints. Strong results concern visual understanding and structured visual outputs; the paper does not demonstrate action execution or future-world prediction.
6.1 Read the original evidence
Table 1. Resolution helps unevenly, and each row carries its own evaluation protocol. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read horizontally within a task before comparing any rows. The three columns are checkpoint resolutions, while the task groups contain different metrics and splits. The pale row shading identifies resolution-sensitive tasks. The small corner marker denotes evaluation of a previously transferred model on another dataset, as explained in the original caption: NoCaps, for example, uses the COCOcap specialist. Central values are one randomly selected run from five, rather than five-run means. The displayed deviations describe transfer reruns; for server-dependent evaluations they come from validation splits. Dashes are unreported settings. Appendix B identifies the metric, and Appendix J identifies the specialist's training recipe. e-resultse-docvqae-cocoe-refcocoe-videoe-settingse-countbenche-widget
What it supports. DocVQA rises from 43.7 ANLS at 224 pixels to 78.0 at 448 and 84.8 at 896; COCOcap changes more modestly, from 141.9 to 144.6 CIDEr. These illustrate a task-dependent return on higher resolution. The table also contains segmentation and observed-video results, extending the evidence beyond ordinary single-image question answering.
Where the evidence stops. The 896-pixel scores lack reported deviations. Cross-row averaging would mix incompatible metrics. CountBenchQA used the original full image split, and Appendix E warns that earlier WidgetCap scores require protocol corrections before comparison.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Document question answering (DocVQA) 44,812 train-plus-validation examples; resolution-specific fine-tuning; ANLS evaluated on the test server. | 43.7 ± 0.5 at 224px; 78.0 ± 0.3 at 448px; 84.8 at 896px. ANLS, higher is better, reported on a 0–100 scale | The 896px checkpoint exceeds the 224px result by 41.1 ANLS points. A large resolution-associated gain, with both extra Stage2 training and more image tokens. The central scores are single runs; server-task deviations come from validation, and no 896px deviation is supplied. e-resultse-docvqae-stage2 |
| Referring-expression segmentation (RefCOCO testA) Combined RefCOCO/RefCOCO+/RefCOCOg training data, 24,407 examples after excluding all validation and test images; evaluated on RefCOCO testA. | 75.7 ± 0.1 at 224px; 77.9 ± 0.1 at 448px; 78.7 at 896px. Mean intersection over union (mIoU), higher is better | 896px improves by 3.0 mIoU points over 224px. Structured visual output works through token generation. This testA result does not summarize every referring-expression split; uncertainty is absent at 896px. e-resultse-refcoco |
| Video question answering (ActivityNet-QA) 43,130 train-plus-validation examples; 16 frames per clip at 224px; one-epoch transfer and greedy decoding; test split. | 50.8 ± 0.4. Exact-match accuracy (%) | Table 1 supplies neither another resolution nor a matched external baseline for this row. Demonstrates supervised transfer to observed-video understanding. It measures neither future-video generation nor successful execution of the described activity. e-resultse-video |
| Paired visual question answering (MMVP) Evaluation without MMVP fine-tuning, using the VQAv2-transferred model; 224px input. | 47.3; no uncertainty reported. Paired accuracy (%) | The paper cites GPT4-V at 38.7 and Gemini at 40.7, corresponding to advantages of 8.6 and 6.6 percentage points. A strong cross-task result as reported by the authors. This is not the unadapted base checkpoint, and the cited external systems were not rerun under a shared training protocol. e-resultse-mmvp |
| Connector design ablation Default shortened Stage1 ablation regime: 100 million examples, followed by validation-based transfer evaluation. | All weights tuned: linear 77.2, MLP 77.1. Both backbones frozen: linear 70.7, MLP 69.7. Average transfer score across tasks, higher is better | The tested one-hidden-layer GeLU MLP does not improve either aggregate. Supports the simple connector within this setup. No uncertainty is given for these aggregate values; this does not establish superiority over every possible connector. e-connectore-objective |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 5. Post-transfer averages and pretraining loss reveal different effects of task design. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. The left panel extends causal masking progressively: 'Suf' covers the answer only, '+Pref' also covers the text prefix, and '+Img' covers the image positions as well. Within the latter groups, curved arrows identify the variants that also predict prefix tokens. The middle panel changes a different factor: whether pretraining includes a task indicator. Its blue and mauve bars map to the same colors in the right-hand learning curves. Higher transfer scores are better, whereas lower upstream negative log-likelihood is better. Cross-check the choices with Section 5.2 and Figure 17; Figure 18 supplies the per-task task-indicator comparison. e-objectivee-mask
What it supports. The suffix-only mask and loss obtain the highest average in the left panel. Adding prefix supervision lowers the corresponding causal-prefix variants. Task indicators barely change the average transfer result, yet they reduce ambiguity during captioning and segmentation pretraining. A downstream average alone can therefore conceal a meaningful training effect.
Where the evidence stops. These are normally 100-million-example Stage1 ablations evaluated after transfer, not the full billion-example final run. The average pools heterogeneous task scores; no numerical uncertainty or calibrated values are given for the small loss curves.
Figure 9. More input tokens help even when the image first loses fine detail. Original paper, p. 11 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. The blue bar uses the 224-pixel model. The orange bar uses the longer 448-pixel input sequence, but every image is first reduced to 224 pixels and then resized upward. The green bar supplies the same high-resolution model with the image directly resized to 448 pixels. Crucially, the orange treatment is used throughout Stage2 and transfer, not just as a test-time corruption. Compare blue to orange to examine the effect associated with increased token computation at low detail, then orange to green to examine retained image information. The left group averages nine resolution-sensitive tasks; the right group averages eleven relatively insensitive tasks. e-resolutione-stage2e-architecture
What it supports. For the sensitive group, both steps produce substantial gains, with roughly comparable average contributions. The insensitive group changes little. The result motivates choosing resolution according to the task and shows why interpreting the main table's improvement solely as better small-text visibility would miss part of the mechanism.
Where the evidence stops. Section 5.7.1 extends the roughly equal split to individual tasks, but Figure 21 shows uneven increments, notably for ChartQA-human. Treat equality as an aggregate tendency; the bars also do not measure parameter-count changes or isolate every training confound.
7. Analysis & limitations
7.1 What the evidence leaves open
Resolution gains combine visual information, sequence length and continued training. Figure 9 controls information by downsampling before upsampling, supporting contributions from both detail and additional token computation. Its mixed-metric average should not be read as one task's accuracy. e-stage2e-resolution
Few-example curves report best hyperparameter/seed runs without a separate minival, and the authors acknowledge instability. The reported proximity to full-data performance indicates potential rather than the expected outcome of one arbitrary small-data run. e-fewshot
CountBenchQA metrics use the full 540-image split although only 490 images remain in the cited package; they were not recomputed. Appendix E also identifies incompatible WidgetCap splits and reference-caption handling in prior comparisons. e-countbenche-widget
Section 5.4 names TT and TF for frozen-ViT comparison, but Figure 7's ViT/Gemma legend and near-equal bars indicate TT and FT. Read the plot with its explicit ordering; the prose inconsistency remains unresolved. Task-specific freezing settings should likewise follow Appendix J. e-freezinge-settings
7.2 Questions for discussion
- Which tasks benefit specifically from prompt-conditioned image representations rather than from longer pretraining?
- How much of a higher-resolution checkpoint's gain survives when image detail and token count are controlled separately?
- How stable is few-example transfer when hyperparameter selection and evaluation are strictly separated?
8. Reproducibility audit
8.1 Requirements and known gaps
A concrete transfer target is DocVQA: ten epochs, batch 256, learning rate 1e-5, weight decay 1e-6, no dropout or label smoothing, unfrozen ViT and greedy decoding. Use the checkpoint matching each resolution and preserve validation-based selection before the final train-plus-validation run. e-docvqae-transfer
Transfer reconstruction requires the exact pretrained checkpoint, tokenizer, special-token conventions, image preprocessing, data splits and evaluation code. Rebuilding pretraining additionally requires private data and mixture/configuration details not fully enumerated in the PDF. The named codebase and hardware do not establish reproducibility by themselves. e-architecturee-maske-mixturee-computee-settings
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Does joint input attention help after matched transfer?
Reader-proposed check, not performed: from identical SigLIP/Gemma initializations, run a reduced-scale Stage1 experiment with the same declared multimodal subset, examples, 224-pixel inputs, schedule and seeds. Compare suffix-only causal masking with causal masking on image, prefix and suffix, keeping suffix-only loss in both arms. Add a separate causal-prefix arm with and without prefix loss to isolate supervision. Give each arm the same transfer search budget on captioning and referring-expression segmentation, reporting CIDEr and mIoU separately across seeds. A consistent advantage for joint input attention would support the mechanism; no advantage, or gains confined to one task, would narrow it. A public-data substitute tests the mechanism, not exact full-pretraining reproduction. e-architecturee-maske-stage1e-objectivee-cocoe-refcocoe-compute
Check 2: Do longer sequences help when fine detail is removed?
Reader-proposed check, not performed: start three arms from the same 224-pixel Stage1 checkpoint and use matched continued-training examples and task mixtures. Compare native 224 inputs, 224-downsampled-then-448 inputs, and native 448 inputs, applying each transformation during continued training, transfer and evaluation. Use identical DocVQA and COCOcap transfer splits, controlled hyperparameter budgets and repeated seeds; report ANLS, CIDEr, latency and token counts separately. Reproducing an intermediate score for the downsampled 448 arm would support two contributing mechanisms. If it matches native 224, the extra-token explanation weakens; if it matches native 448, retained fine detail contributes little under that protocol. Report this as a controlled mechanism test rather than equating it with Table 1. e-stage2e-resolutione-docvqae-cocoe-architecturee-transfer
8.3 Reading coverage
Visual audit: Visually inspected the title and author page; Figures 1–12; Tables 1, 2 and 5; selected Appendix B examples and protocols; Appendices D, E and J; and Figures 17–19, 21–22 and 25. All five final original crops were separately viewed at their native dimensions. Arrow directions and attention permissions were checked against Sections 3.1 and 5.2; the two ablation guides were checked against their appendix breakdowns. Every evidence page supporting retained method, numeric, training, inference and proposed-check details is included above. All supplied text was read; uncited appendix figures, other task-example images, and Tables 3–4 were not part of the visual pass. No external supplement, code, checkpoint or dataset was inspected.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 30, 31, 32, 35, 39, 42, 43, 47, 48, 49, 51, 52, 53, 55, 56, 59. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Title and abstract; Sections 1–2: Introduction and Related work
- Sections 3.1–3.2.6: architecture, all training stages, task mixture and implementation
- Section 4: Results; Sections 5.1–5.7.4: all ablations
- Sections 6.1–6.3: repeatability, hyperparameters and limited examples; Sections 7–8
- References and author contributions
- Appendix A: related work; Appendix B.1–B.39: every task
- Appendices C–H: augmentation, CountBenchQA, WidgetCap corrections, visual prompts, Objaverse and multitask transfer
- Appendices I–J: inference and all transfer hyperparameters
- Appendix K.1–K.9: all per-task ablation text and extracted labels
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- Reviewed the supplied 59-page arXiv:2407.07726v2 artifact, stamped 10 October 2024, whose title block retains July 2024. The title and 35-author list match the catalog. The catalog's initial submission date does not identify this revision; v1 was not supplied and revision changes were not compared.
- Text extraction does not reconstruct figure images; the retained PDF was visually inspected separately for the figures, tables and evidence pages declared in the illustrated edition. All 12 supplied text chunks were read individually; visual inspection was selective rather than every PDF page.
- Separate supplemental material availability has not been fully verified. No separate supplement was supplied.
- Code, model checkpoints and datasets were not inspected; no experiment was run or reproduced. Linked resources were treated only as provenance.
- Author-specific institutional affiliations are not assigned in the title block; optional affiliation metadata is omitted.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e-identityPDF p. 1, title, complete author block and arXiv revision stamp
The title and 35 named authors match the catalog. The artifact is arXiv:2407.07726v2, stamped 10 October 2024, while the title block retains July 2024.
Go to primary source ↓e-architecturePDF pp. 2–3, Section 3.1 and Figures 1–2
SigLIP ViT-So400m and pretrained Gemma-2B v1.0 are joined by a zero-initialized linear projection. Images are resized to fixed squares; 224, 448 and 896 pixels yield 256, 1024 and 4096 image tokens.
Go to primary source ↓e-maskPDF p. 3, Figure 2 and Section 3.1; pp. 8–9, Section 5.2
Image and prefix positions attend bidirectionally within the input block; suffix prediction is autoregressive. The separately tokenized newline separates prefix and suffix. Next-token loss is applied to the suffix only.
Go to primary source ↓e-structuredPDF p. 9, Section 5.3; p. 35, Appendix B.22
The vocabulary adds 1024 binned coordinate tokens and 128 VQ-VAE mask-code tokens. The RefCOCO example serializes four coordinates followed by mask codes. Small Gaussian initialization with sigma 0.02 outperforms the average-embedding initialization after training.
Go to primary source ↓e-stage1PDF p. 4, Section 3.2.2
Stage1 trains all components for one billion examples at 224 pixels with 256 image tokens and text length 128. SigLIP receives a slow linear learning-rate warm-up to limit disruption from initially misaligned gradients.
Go to primary source ↓e-stage2PDF pp. 4–5, Section 3.2.3
Continued pretraining adds 50 million examples at 448 pixels and then 10 million at 896 pixels. Text length rises to 512; OCR, detection and segmentation can supply denser and longer targets.
Go to primary source ↓e-mixturePDF pp. 5–6, Section 3.2.5
Tasks include multilingual captioning on WebLI and CC3M-35L, OCR, generated question answering and question generation, object-centric questions, detection, segmentation and grounded captioning. Transfer datasets and their near-duplicate images are excluded according to the authors.
Go to primary source ↓e-computePDF pp. 6–7, Section 3.2.6 and Figure 3
The paper reports big_vision, JAX/GSPMD, fully sharded training on Cloud TPUv5e, private portions of the pretraining data, a continuous reciprocal-square-root schedule and cosine transfer cooldown. Stage1 on TPUv5e-256 takes slightly less than three days; each Stage2 takes 15 hours.
Go to primary source ↓e-transferPDF p. 5, Section 3.2.4; p. 7, Section 4
Base checkpoints are intended for adaptation. Multi-image inputs concatenate separately encoded images without added separators; video uses 16 frames. Transfer hyperparameters are selected with validation or held-out minival data before final training.
Go to primary source ↓e-settingsPDF pp. 48–49, Appendix J, introduction and task configuration rows
The appendix specifies per-task resolution, epochs, batch size, learning rate, weight decay, dropout, smoothing, frozen ViT and decoding. Despite the main-text all-parameter transfer statement, several rows freeze ViT, including ScienceQA, 448-pixel GQA and some video tasks.
Go to primary source ↓e-resultsPDF p. 7, Table 1, header, caption and task rows; p. 12, Section 6.1
Table 1 gives one random run out of five and transfer-run standard deviations where shown; server-dependent scores use validation-set deviations. Its indented entries are evaluations of the preceding transferred model on other tasks. Most 896-pixel entries have no uncertainty shown.
Go to primary source ↓e-docvqaPDF p. 32, Appendix B.11; p. 48, Appendix J, DocVQA rows
DocVQA uses 44,812 train-plus-validation examples and test-server ANLS. At every listed resolution, transfer uses ten epochs, batch 256, learning rate 1e-5, weight decay 1e-6, no dropout or label smoothing, unfrozen ViT and greedy decoding.
Go to primary source ↓e-cocoPDF p. 31, Appendix B.7; p. 32, Appendix B.8; p. 48, Appendix J, COCOcap rows
COCOcap uses 113,287 train-plus-restval images with five captions each and reports validation CIDEr. Transfer runs five epochs with beam size two. NoCaps is evaluated without NoCaps fine-tuning using the COCOcap-transferred model.
Go to primary source ↓e-refcocoPDF p. 35, Appendix B.22; p. 49, Appendix J, RefCOCO rows
Referring-expression segmentation combines 24,407 training examples from RefCOCO, RefCOCO+ and RefCOCOg after excluding all validation and test images. The metric is mIoU on test splits; the listed transfers use 100 epochs and label smoothing 0.3.
Go to primary source ↓e-videoPDF p. 5, Section 3.2.4; p. 30, Appendix B.2; p. 49, Appendix J, ActivityNet-QA row
ActivityNet-QA uses 43,130 train-plus-validation examples and exact-match test accuracy. Video inputs use 16 independently encoded frames. Its 224-pixel transfer uses one epoch, batch 128, learning rate 1e-5, weight decay 1e-6, unfrozen ViT and greedy decoding.
Go to primary source ↓e-mmvpPDF p. 14, Section 7, MMVP paragraph; p. 39, Appendix B.36
MMVP measures paired accuracy without MMVP fine-tuning after VQAv2 transfer. The paper reports 47.3% at 224 pixels and cites comparison values of 38.7% for GPT4-V and 40.7% for Gemini.
Go to primary source ↓e-objectivePDF pp. 8–9, Section 5 introduction, Section 5.2 and Figure 5; pp. 51–52, Appendix K.2, Figures 17–18
Ablations normally shorten Stage1 to 100 million examples and use validation results. Suffix-only causal masking and suffix-only loss give the highest average in Figure 5. Task indicators reduce ambiguity for pretraining captioning and segmentation but have little effect on average post-transfer performance.
Go to primary source ↓e-resolutionPDF p. 11, Section 5.7.1 and Figure 9; p. 55, Appendix K.5 and Figure 21
The middle condition downsizes images to 224 before resizing to 448 throughout Stage2 and transfer. On nine resolution-sensitive tasks the two successive score gains are roughly comparable; eleven insensitive tasks change little. Figure 21 supplies task-level variation.
Go to primary source ↓e-connectorPDF p. 10, Section 5.5
With all weights tuned, linear and one-hidden-layer GeLU MLP connectors score 77.2 and 77.1 on average; with both backbones frozen, they score 70.7 and 69.7. These are the paper's aggregate transfer scores.
Go to primary source ↓e-freezingPDF pp. 9–10, Section 5.4 and Figure 7; p. 53, Appendix K.3 and Figure 19
Figure 7 labels the order ViT, Gemma; TT and FT have similar average transfer performance, while freezing the language model or resetting components hurts. The prose instead names TT and TF when discussing frozen ViT, inconsistent with that legend and bar ordering.
Go to primary source ↓e-fewshotPDF p. 13, Section 6.3 and Figure 12; p. 59, Appendix K.9 and Figure 25
Limited-example transfers use 64, 256, 1024 or 4096 examples, sweep hyperparameters and five seeds, and report best runs without separate minival selection. The authors report most tasks within 10% of full-data scores at roughly 4k examples and within 20% at 256; they acknowledge instability.
Go to primary source ↓e-countbenchPDF p. 42, Appendix D
CountBenchQA adds manually formed questions to 540 CountBench images, balanced over counts two through ten. The cited available package contains image data for only 490 entries; the paper's metrics used the full split and were not recomputed.
Go to primary source ↓e-widgetPDF p. 43, Appendix E
Earlier PaLI WidgetCap values used validation rather than test splits, and ScreenAI used only one of five reference captions in CIDEr. The paper explains why uncorrected comparisons to its test result are invalid.
Go to primary source ↓e-inferencePDF p. 47, Appendix I and Table 5, caption and bfloat16 batch-one rows
Measurements use eight TPUv3 devices (four chips), 512 prefill tokens and a 640-token cache. For bfloat16 batch one, FSDP reports 101 ms prefill and 8.2 ms extend; Megatron sharding reports 17 ms and 5.3 ms. The latter shards activations as well as parameters.
Go to primary source ↓8.5 Primary sources
PaliGemma: A versatile 3B VLM for transfer ↗
PDF · 21,186 extracted words
Source fingerprint
aec6edd6b712597647d5e6e10850039d6415620163f45bb64f71d89718e6bb2f