Sigmoid Loss for Language Image Pre-Training
1. Paper overview
In one sentence: Independent image–text matching losses make distributed pretraining more memory-efficient, while the experiments show that larger batches alone are a poor substitute for an appropriate training recipe. e02e03e04e09e10e12e17
| At a glance | What to know |
|---|---|
| Research problem | Source description Softmax contrastive training normalizes image–text similarities across a batch in both directions. This couples the loss to the candidate set and complicates distributed evaluation. The paper asks whether binary matching decisions can simplify training while preserving useful representations. Unmatched pairs are assumed negative, although the authors explicitly acknowledge that this assumption is noisy. e02e03e04 |
| Core mechanism | |
| A key reported result | Zero-shot ImageNet classification with controlled SigLIP loss comparisons: 73.4 ImageNet zero-shot accuracy (%). English WebLI; Base image/text towers; 9B seen examples; batch 32k. Same-batch softmax: 72.9; softmax's best listed 9B result: 73.2 at batch 98k. A 0.5 percentage-point same-batch gain. At batch 4k, the corresponding scores are 68.4 versus 66.6. At 307k, sigmoid falls to 71.6 versus softmax 72.6; the advantage is not universal. e05e12 |
| Reading caution | Reader analysis The 32k recommendation is empirical and schedule-dependent. Table 8 gives SigLiT 84.6 at 32k and 84.7 at 1024k after 18B examples, while short schedules can strongly penalize large batches. More negatives also remain useful: random masking harms transfer, whereas retaining hard negatives fares better. e11e13e17 |
Core contributions
Figure 1. Independent pair terms allow the complete loss to be accumulated one local block at a time. Original paper, p. 3 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the four panels from left to right. Columns are image representations, rows are text representations, and colors identify devices. Initially each device owns four images and four texts. The second panel computes local blocks: plus signs mark true matches, while minus signs mark unmatched combinations. The next panels keep the images local and move the text blocks to other devices. Checkmarks indicate interactions already included; the yellow blocks indicate the current computation. Downward arrows accumulate partial loss, and the final arrows converge on a cross-device sum. All combinations are covered, even though each device only materializes a 4×4 similarity block at a time. e02e03e04
What it supports. The graphic explains a storage advantage: the distributed implementation need not hold the complete global similarity matrix or all-gather all embeddings. Independence allows each block's loss to be added directly. It does not remove the complete objective's quadratic number of pair interactions, so lower peak storage should not be read as linear total computation.
Where the evidence stops. Algorithm 1 uses t·similarity + b, while the displayed Section 3.2 loss reverses b's sign. The tutorial follows the algorithm and negative-bias rationale. This discrepancy does not change the block-exchange diagram, whose arrows agree with its caption and Section 3.3.
2. Motivation
2.1 The problem and the proposed response
Softmax contrastive training normalizes image–text similarities across a batch in both directions. This couples the loss to the candidate set and complicates distributed evaluation. The paper asks whether binary matching decisions can simplify training while preserving useful representations. Unmatched pairs are assumed negative, although the authors explicitly acknowledge that this assumption is noisy. e02e03e04
2.2 What this reading follows
An image–text model needs to align two representations, but that does not require choosing one match through a batch-wide softmax. SigLIP treats each candidate pair as a binary decision. This small change has two consequences worth following separately: the computation can be distributed in compact blocks, and the balance of positive and negative pairs becomes an explicit experimental variable. The paper tests both frozen-image tuning and joint encoder training, then scales the recipe to multilingual and released models. Read the controlled loss comparisons before the headline model table: the latter demonstrates useful systems, while also changing training duration, model size and image resolution. e02e03e04e09e10e12e17
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Foundational work |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
This table preserves the labels recorded at reading time. The current major category is Components of WAMs. View the current classification.
3.1 Evidence-based assessment
Supports the recorded classification
The recorded foundational-work classification is supported. The source specifies a vision encoder and a language encoder used for representation comparison. Joint optimization of these towers does not constitute a One Model world-action architecture. There is no future-state prediction, inverse dynamics, action extraction or execution evaluation; architecture, prediction paradigm and quadrant are therefore not applicable in the catalog's WAM sense. e02e03e20
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Turn one batch-wide choice into many pair decisions
The two encoders still learn a shared representation space. What changes is the loss applied to their similarity scores. Softmax asks a matching pair to beat all the alternatives used in its normalization, once in each direction. Algorithm 1 instead assigns every image–text combination a signed label and applies log-sigmoid to its signed score. The diagonal supplies positive supervision and the other combinations supply assumed negatives. The loss is divided by the number of input pairs, even though it sums over all combinations. A learned negative bias is therefore useful at initialization, when negatives vastly outnumber positives. The printed loss has a conflicting bias sign; the report's equation deliberately transcribes the pseudocode rather than concealing that discrepancy. e02e03e16
5.2 Separate representation reuse from end-to-end training cost
The four-chip SigLiT result is easiest to understand as text alignment to a strong existing visual representation. The vision encoder is frozen, and its embeddings are precomputed, so the reported training stage avoids repeatedly updating that tower. SigLIP changes the problem by learning both towers, either from scratch or by unlocking pretrained image weights. The fine-tuning experiments then show that a familiar regularizer can undermine the reused representation: removing weight decay from pretrained image weights improves the plotted transfer behavior. Reader interpretation: efficiency claims should be attached to a complete initialization and training protocol. The 84.5% four-chip SigLiT result and the from-scratch SigLIP results answer different resource questions, because the frozen backbone has its own prior training history. e01e06e07e08
5.3 Distinguish more examples, more pairs and more useful negatives
A larger batch changes several quantities simultaneously. At fixed examples seen, it supplies more within-batch pair combinations but fewer parameter updates. That explains why the training-duration plot belongs beside the batch-size plot. The negative-selection experiment takes another route: keep the input batch and ignore selected pair losses. Its random, hard and easy policies reveal that pair identity matters, not just the positive:negative ratio. Yet extending training to match total pairs introduces another budget change. Reader interpretation: these experiments motivate measuring examples, selected pairs, updates and runtime together. They do not prove that an efficient mining algorithm has already replaced exhaustive pair evaluation. Likewise, the multilingual *32k improvement combines a scaled training recipe with the same nominal batch size, so it is not evidence for a larger-batch advantage. e11e13e17e09e14
5.4 Training and inference
During training
The standard English SigLIP study uses WebLI, a ViT-B/16 image tower and Base text transformer, 224×224 images, and a C4-trained 32k SentencePiece vocabulary with at most 16 text tokens. Default learning rate and weight decay are 0.001 and 0.0001; the paper uses ScalingViT-Adafactor and β₂ = 0.95. e05e18e22
The SigLiT sweep uses frozen ViT-g embeddings at 288×288 and a Base text tower. Appendix A specifies β₁ = 0.9, β₂ = 0.95, linear warmup over 200M examples, then cosine decay. The four-chip B/8 recipe instead uses LION and a 12-layer Large text tower. e06e07
When initializing an unlocked image tower from pretrained weights, use a 0.1 learning-rate multiplier and remove weight decay from those weights. Figure 4 links this change to better zero-shot performance and preservation of 10-shot linear classification quality. e08
Scaled released models train for 40B seen examples at batch 32k with 256 image patches and 64 text tokens. Resolution adaptation adds 5B examples at 100 times smaller learning rate without weight decay. Multilingual training uses unfiltered WebLI; a 250k vocabulary can be factorized through a 96-dimensional bottleneck instead of a full 768-wide Base lookup. e09e14e20
During inference
The model card specifies feature-vector comparison for zero-shot classification and image–text retrieval. No training batch, negative exchange, dynamics rollout or control loop is needed at deployment. Exact prompt templates and detailed retrieval split construction are not supplied in this PDF. e20e10
5.5 Implementation flow
- Encode the modalities
A vision transformer f and text transformer g produce L2-normalized vectors x_i and y_j. Matching pairs receive z_ij = +1; unmatched combinations receive −1. SigLIP trains both towers; SigLiT trains the text tower against frozen, precomputed image embeddings. e02e03e06
- Score and classify each pair
Algorithm 1 forms logits t times the embedding dot product plus b, with t = exp(t′). It sums negative log-sigmoid values over all pairs and divides by the number of input pairs, not its square. The learned bias starts negative to accommodate the many more negative combinations. e02e03e16
- Accumulate distributed blocks
Figure 1 keeps images local, cycles text representations across devices, and accumulates every interaction before a final cross-device sum. Each device materializes a local square block. This reduces peak similarity storage, while the full objective still evaluates quadratically many pairs. e04
6. Experiments & results
SigLIP learns aligned image and text representations with a pairwise sigmoid objective. Its practical contribution combines independent pair losses, memory-efficient distributed evaluation, and optimization choices. Controlled experiments favor sigmoid most at small batches; increasing batch size indefinitely does not reliably improve transfer. SigLiT reuses a frozen vision encoder, whereas SigLIP trains both towers. These are representation-learning methods for classification and retrieval, not action predictors.
6.1 Read the original evidence
Figure 2. Batch-size saturation appears in three settings, with different metrics and training budgets. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start by separating the panels: SigLiT uses a frozen vision model and 18B seen examples; SigLIP trains its towers for 9B; mSigLIP uses 30B multilingual examples. The first two vertical axes show ImageNet zero-shot accuracy, while the third shows mean text-to-image retrieval across 36 languages. Solid green circles denote sigmoid and dashed orange stars denote softmax where both are present. Follow each curve within its panel rather than comparing heights across panels. The most consistent gain is at smaller batches. At larger batches, the frozen-image curve mostly flattens, while the jointly trained and multilingual curves can turn downward. e06e11e12e13e14
What it supports. The SigLIP table underlying the middle plot reports 73.4% at batch 32k, compared with softmax's best listed 73.2% at 98k. In SigLiT, Table 8 reports only 84.6% to 84.7% improvement between 32k and 1024k after 18B examples. These findings favor a practical batch range rather than maximal batch size.
Where the evidence stops. These panels hold examples seen fixed within a setting, not update count. Figure 3 shows schedule dependence. Plot labels such as 262k and 245k differ from tables' 256k and 240k; their displayed labels are preserved without assuming exact equivalence.
Table 5. Matched data and model settings reveal where sigmoid helps and where it loses. Original paper, p. 14 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read a horizontal sigmoid–softmax pair within either the 3B or the 9B group first. That holds the reported training duration and batch size fixed. Then compare rows within that group to see the batch-size effect. Finally, compare the two duration groups to see how seeing more examples changes the outcome. All entries are zero-shot ImageNet accuracy percentages for the Base SigLIP setup described in Section 4.2, using English WebLI and 224×224 inputs. Dashes are missing results, not zeros. The highlighted entries draw attention to strong settings, but the complete rows are needed to see the large-batch reversal. e05e12e15e22
What it supports. At 3B examples and batch 512, sigmoid reaches 51.5% versus 47.7%, a 3.8-point difference. At 9B and batch 32k, the difference is smaller: 73.4% versus 72.9%. At 307k it reverses, with 71.6% versus 72.6%. The loss is useful under many tested settings, not uniformly superior.
Where the evidence stops. Table 5 has no uncertainty estimates. The five-run statistics in Table 7 apply specifically to 3B examples at 32k. Table 6 reports different default scores for a similarly described 3B sweep; the source does not reconcile them.
Table 3. Scaled SigLIP models are strong released encoders, with performance depending on both architecture and resolution. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. First identify a model's vision size and patch count, then read across the task columns. Validation, v2 and ReaL are ImageNet evaluations; ObjectNet tests a different image distribution. COCO I→T retrieves text from an image, while T→I retrieves images from text. Dashes preserve results absent from this table. Within SigLIP, compare the Base rows as patch count increases, and then the Large and SO rows. Section 4.6 describes extended training and resolution adaptation for these models. The table's retrieval headers resolve the duplicated I→T abbreviation in the surrounding prose, which otherwise labels both retrieval directions the same way. e09e10e20
What it supports. The SO (400M), 729-patch row reaches 83.2% ImageNet validation accuracy and 82.9% on ObjectNet. Its COCO recall@1 is 70.2% I→T and 52.0% T→I. The Large 576-patch SigLIP row is slightly stronger on retrieval at 70.6% and 52.7%, so the best classification row is not best on every task.
Where the evidence stops. This is a comparison of complete systems, not an isolated loss ablation. Architectures, patch counts and training histories differ. The PDF does not specify the exact COCO split here, and this reading does not verify the linked model releases.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Zero-shot ImageNet classification with controlled SigLIP loss comparisons English WebLI; Base image/text towers; 9B seen examples; batch 32k. | 73.4 ImageNet zero-shot accuracy (%) | Same-batch softmax: 72.9; softmax's best listed 9B result: 73.2 at batch 98k. A 0.5 percentage-point same-batch gain. At batch 4k, the corresponding scores are 68.4 versus 66.6. At 307k, sigmoid falls to 71.6 versus softmax 72.6; the advantage is not universal. e05e12 |
| Zero-shot ImageNet classification with four-chip SigLiT Frozen ViT-g/14, Large text tower, LiT image–text data; batch 20k; 107k steps; four TPU-v4 chips, under two days. | 84.5 ImageNet zero-shot accuracy (%) | The B/8 four-chip one-day setup is 79.8 in Table 1 but 79.7 in its caption and Section 4.4. Efficient text alignment benefits from an existing vision backbone and precomputed embeddings; this is not a from-scratch total-system compute comparison. e01e07 |
| Zero-shot classification and retrieval with released SigLIP models Table 3; scaled WebLI training; SO (400M) image encoder with 729 patches. | 83.2 / 70.2 / 52.0 ImageNet validation accuracy / COCO image-to-text R@1 / COCO text-to-image R@1 (%) | EVA-CLIP E (5B), 256 patches: 82.0 / 68.8 / 51.1. SigLIP L, 576 patches: 82.1 / 70.6 / 52.7. The SO model leads these classification rows but does not exceed SigLIP L on COCO retrieval. Different models, patch counts and training recipes prevent attributing the comparison solely to sigmoid loss; the COCO split is not specified here. e09e10 |
| Multilingual zero-shot retrieval on XM3600 mSigLIP Base; unfiltered WebLI; 36-language average; batch 32k. | 34.9 / 47.4 at 30B seen examples; 42.6 / 54.1 for the scaled *32k model. Text-to-image / image-to-text recall@1 (%) | At 30B and batch 240k: 32.7 / 45.0. Scaling training duration and the input recipe helps more than merely enlarging the batch in these settings. Language averages hide uneven performance. e09e14 |
| Bias initialization ablation SigLIP Base; batch 8k; 900M seen examples; t′ initialized to log 10. | 63.0 / 82.4 / 61.0 with learned b initialized to −10. Zero-shot ImageNet / Oxford-IIIT Pet / CIFAR-100 accuracy (%) | Without bias: 62.0 / 81.8 / 59.9; b initialized to zero: 61.7 / 79.9 / 59.0. Both including the bias and choosing its initial value matter under this protocol; Table 4 supplies no uncertainty estimates. e16 |
| Repeated-run ImageNet evaluation SigLIP Base; batch 32k; 3B seen examples; five repetitions. | 70.1 ± 0.2 with sigmoid and ViT-Adafactor. Mean ± standard deviation of zero-shot accuracy (%) | Softmax with ViT-Adafactor: 69.9 ± 0.1; sigmoid with AdamW: 70.3 ± 0.1. These are run standard deviations, not confidence intervals. They do not establish uncertainty for the larger models or other batch sizes. e15 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 4. Initializing the bias for a predominantly negative pair set improves early optimization and final transfer. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. The left columns specify initial values, not the parameters' final learned values. In the first row, n/a means that the bias term is absent. Compare that row with the second while keeping t′ = log 10: this isolates adding a bias initialized to −10. Compare the second with the fourth to study the bias initialization at the same temperature. The third and fifth rows also alter temperature, so their changes cannot be attributed to bias alone. INet-0, Pet-0 and C100-0 denote zero-shot accuracy on ImageNet, Oxford-IIIT Pet and CIFAR-100. The common protocol is Base architecture, batch 8k and 900M seen examples. e02e03e16
What it supports. Adding the negatively initialized bias raises the three scores from 62.0/81.8/59.9 to 63.0/82.4/61.0. Starting an included bias at zero performs worse than starting at −10. The table therefore supports a specific initialization recipe, rather than the broader claim that any extra trainable bias must improve the model.
Where the evidence stops. These are single table entries without reported uncertainty. The bias-sign conflict between Algorithm 1 and the displayed loss matters for reproduction: copying the printed exponent literally would change the effect of the recommended negative initialization.
Figure 6. Hard negatives preserve more learning signal than random rebalancing, but the strongest condition trains longer. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Move left to right along the positive:negative ratios: fewer negatives remain for each positive. Blue circles keep a random subset; orange stars keep the highest-loss negatives; green triangles keep the easiest; pink crosses keep hard negatives while extending training to match total pairs seen. The left panel measures ImageNet zero-shot accuracy. The middle shows the learned bias, which generally becomes less negative as the pair mixture becomes less negative-heavy. The right follows mean positive and negative logits, with an upper and lower branch for each policy. The important comparison is policy at the same ratio, while remembering that pink also changes training duration. e17
What it supports. Randomly discarding negatives increasingly damages transfer, and keeping only easy negatives fails quickly. Retaining the hardest negatives preserves much more quality. The longer matched-pair condition improves slightly, motivating the authors' view that imbalance itself is not the main problem. Useful negative content and the total exposure both matter.
Where the evidence stops. The matched-pair condition spends additional updates and is not a fixed-compute comparison. Some easy-negative curves leave the plotted range; their unseen values should not be inferred. Section 4.8's literal 900M-steps duration remains unresolved rather than being changed to examples.
7. Analysis & limitations
7.1 What the evidence leaves open
The 32k recommendation is empirical and schedule-dependent. Table 8 gives SigLiT 84.6 at 32k and 84.7 at 1024k after 18B examples, while short schedules can strongly penalize large batches. More negatives also remain useful: random masking harms transfer, whereas retaining hard negatives fares better. e11e13e17
Artificial image noise, random tokens and shuffled alignments favor sigmoid in the tested M-scale setting, but do not establish robustness to every real web-data failure. Language-level XM3600 results are highly uneven. e19e14
Source ambiguities remain: Section 4.8 says 900M steps for negative selection; it is not silently reinterpreted as examples. Table 6's default sweep scores differ from Table 5 despite a similarly described 3B protocol. The PDF does not explain that difference. Bias-sign and Table 1 inconsistencies are retained above. e17e22e12e03e01
7.2 Questions for discussion
- How much of the small-batch advantage remains under equally tuned compute budgets?
- Can hard-negative selection save measured runtime after accounting for mining and additional updates?
- Which language-specific data changes would improve retrieval more than a larger multilingual batch?
8. Reproducibility audit
8.1 Requirements and known gaps
A reproduction needs the correct WebLI or LiT data, preprocessing, tokenizer, chosen pretrained backbone or random initialization, loss normalization and optimizer recipe. The PDF names big_vision and Google Cloud TPUs but supplies no pinned software version, complete dataset-filter implementation or exact evaluation prompts. e05e06e07e20
Proposed checks: first verify dense versus chunked loss and gradients on identical embeddings, including bias-sign controls; then compare negative-selection policies with both fixed-example and matched-pair budgets. Report runtime as well as transfer quality so extra training cannot masquerade as a free mining improvement. e02e03e04e17
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Check loss and gradient equivalence before scaling devices
Reader-proposed check, not performed: use a tiny deterministic set of image/text embeddings and compare Algorithm 1's dense objective with Figure 1's complete block accumulation. Keep normalization, labels, temperature, bias and every pair identical; compare loss and gradients for embeddings, t′ and b. Require agreement within a predeclared floating-point tolerance, and verify that every pair is visited exactly once. Add a deliberate control using the printed Section 3.2 bias sign: at b = −10 it should expose the sign discrepancy, rather than silently passing as the same implementation. Only after equivalence holds, increase batch size and measure peak similarity memory and wall-clock time with hardware and precision held fixed. e02e03e04e16
Check 2: Test whether hard-negative selection saves useful computation
Reader-proposed check, not performed: start with one frozen-image SigLiT setup and compare all negatives, random retention and highest-loss retention at a common selected ratio, such as the figure's 1:164 condition. First hold input examples and optimizer updates fixed; then add a separately labeled longer schedule that matches selected-pair exposure, following Figure 6. Record ImageNet zero-shot accuracy over multiple seeds, actual pair counts, learned bias, total updates, mining overhead and wall-clock time. The falsifiable question is whether hard selection preserves accuracy while reducing total measured runtime relative to all pairs; matching quality only through extra updates would not demonstrate a computational saving. Resolve the source's 900M-steps wording before claiming an exact replication of its schedule. e06e13e17e15
8.3 Reading coverage
Visual audit: Visually inspected the title/author page, Algorithm 1, both objective formulations, Figure 1's device exchanges and sum, Figures 2–8, Tables 1–9, relevant training/evaluation prose, appendices A–F and the model card. Every page supporting retained numerical, method, training, evaluation or proposed-reproduction details is included. All six final original crops were inspected; narrow Tables 4 and 5 were rendered at 400 DPI. The Figure 1 crop excludes explanatory subcaptions, which were read on the full page. No scientific marks were changed. Bias-sign, Table 1 value, retrieval-abbreviation, batch-label, Table 6 protocol and negative-study duration discrepancies are preserved in the report and relevant cautions. Reference pages 11–13 were read in the complete text but not visually rendered. External supplements and code remain outside this reading.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 14, 15, 16, 17. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract and title/author block, p. 1
- 1. Introduction, pp. 1–2
- 2. Related Work, p. 2
- 3. Method, including 3.1–3.3 and Algorithm 1, pp. 2–4
- 4. Results, including 4.1–4.10, pp. 4–9
- 5. Conclusion and acknowledgements, p. 10
- References, pp. 11–13
- A. More results for SigLiT, pp. 14–15
- B. More results for SigLIP, p. 14
- C. Robustness of SigLIP results, p. 14
- D. More results for mSigLIP, pp. 14–16
- E. Label noise experiments, p. 15
- F. Model Card, p. 17
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- All six supplied text chunks were read individually. The extraction-only visual gap was resolved by inspecting PDF pages 1–10 and 14–17, all eight figures, all nine tables, and every final crop. Reference pages 11–13 were read as text.
- Identity: the observed title and all four authors match the catalog. The title page credits Google DeepMind, Zürich, Switzerland, and marks Xiaohua Zhai and Lucas Beyer as equal contributors. No revision number or date is printed in the inspected material. Page 10 discusses corrections to earlier versions without identifying this revision. The supplied unversioned arXiv artifact is reviewed by its immutable hash; equivalence to the catalog's ICCV 2023 edition is not established.
- No external supplements, code, checkpoints or linked resources were inspected, and no experiments were reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title/author block, equal-contribution footnote, Table 1 and caption
Title and four authors match the catalog; affiliation is Google DeepMind, Zürich, Switzerland. Zhai and Beyer share equal-contribution marks. Table 1 gives B/8 SigLiT 79.8, while the caption says 79.7; g/14 is 84.5.
Go to primary source ↓e02PDF p. 2, Algorithm 1 lines 6–11 and Section 3
Embeddings are L2-normalized; logits equal dot product times exp(t_prime) plus b. Diagonal labels are +1 and others −1; summed log-sigmoid loss is divided by n. Unmatched pairs are only assumed unrelated.
Go to primary source ↓e03PDF p. 3, Sections 3.1–3.2, unnumbered objectives and bias paragraph
The image model is a ViT and the text model a transformer. The displayed sigmoid denominator uses exp[z_ij(−t x_i·y_j+b)], unlike Algorithm 1's additive logit bias. The text initializes t′ to log 10 and b to −10.
Go to primary source ↓e04PDF p. 3, Figure 1 panels a–d and Section 3.3; p. 4, Section 3.3 continuation
The 3-device illustration cycles text representations against local images and finishes with a cross-device sum. All pairs interact; each device materializes a local 4×4 block in the example. General similarity storage changes from global-batch squared to local-batch squared.
Go to primary source ↓e05PDF p. 5, Section 4.2; p. 4, Section 4 evaluation paragraph
English WebLI SigLIP uses B/16 ViT and Base text models, 224×224 images, a 32k C4 SentencePiece tokenizer and 16 text tokens. Evaluation includes ImageNet zero-shot transfer and XM3600 retrieval; default optimizer is ScalingViT-Adafactor.
Go to primary source ↓e06PDF p. 4, Section 4.1; p. 14, Appendix A first paragraph
The SigLiT sweep uses ViT-g precomputed embeddings and a Base text tower on LiT data. Appendix A gives resize-only 288×288 inputs, Adafactor β₁=0.9 and β₂=0.95, learning rate 0.001, 200M-example warmup, cosine decay and weight decay 0.0001.
Go to primary source ↓e07PDF p. 6, Section 4.4; p. 14, Appendix A second paragraph
The four-chip B/8 model has a frozen backbone and 12-layer Large text tower; LION uses peak learning rate 1e−4, weight decay 1e−7 and 6.5k warmup steps within 65k steps. The g/14 variant uses a Large text tower, 20k batch and 107k steps, reaching 84.5 under two days.
Go to primary source ↓e08PDF p. 6, Figure 4; p. 7, Section 4.5 continuation
The unlocked AugReg-B/16 setup applies a 0.1 image-tower learning-rate multiplier. Disabling pretrained-encoder weight decay improves zero-shot and 10-shot behavior; the 16-chip, batch-16k run sees 2.4B examples and reaches 71% in three days.
Go to primary source ↓e09PDF p. 7, Section 4.6
Scaled SigLIP uses B, L or So-400m towers, 40B examples, batch 32k, 256 patches and 64 tokens. Resolution adaptation adds 5B examples, 100-fold smaller learning rate and no weight decay. Scaled mSigLIP is denoted *32k. The prose mistakenly repeats I→T for text-to-image; table headers identify directions.
Go to primary source ↓e10PDF p. 8, Table 3, all columns, especially L/576 and SO/729 rows
SO/729 scores 83.2 ImageNet validation, 77.2 v2, 87.5 ReaL, 82.9 ObjectNet, and COCO R@1 70.2 I→T / 52.0 T→I. L/576 scores 82.1, 75.9, 87.0, 81.0 and 70.6/52.7; EVA-CLIP E/256 gives 82.0 and COCO 68.8/51.1. Dashes mark unreported values.
Go to primary source ↓e11PDF p. 4, Figure 2 and caption; p. 5, Figure 3 and Section 4.1 continuation
Figure 2 compares 18B-example SigLiT, 9B-example SigLIP and 30B-example mSigLIP. Curves saturate or decline with larger batches. Figure 3 shows the interaction with training duration. Figure labels include 262k and 245k where tables use 256k and 240k.
Go to primary source ↓e12PDF p. 14, Appendix B, Table 5, 3B and 9B columns
At 9B examples, sigmoid/softmax accuracy is 68.4/66.6 at 4k, 73.4/72.9 at 32k, 73.0/73.2 at 98k and 71.6/72.6 at 307k. At 3B, 512 gives 51.5/47.7 and 32k gives 69.9/69.9. Missing entries are dashes.
Go to primary source ↓e13PDF p. 15, Table 8, 450M and 18B columns
At 18B examples, SigLiT sigmoid is 84.6 at 32k and 84.7 at 64k, 128k, 256k and 1024k; softmax at 32k is 84.4. At 450M, sigmoid falls from 81.9 at 32k to 72.8 at 256k.
Go to primary source ↓e14PDF pp. 5–6, Section 4.3 and Table 2; p. 15, Figure 8; p. 16, Table 9
Multilingual token lookup uses an N×K matrix and K×W projection, with K=96 and W=768 for Base. At 30B examples, 32k-batch average R@1 is 47.4 I→T and 34.9 T→I, versus 45.0/32.7 at 240k. Scaled *32k gives 54.1/42.6, with substantial language variation.
Go to primary source ↓e15PDF p. 14, Appendix C standard-deviation/alternative-optimizer paragraphs and Table 7
Five repetitions at batch 32k and 3B examples give ImageNet accuracy 69.9±0.1 for softmax Adafactor, 70.1±0.2 for sigmoid Adafactor and 70.3±0.1 for sigmoid AdamW; uncertainty is standard deviation.
Go to primary source ↓e16PDF p. 9, Section 4.9 and Table 4, all rows
At Base/8k/900M, b=−10 and t′=log 10 yields 63.0/82.4/61.0 on ImageNet/Pet/CIFAR-100; no bias yields 62.0/81.8/59.9; b=0 at the same t′ yields 61.7/79.9/59.0. Setting b=0 and t′=log 1 yields 53.7/73.2/53.8.
Go to primary source ↓e17PDF p. 7, Figure 6; pp. 8–9, Section 4.8
The SigLiT 16k experiment masks random, hardest or easiest negatives, with an additional longer-training matched-pairs condition. The text says 900M steps. Random and easy removal reduce quality; hard retention preserves more, and matching pairs uses extra updates. Bias/logits become more positive with fewer negatives.
Go to primary source ↓e18PDF p. 6, Figure 5; pp. 7–8, Section 4.7
Loss, gradient norms and update norms illustrate instability at β₂=0.999. The authors use β₂=0.95 to stabilize Adam/Adafactor training; gradient spikes can persist without the same destabilizing updates.
Go to primary source ↓e19PDF p. 9, Figure 7 and Section 4.10; p. 15, Appendix E
M/16 image and M text towers train from scratch for 3.6B examples, batch 16384, under image-noise, random-token and shuffled-alignment corruption. Sigmoid curves exceed softmax in these experiments. Appendix E gives cosine decay, 10% warmup and peak learning rate 0.001.
Go to primary source ↓e20PDF p. 17, Appendix F, Model Architecture, Inputs, Outputs, Intended Use, Training Data, Evaluation Data, Hardware & Software
Released models have vision and text transformer encoders, compare feature vectors for zero-shot classification/retrieval, and use mostly-English or language-unfiltered WebLI. The card names big_vision and Google Cloud TPUs but gives no software version or evaluation prompt list.
Go to primary source ↓e21PDF p. 10, acknowledgements
The authors acknowledge a pseudocode error in the first version and temperature-notation typos in the second and third versions. This identifies revision history in general, not the specific revision number/date of the supplied artifact.
Go to primary source ↓e22PDF p. 14, Appendix C hyperparameter paragraph and Table 6
Appendix C states default learning rate 0.001 and weight decay 0.0001 across batch sizes and describes a 3B-example hyperparameter sweep. Table 6 default scores 70.1/70.0/68.2 at 8k/16k/32k do not match Table 5's similarly described 3B rows; the difference is unexplained.
Go to primary source ↓8.5 Primary sources
Sigmoid Loss for Language Image Pre-Training ↗
PDF · 11,150 extracted words
Source fingerprint
2efade8de3baeffe9eba759aab9f510ea8f8ac3c365f02f0da75d9baf9ffa2db