Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models
1. Paper overview
In one sentence: Prismatic improves image-conditioned language models through simpler optimization and richer patch features, while exposing task tradeoffs that aggregate scores can hide. designarchitectureoptimizationfusionruntimevqa-prismlocalization-prism
| At a glance | What to know |
|---|---|
| Research problem | Source description Popular VLMs share a patch-as-token architecture but vary preprocessing, pretrained components and optimization. Subjective model-judged evaluations obscure which choices cause gains. The paper builds twelve objective benchmarks spanning question answering, referring-expression localization and challenge tasks, then varies design choices around a reproduced LLaVa v1.5 baseline. designarchitectureprotocol |
| Core mechanism | Author claim A modular training framework and standardized evaluation suite support comparisons across four design axes; code and checkpoint releases are author-reported resource contributions. design |
| A key reported result | RefCOCO localization with controlled PRISM 7B: 73.62 Accuracy at IoU 0.5 (%). Validation split; greedy coordinate-string generation; author-labeled controlled Prism-DINOSigLIP 7B versus official and reproduced LLaVa v1.5 7B. Official LLaVa: 55.12; reproduced LLaVa: 60.54. Derived gains are 18.50 and 13.08 percentage points. This measures localization, not manipulation; Section 5's control label has an unresolved data-accounting boundary. localization-prismprotocolprismoptimization |
| Reading caution | Source description The study excludes learned patch-downsampling architectures, larger-scale generalization and rich extended dialogue. Cultural bias, unsafe outputs and hallucination remain. Qualitative safeguards from ShareGPT co-training provide limited evidence of reliability. limitationssafety |
Core contributions
- Author claim
A modular training framework and standardized evaluation suite support comparisons across four design axes; code and checkpoint releases are author-reported resource contributions. design
- Source description
The study removes projector-only alignment, evaluates frozen versus trainable vision, compares preprocessing and feature fusion, and tests base versus chat LMs, training duration and data diversity. PRISM combines selected findings at 7B/13B scale. optimizationvisionfusionlanguagescalingprism
Figure 2, left design-space panel. One image-to-language interface supports four families of design experiments. Original paper, p. 2 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the photograph in the lower-left box, then follow the arrows from Visual Representation to Projection and upward into Language Model. The snowflake marks the encoder as frozen in the illustrated configuration; flames mark the projector and LM as trainable. Section 2 supplies the input omitted from this overview: text-prompt embeddings join the projected image patches before language generation. The surrounding boxes name independent questions about optimization, image processing, LM initialization and training scale. The displayed 384-by-384 resize is an example configuration, not a universal resolution for every experiment. The crop preserves the scientific design panel; the original figure's code illustration sits outside it. designarchitectureimplementationvisionoptimization
What it supports. The intervention points are concrete: one can change the visual representation or training schedule while retaining the same patch-to-LM interface. Appendix A.2 specifies penultimate ViT features and a two-layer GELU projector. This shared structure makes the study useful for choosing a VLM backbone and understanding where its inputs enter.
Where the evidence stops. The diagram depicts a study framework, not every PRISM variant. Freezing symbols agree with the selected single-stage recipe, but trainable-vision ablations also exist. No action decoder or future-state rollout is specified.
2. Motivation
2.1 The problem and the proposed response
Popular VLMs share a patch-as-token architecture but vary preprocessing, pretrained components and optimization. Subjective model-judged evaluations obscure which choices cause gains. The paper builds twelve objective benchmarks spanning question answering, referring-expression localization and challenge tasks, then varies design choices around a reproduced LLaVa v1.5 baseline. designarchitectureprotocol
2.2 What this reading follows
A visual language model can describe a scene fluently yet struggle to locate a referred object or read a sign. Prismatic asks which parts of the training recipe produce those differences. Its experiments keep a common image-to-language interface while changing optimization, visual features, language backbones and training scale. The resulting PRISM models improve several objective benchmarks and reduce reported training time. Read the paper as a set of conditional design findings: the raw tables distinguish strong localization gains from weaker or negative changes elsewhere. The six visuals below connect the architecture to those comparisons and explain which claims survive closer inspection. designarchitectureoptimizationfusionruntimevqa-prismlocalization-prism
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Foundational work |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
This table preserves the labels recorded at reading time. The current major category is Components of WAMs. View the current classification.
3.1 Evidence-based assessment
Supports the recorded classification
The recorded foundational Language & VLM backbones category fits the architecture and experiments. World-action architecture, prediction paradigm and quadrant are not applicable: the model predicts language conditioned on images, with no learned future-state/action mechanism. A trainable projector and LM do not establish a One Model world-action classification. architecturedesignlimitations
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Follow one patch from pixels to a coordinate string
The architecture has a simple information flow, but its outputs can look more structured than ordinary prose. A vision transformer turns the image into patch features. The projector maps every patch into the language model's embedding space, and the resulting visual prefix is concatenated before the prompt. During supervised training, a referring-expression example pairs that input with a text string of normalized box coordinates. The model therefore learns to generate coordinates through the same conditional language objective used for questions and captions. At evaluation, greedy decoding produces the string and the benchmark evaluates its overlap with the target box. There is no separate action-execution stage in this formulation. This explains why strong localization can make a useful perceptual backbone without demonstrating a robot policy. architecturedataimplementationprotocollimitations
5.2 Separate learned alignment from a separate alignment stage
Eliminating projector-only pretraining does not eliminate the need to learn a useful vision-language mapping. In single-stage training, the randomly initialized projector learns that mapping while the LM also changes, using the instruction mixture. The frozen visual encoder supplies a stable representation throughout. My interpretation is that the experiment questions a particular optimization schedule, not whether image and language features need to be connected. Its aggregate gain coexists with a TextVQA regression, so the answer depends partly on the task profile. The full-vision finetuning ablation asks a different question: whether to change the visual features themselves. Its strong localization degradation motivates keeping these two interventions separate. Repeated-seed tests with explicit data and compute accounting would make the schedule recommendation more robust. optimizationarchitecturestage-tradeofffinetuningstatistics
5.3 Interpret the combined recipe without turning it into a universal rule
PRISM combines improvements that were investigated separately, but their interactions still matter. The paper finds no significant aggregate advantage for instruct-tuned over base LMs, and stronger language-only Mistral performance does not ensure a significant VLM advantage. Two epochs improve aggregate performance over one, while the additional-data result's p-value exceeds the stated significance threshold. Naive resizing also lacks significance over letterboxing at that threshold. My reading is that these findings define useful candidate defaults rather than guarantees. Likewise, channel-wise DINOv2 fusion preserves visual sequence length but changes the information given to the projector. Localization gains support that choice; TextVQA losses constrain it. The final decision should compare raw task scores under a fixed protocol and then ask whether the deployment values the gains more than the regressions. languagescalingstatisticsvisionfusionfusion-numbersprism
5.4 Training and inference
During training
The baseline separates 558K caption-alignment pairs from a 665K instruct mixture containing 40K language-only ShareGPT examples. Single-stage training skips alignment and jointly updates projector and LM with vision frozen. Later design-axis experiments inherit this single-stage baseline. dataoptimization
The consolidated recipe uses two epochs and additional LVIS-Instruct-4V/LRV-Instruct data. Controlled variants must remain separate from these expanded-data models. Base LMs still receive multimodal instruction tuning; 'base' describes their starting checkpoint. prismdatalanguage
Default single-stage settings are AdamW, learning rate 2e-5, batch size 128, weight decay 0.1, gradient clipping at 1.0, cosine decay and 0.03 warmup. Training uses FSDP and BF16 mixed precision. hyperparametersimplementation
Contrastively pretrained CLIP/SigLIP backbones outperform the tested alternatives in aggregate. Increasing resolution helps, but lengthens the patch sequence. Naive resizing is favored over letterboxing without meeting the paper's significance threshold (p=0.0176). The tested backbone sizes and pretraining distributions are not identical. visionstatistics
Base and instruct-tuned LM initialization show no significant aggregate difference (p=0.34854); Mistral's language-only strength also does not yield a significant VLM advantage (p=0.03097). Removing language-only co-training gives no significant benchmark improvement and weakens safeguards in the authors' qualitative probes. languagesafetystatistics
Two epochs improve aggregate performance over one (p=0.00496), after which the authors report a plateau. This is an aggregate pattern: individual benchmark optima differ, and added-data variants show both gains and regressions. scaling
During inference
Evaluation uses greedy decoding and the appropriate model/task prompt. TextVQA in the main results has no external OCR tokens; the appendix's TextVQA+OCR is a distinct condition. RefCOCO-family accuracy uses IoU 0.5, whereas OCID-Ref uses 0.25. protocol
Validation splits are used except GQA test-dev, VSR zero-shot test and POPE's single evaluation split. OCID-Ref probes localization in robotic clutter images, without testing a robot controller. protocollimitations
5.5 Implementation flow
- Encode the image
Resize and normalize for the chosen pretrained ViT, then extract penultimate-layer patch features. PRISM favors naive resizing to square; the broader study also tests letterboxing and cropping. implementationvisionprism
- Fuse and project
For fused variants, concatenate DINOv2 and SigLIP features along channels at corresponding patches. A two-layer GELU MLP maps each patch into the LM embedding space. Fusion preserves image-token count; it still requires both encoders. fusionimplementation
- Condition language generation
Place projected image embeddings before prompt embeddings and generate the answer. Task-specific prompts request phrases, option labels or coordinate strings. There is no action extractor, dynamics rollout or execution-feedback loop in this architecture. architecturedataimplementationprotocol
6. Experiments & results
Prismatic VLMs studies which ingredients matter when a pretrained vision encoder feeds patch embeddings to a language model. Its main contribution is a controlled design investigation, supported by an evaluation suite and author-reported training infrastructure. Single-stage optimization and DINOv2–SigLIP feature fusion improve aggregate performance, while raw tables reveal meaningful task tradeoffs. The resulting PRISM family strengthens visual question answering and localization. These are image-to-text and coordinate-prediction results; the paper does not evaluate executed robot actions or future-state prediction.
6.1 Read the original evidence
Table 2. Raw VQA results separate the strongest scores from the conditions that produced them. Original paper, p. 20 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Begin with the column headers: TextVQA+OCR includes an extra OCR-derived text input, whereas the rightmost TextVQA column uses only the image and question. They are distinct protocols. The top block contains official baseline models; the next contains the authors' reproductions, which are also distinct comparison points. At the bottom, keep the 7B and 13B groups separate, and read the Controlled label on each row rather than assuming every PRISM model used the same training budget. The middle blocks make it possible to trace a final score back to individual design experiments. For the headline VQAv2 comparison, use the full Prism-DINOSigLIP 13B row and official LLaVa 13B. vqa-prismprismprotocolstatistics
What it supports. Full Prism-DINOSigLIP 13B reports VQAv2 accuracy 81.66 versus LLaVa 13B's 78.13, a derived gain of 3.53 percentage points. Yet full Prism-DINOSigLIP 7B scores 52.82 on VizWiz, below LLaVa 7B's 54.24 and its own controlled variant's 59.82. The table supports gains with exceptions.
Where the evidence stops. Full PRISM combines multiple recipe and data changes, so this comparison cannot identify one cause. No seed-wise uncertainty accompanies these entries. Preserve the OCR distinction and the exceptions to Section 5's broad superiority claim.
Table 3. Localization makes both the fusion benefit and the cost of vision finetuning visible. Original paper, p. 21 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the first three columns as the RefCOCO family, then treat OCIDRef separately: Appendix B.1 sets the accuracy threshold to IoU 0.5 for the former and 0.25 for the latter. These are correctness rates for generated bounding boxes. The upper optimization block compares the frozen and finetuned encoders; the Ensembling Visual Features block isolates representation choices; the bottom groups combine recommendations into PRISM. Use the controlled 7B DINOSigLIP row for the featured result and compare both official and reproduced LLaVa rows. Keep the N/A cells for InstructBLIP as missing measurements, rather than interpreting them as zero accuracy. localization-prismfinetuningfusion-numbersprotocollimitationsprism
What it supports. Controlled Prism-DINOSigLIP 7B reaches 73.62 on RefCOCO, against official LLaVa 7B at 55.12 and its reproduction at 60.54. The optimization block gives an equally useful diagnostic: single-stage vision finetuning scores 42.56 compared with 64.08 for frozen vision. These results motivate testing the representation and its update schedule separately.
Where the evidence stops. OCID-Ref contains robotics-oriented clutter images, but the measured output is a box. Neither this table nor its gains establish grasping success, closed-loop control, or prediction of how an action changes the scene.
Table 4. Challenge tasks reveal bottlenecks that a single aggregate score would hide. Original paper, p. 22 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each column as a different constrained decision task. VSR asks about spatial relations, POPE asks whether named objects are present, TallyQA chooses a count and AI2D selects an answer about a diagram. Appendix B.1 evaluates these through multiple-choice accuracy; TallyQA is restricted to sixteen options, zero through fifteen. Many visual-encoder and preprocessing rows remain at VSR 51.47, which should temper claims of broad spatial understanding. Next compare controlled and full PRISM rows within the same LM scale. More training and additional data do not improve every column monotonically. Finally, compare the 13B DINOSigLIP row to its same-scale official LLaVa baseline. challenge-prismfusion-numbersprotocolstatisticssafetylimitationsprism
What it supports. Full Prism-DINOSigLIP 13B improves TallyQA from 64.83 to 70.41 and VSR from 69.07 to 72.18 relative to official LLaVa 13B. Within PRISM 7B, however, moving from controlled to full DINOSigLIP lowers VSR from 66.28 to 59.57. The task profile is more informative than a blanket improvement label.
Where the evidence stops. POPE accuracy is an object-presence benchmark score, not a general safety or hallucination rate. These columns also do not measure extended dialogue. Benchmark success must remain separate from the paper's small qualitative safety probes.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| RefCOCO localization with controlled PRISM 7B Validation split; greedy coordinate-string generation; author-labeled controlled Prism-DINOSigLIP 7B versus official and reproduced LLaVa v1.5 7B. | 73.62 Accuracy at IoU 0.5 (%) | Official LLaVa: 55.12; reproduced LLaVa: 60.54. Derived gains are 18.50 and 13.08 percentage points. This measures localization, not manipulation; Section 5's control label has an unresolved data-accounting boundary. localization-prismprotocolprismoptimization |
| VQAv2 with expanded-data PRISM 13B Validation split; greedy decoding; full Prism-DINOSigLIP 13B with the consolidated training recipe. | 81.66 Official VQA accuracy (%) | Official LLaVa v1.5 13B: 78.13. A derived 3.53-point gain combines architecture, optimization and training-data/time changes; it does not isolate fusion. vqa-prismprotocolprism |
| RefCOCO ablation of DINOv2–SigLIP fusion Single-stage 7B design-axis comparison; 384px naive resizing; validation split and greedy decoding. | 73.86 with fused features Accuracy at IoU 0.5 (%) | SigLIP alone: 61.38. The derived 12.48-point gain is mechanism-relevant. TextVQA without OCR simultaneously falls from 54.87 to 52.18, so the benefit is task-dependent. fusionfusion-numbersprotocol |
| RefCOCO ablation of vision-backbone finetuning Single-stage 7B comparison under the paper's finetuning recipe; validation split. | 42.56 with trainable ViT Accuracy at IoU 0.5 (%) | Frozen ViT: 64.08. A derived 21.52-point loss supports freezing for this recipe. It does not establish that visual adaptation fails with other data or objectives. finetuningprotocol |
| TallyQA counting with expanded-data PRISM 13B Validation split; greedy multiple-choice generation over numbers 0–15; full Prism-DINOSigLIP 13B. | 70.41 Multiple-choice accuracy (%) | Official LLaVa v1.5 13B: 64.83. A derived 5.58-point gain concerns constrained counting responses, not unrestricted numerical reasoning. challenge-prismprotocolprism |
| Training time for controlled PRISM Figure 1's reported training runs on eight A100 GPUs at each LM scale. | 7B: 8.80; 13B: 15.75 Wall-clock training time (hours; lower is better) | LLaVa v1.5: 13.09 and 23.32 hours respectively. These reported system-level times combine implementation and recipe effects. They are distinct from the 20–25% saving attributed specifically to removing alignment. runtimeoptimization |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 4. Removing projector-only alignment saves a stage while improving aggregate performance. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the left radar at 7B and the middle radar at 13B. Gray is the official LLaVa baseline, green the authors' reproduction and orange single-stage training. Each spoke represents a different evaluation; radial positions use normalized scores, while printed numbers give absolute metrics. On the right, the two-stage pipeline first updates only the projector, then updates projector and LM. The single-stage variant starts with the latter configuration and keeps vision frozen. The time bars separate alignment from finetuning: at 7B they show 2.72 and 8.56 hours, respectively, on eight A100 GPUs. These bars concern this ablation rather than the final PRISM system. optimizationstatisticsstage-tradeoffprotocol
What it supports. Section 4.1 reports aggregate improvement with p=0.00558 and a 20–25% training-cost saving from omitting alignment. RefCOCO at 7B rises from 60.54 in the reproduction to 64.08 with single-stage training. The finding supports a simpler default under the evaluated data mixture and optimization settings.
Where the evidence stops. The caption's broad improvement language is aggregate: TextVQA without OCR falls from 46.44 to 44.45 at 7B. Different data and objectives may change the value of alignment; the plot does not establish universal dominance.
Figure 7. Fusing spatial and contrastive features helps localization, with important exceptions. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow the first two panels separately. Their gray outlines are single-encoder baselines; orange lines add DINOv2, with dashed and solid styles distinguishing letterboxing from naive resizing. The third panel is a separate LM-initialization comparison and should not be interpreted as another fusion experiment. Section 4.2 explains that fusion concatenates features along the channel dimension at each patch, before the shared projector. It does not append another sequence of image tokens. Inspect the localization spokes and then TextVQA: the CLIP fusion visibly loses text-reading performance, while the SigLIP fusion gives a stronger overall balance. Use Tables 2–4 for exact values where multiple plotted labels overlap. fusionfusion-numberslanguagestatisticsprotocol
What it supports. For naive-resize SigLIP, Table 3 gives RefCOCO 61.38 without DINOv2 and 73.86 with it; the corresponding OCID-Ref values are 41.49 and 52.82. The authors report an aggregate fusion benefit at p=0.00164. Preserving patch-token count provides a practical reason to test richer per-patch features.
Where the evidence stops. The spatial/semantic explanation is a hypothesis. SigLIP fusion reduces TextVQA from 54.87 to 52.18 and slightly reduces AI2D. Keeping LM token count fixed does not remove the cost of running a second visual encoder.
7. Analysis & limitations
7.1 What the evidence leaves open
The study excludes learned patch-downsampling architectures, larger-scale generalization and rich extended dialogue. Cultural bias, unsafe outputs and hallucination remain. Qualitative safeguards from ShareGPT co-training provide limited evidence of reliability. limitationssafety
Broad superiority language requires qualification: full Prism-DINOSigLIP 7B scores 52.82 on VizWiz versus LLaVa's 54.24. Fusion also worsens TextVQA. InstructBLIP localization is N/A, so it supplies no measured localization baseline. prismvqa-prismfusion-numberslocalization-prism
Appendix B.2 sets p<0.01. Added-data p=0.01459 does not meet that threshold despite the later summary's significance language. Z-score aggregates depend on the compared model pool; the tables do not report seed-wise uncertainty. scalingstatisticsprism
Vision-backbone comparisons confound pretraining objective with data distribution and sometimes size. Spatial/semantic complementarity of fused features is a plausible author explanation, not an isolated causal result. visionfusion
7.2 Questions for discussion
- Which task regressions would make feature fusion unacceptable for a deployment despite better aggregate performance?
- Does eliminating alignment remain beneficial when instruction data, visual backbones or compute budgets change?
8. Reproducibility audit
8.1 Requirements and known gaps
Reproduction needs exact backbone/LM checkpoints, the LLaVa mixture, task prompts, transforms, frozen-module schedule and Table 1 settings. The PDF names implementation libraries but does not pin versions, enumerate every checkpoint revision, specify AdamW betas or fully detail fused-grid alignment. dataimplementationhyperparametersfusion
Clarify Section 5's 'same data and training budget' control against Section 4.1's removal of caption alignment. Report actual examples, optimizer steps and GPU-hours separately; the supplied PDF does not reconcile that accounting explicitly. prismoptimizationruntime
Reader-proposed checks: compare single-stage and two-stage training across repeated seeds with step and compute accounting; separately test aligned versus spatially shuffled DINOv2 fusion under matched projector capacity. Record task-level gains and regressions, not only aggregate scores. optimizationstage-tradeofffusionfusion-numbersstatistics
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Does the single-stage advantage survive matched accounting and repeated seeds?
Reader-proposed experiment, not run: reproduce the CLIP-336px/Vicuna-7B baseline with frozen vision and compare projector-only alignment followed by instruction tuning against immediate joint projector/LM instruction tuning. Fix the instruction mixture, initialization protocol, task prompts and batch order for paired runs, and repeat across at least three seeds. Report both equal instruction-update comparisons and a separate equal GPU-hour comparison; explicitly count whether the 558K alignment examples are used. Measure RefCOCO, TextVQA without OCR and the twelve-task aggregate, with seed variability. The recommendation weakens if the aggregate gain disappears across seeds or if the task regression dominates at matched compute. A stable gain and cost reduction would support the paper's schedule finding. optimizationdataimplementationhyperparametersstage-tradeoffstatisticsruntime
Check 2: Does spatial correspondence explain the DINOv2 fusion gain?
Reader-proposed experiment, not run: hold the SigLIP-384px backbone, naive resizing, LM, data and training schedule fixed. Compare aligned DINOv2–SigLIP channel fusion against DINOv2 features randomly permuted across spatial patch positions, plus a duplicated-SigLIP control with the same projector input width. Keep vision frozen and LM image-token count equal; document grid alignment and actual encoder compute. Evaluate RefCOCO and OCID-Ref at their respective IoU thresholds, alongside TextVQA without OCR. The proposed spatial-complementarity explanation predicts stronger localization from aligned fusion than from shuffled or duplicate features. If all controls match the gain, projector capacity or nonspecific additional features become more plausible explanations. Monitor the text-reading regression rather than accepting aggregate gains alone. fusionfusion-numbersimplementationhyperparametersprotocol
8.3 Reading coverage
Visual audit: Inspected the title/author/affiliation page, all Figures 1–12 and Tables 1–4, plus the supporting method, training, evaluation, limitations and impact text on the declared pages. Pages 3–4 support the equations and protocols; 6–10 support preprocessing, fusion, LM comparisons, scaling, PRISM and limitations; 16–19 support extra diagnostics, data, implementation, hyperparameters and statistics. All six final original crops were viewed, including the corrected table-title margins. Figure 2 arrow directions and frozen/trainable markers were checked against Sections 2 and 4.1 and Appendix A.2. Radar labels were cross-checked against Tables 2–4 rather than treated as one model's complete score vector. Reference pages 11–15 were read as text. Separate supplements, code and checkpoints remain outside the supplied reading.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 16, 17, 18, 19, 20, 21, 22. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract and title metadata (p. 1)
- Sections 1–3: motivation, architecture, implementation and evaluation (pp. 1–4)
- Sections 4.1–4.4: optimization, vision, language models and scaling (pp. 4–8)
- Sections 5–7: PRISM, limitations and conclusion (pp. 8–9)
- Impact Statement and Acknowledgements (p. 10)
- References (pp. 11–15)
- Additional Figures 10–12 (pp. 16–17)
- Appendix A.1–A.3: datasets, implementation and hyperparameters (pp. 17–18)
- Appendix B.1–B.3: evaluation, significance and exhaustive results (pp. 19–22)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- The extraction limitation was addressed by inspecting the original PDF figures and tables; all eight supplied text chunks were read individually.
- Identity matches the catalog title and six authors. The supplied artifact is the ICML 2024/PMLR 235 edition; no separate revision identifier or revision date is established, and no other edition was compared.
- Code, external resources and model checkpoints were not inspected; no experiments were reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
identityPDF p. 1, title, author line, affiliation footnote and proceedings notice
The title and six authors match the catalog apart from title whitespace. Credits are Siddharth Karamcheti, Suraj Nair, Ashwin Balakrishna, Percy Liang, Thomas Kollar and Dorsa Sadigh. Affiliations are Stanford University Computer Science and Toyota Research Institute. The artifact identifies ICML 2024, PMLR 235; no separate revision identifier is given.
Go to primary source ↓designPDF p. 2, Introduction and Figure 2
The study varies optimization, image processing/visual representations, language models and training time/data. The diagram routes visual features through a projector to an LM, marking vision frozen and projector/LM trainable. Training code, evaluation code and checkpoints are author-announced resources.
Go to primary source ↓architecturePDF p. 3, Section 2, Model Architecture through Language Model
An image is encoded by V_omega into patch features p_img, projected by F_psi into image embeddings e_img, and concatenated before prompt embeddings. LM_theta generates text; training minimizes negative log likelihood of target output given image and prompt.
Go to primary source ↓implementationPDF p. 18, Appendix A.2, architecture components and prompting
The implementation uses PyTorch FSDP, BF16 mixed precision, backbone-specific image normalization, penultimate ViT features and a two-layer GELU MLP projector. Image embeddings precede prompt embeddings. Vicuna uses a chat system prompt and USER/ASSISTANT format; base LMs use In/Out without that system prompt.
Go to primary source ↓dataPDF pp. 3 and 17–18, Section 2 Pretraining Dataset and Appendix A.1
LLaVa v1.5 data comprise 558K alignment image-caption pairs and a 665K instruct mixture including synthetic conversations, VQA, captioning, referring-expression tasks and 40K language-only ShareGPT examples. Bounding boxes are normalized coordinates generated as text.
Go to primary source ↓optimizationPDF p. 5, Section 4.1 and Figure 4
Two-stage training first trains only the projector, then projector plus LM; vision stays frozen. Single-stage training omits alignment, improves aggregate performance with p=0.00558, and saves a reported 20–25% training cost. Figure 4 shows 7B alignment/finetuning times 2.72/8.56 hours and 13B times 4.85/15.36 hours on eight A100s.
Go to primary source ↓finetuningPDF p. 5, Section 4.1, Figure 5; p. 21, Table 3, Reproduction & Optimization Procedure / RefCOCO
Finetuning vision degrades aggregate performance, p=0.00381. Frozen single-stage RefCOCO accuracy is 64.08; finetuned single-stage is 42.56 and finetuned multi-stage is 19.24. The authors speculate about data scale/diversity and objective mismatch.
Go to primary source ↓visionPDF p. 6, Section 4.2 and Figure 6, including footnote 2; p. 7, opening continuation
Contrastive vision backbones outperform the tested alternatives in aggregate. The 224px SigLIP comparison uses a 400M-parameter shape-optimized model versus 307M ViT-L alternatives. Naive resizing versus letterboxing has p=0.0176; higher resolution has p=6.05e-4. Doubling image resolution quadruples patch count, yielding sixteen-fold traditional attention complexity.
Go to primary source ↓fusionPDF p. 7, Section 4.2, Ensembling Different Visual Representations and Figure 7
Features concatenate along channels per patch; the projector input grows while patch-token count stays fixed. DINOv2+SigLIP improves aggregate performance, p=0.00164; DINOv2+CLIP does not, p=0.37313. Spatial versus semantic complementarity is an author hypothesis.
Go to primary source ↓fusion-numbersPDF pp. 20–22, Tables 2–4, Ensembling Visual Features, SigLIP and DINOv2 + SigLIP 384px (Naive Resize) rows
SigLIP alone versus fused: RefCOCO 61.38/73.86, OCIDRef 41.49/52.82, TextVQA without OCR 54.87/52.18, VSR 51.47/51.55, POPE 86.52/88.30 and AI2D 54.89/54.82. Table 2 reports CLIP versus DINOv2+CLIP naive-resize TextVQA 49.66/15.67.
Go to primary source ↓languagePDF pp. 7–8, Section 4.3; p. 16, Figure 11; p. 17, Figure 12
Base versus instruct-tuned LM performance is not significantly different in aggregate, p=0.34854. Mistral versus Llama-2 is also nonsignificant at the stated threshold, p=0.03097. Selected qualitative examples show verbosity and hallucination differences, not population hallucination rates.
Go to primary source ↓safetyPDF p. 8, Section 4.3 and Figure 8; p. 10, Risks and Known Biases
Removing ShareGPT co-training has no significant aggregate benchmark benefit, p=0.13655, but adversarial examples show reduced safeguards, particularly for base LMs. Safety probing is cursory; cultural/English bias, unreliable outputs and image-based vulnerabilities remain.
Go to primary source ↓scalingPDF p. 8, Section 4.4; p. 16, Figure 10; pp. 20–22, Tables 2–4, Scaling Train Time and Scaling Data
Two epochs improve aggregate performance over one, p=0.00496, with a reported aggregate plateau thereafter. Added LVIS-Instruct-4V and LRV-Instruct produce p=0.01459; LRV alone gives VSR 64.08 versus base 51.47, whereas both additions give 54.91. Improvements vary by task.
Go to primary source ↓prismPDF pp. 8–9, Section 5 and Figure 9
PRISM combines single-stage training, high-resolution representations including fused DINOv2/SigLIP, naive resizing, base LMs, two epochs and additional data. Controlled models are described as using the same data and training budget as LLaVa. Section 5 broadly claims uniform superiority.
Go to primary source ↓protocolPDF p. 4, Section 3 final paragraph; p. 19, Appendix B.1
Evaluation uses validation splits except GQA test-dev, VSR zero-shot test and POPE's single evaluation split. Greedy decoding and task-specific trigger prompts are used. RefCOCO-family accuracy uses IoU 0.5; OCID-Ref uses 0.25. Main-paper TextVQA excludes OCR tokens; Table 2 separately includes TextVQA+OCR. Challenge tasks use multiple-choice accuracy, with TallyQA options 0–15.
Go to primary source ↓statisticsPDF p. 4, Experiment Design; p. 19, Appendix B.2
Task metrics are normalized using each task's mean and standard deviation across models. Global scores average twelve task Z-scores. Comparisons use normalized differences between base/alternate model pairs and a one-sided Fisher T-test, with p<0.01. Seed-wise confidence intervals do not accompany the exhaustive tables.
Go to primary source ↓vqa-prismPDF p. 20, Table 2, Official Models and Prism 7B/13B blocks
Prism-DINOSigLIP 13B reports VQAv2 81.66, GQA 66.13 and TextVQA without OCR 57.08, versus official LLaVa v1.5 13B 78.13, 63.17 and 48.99. Prism-DINOSigLIP 7B reports VizWiz 52.82 versus official LLaVa 7B 54.24 and controlled Prism-DINOSigLIP 7B 59.82.
Go to primary source ↓localization-prismPDF p. 21, Table 3, Official Models, Reproduction and Prism blocks
Prism-DINOSigLIP 7B (Controlled) reports RefCOCO 73.62 and OCIDRef 50.56; official LLaVa 7B reports 55.12 and 35.07, while its reproduction reports 60.54 and 41.75. Full Prism-DINOSigLIP 13B reports RefCOCO 79.39 and OCIDRef 54.62. InstructBLIP localization is N/A.
Go to primary source ↓challenge-prismPDF p. 22, Table 4, Official Models and Prism blocks
Prism-DINOSigLIP 13B scores VSR/POPE/TallyQA/AI2D 72.18/88.07/70.41/57.96 versus official LLaVa 13B 69.07/87.10/64.83/57.13. Controlled/full Prism-DINOSigLIP 7B score VSR 66.28/59.57 and POPE 88.28/88.12.
Go to primary source ↓runtimePDF p. 1, Figure 1 training-time bars; p. 3, Training Implementation & Verification; p. 10, Training and Finetuning Accessibility
Figure 1 reports eight-A100 training times of LLaVa/controlled PRISM 13.09/8.80 hours at 7B and 23.32/15.75 hours at 13B. Separately, Section 2 reports 20% faster implementation step times on the same AWS p4de.24xlarge with eight A100s.
Go to primary source ↓hyperparametersPDF p. 18, Appendix A.3 and Table 1
Single-stage settings are batch size 128, gradient norm 1.0, weight decay 0.1, learning rate 2e-5, AdamW, cosine decay and warmup ratio 0.03. Alignment-only training changes batch size to 256 and learning rate to 1e-3.
Go to primary source ↓limitationsPDF p. 9, Section 6; p. 10, Risks and Known Biases
The study excludes learned patch-downsampling architectures and leaves larger-scale generalization open. Objective benchmarks do not cover rich extended dialogue. Robotic control-policy integration is future work. Cultural bias, toxicity and hallucination remain limitations.
Go to primary source ↓stage-tradeoffPDF pp. 20–21, Tables 2–3, LLaVa v1.5 7B (Reproduction) and Single-Stage 7B rows
Removing alignment changes RefCOCO from 60.54 to 64.08 while TextVQA without OCR changes from 46.44 to 44.45. Aggregate improvement does not imply each task improves.
Go to primary source ↓8.5 Primary sources
Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models ↗
PDF · 14,024 extracted words
Source fingerprint
5b19876fd49109700321a90abf00025227b17d82131d052919645ea7b2245003