Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond
1. Paper overview
In one sentence: Qwen-VL combines spatially informed visual compression with staged language supervision to read, describe and localize image content, while its ablations leave downstream causal benefits only partly established. e-probleme-architecturee-interfacee-sfte-text-vqae-queriese-attention
| At a glance | What to know |
|---|---|
| Research problem | Source description The authors target coarse visual understanding in open LVLMs: useful assistants must read text and identify specific regions, as well as describe whole images. Qwen-VL combines these tasks within a shared image-and-language interface. e-problem |
| Core mechanism | Source description A position-aware adapter connects OpenCLIP ViT-bigG to Qwen-7B, keeping the visual sequence entering the language model at 256 features. e-architecture |
| A key reported result | Text-oriented VQA on TextVQA: Qwen-VL 63.8; Qwen-VL-Chat 61.5. VQA Score, higher is better. Validation split; evaluated Qwen-VL and its SFT variant. InstructBLIP 50.7; mPLUG-DocOwl 52.6. Supports text-reading capability; TextVQA also appears in multi-task training, so this is not an unseen-dataset claim. e-text-vqae-benchmark-protocolse-multitask |
| Reading caution | Reader analysis The query-count experiment uses ViT-L/14 at 224×224 and smoothed training-loss curves. It motivates 256 queries, but does not isolate their downstream benefit in the final ViT-bigG model or ablate positional encoding. e-queriese-architecture |
Core contributions
- Source description
A position-aware adapter connects OpenCLIP ViT-bigG to Qwen-7B, keeping the visual sequence entering the language model at 256 features. e-architecture
- Source description
A staged curriculum combines large-scale bilingual image-text alignment, seven-task pre-training and dialogue tuning; bounding boxes are learned as ordinary text with structural delimiters. e-interfacee-pretraininge-multitaske-sft
- Author claim
The authors report broad improvements over compared generalist models, but their tables retain exceptions and stronger specialist results. e-caption-vqae-text-vqae-grounding
Figure 3. One visual-language pathway is optimized under three different training regimes. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each column from the camera upward: ViT produces image features, CrossAttn compresses them using learnable query embeddings, and QwenLM consumes the resulting sequence alongside text. Then read left to right to track optimization. The snowflake on stage 1 QwenLM means the language model is frozen; the flames on ViT and CrossAttn mark trained components. Stage 2 updates all three at higher image resolution. Stage 3 freezes ViT while tuning the adapter and language model for dialogue. These symbols agree with Sections 3.1–3.3. The three columns depict successive training regimes, rather than three separately executed inference branches. e-architecturee-pretraininge-multitaske-sfte-format
What it supports. The architecture stays conceptually stable while training changes what adapts. Stage 1 aligns visual input with an existing language model; stage 2 exposes the whole model to fine-grained tasks; stage 3 produces the conversational variant. In the source recipe, resolution rises from 224×224 to 448×448 while adapter output remains 256 features.
Where the evidence stops. The diagram omits the adapter's 2D positional details and the target-token supervision mask. Those must be read in Section 2.1 and Appendix B. It does not establish which stage independently causes a benchmark improvement.
2. Motivation
2.1 The problem and the proposed response
The authors target coarse visual understanding in open LVLMs: useful assistants must read text and identify specific regions, as well as describe whole images. Qwen-VL combines these tasks within a shared image-and-language interface. e-problem
2.2 What this reading follows
A language model can describe an image without reliably reading its small text or identifying a requested region. Qwen-VL addresses these finer tasks by connecting a visual encoder to a language model through a compact, position-aware adapter, then training a common textual interface for answers and coordinates. Read the architecture together with the training stages: different modules are frozen at different times, and the dialogue model is a later checkpoint. The results show broad capability, but specialist gaps and nonuniform effects of instruction tuning matter. The appendix explains practical choices about visual tokens and attention through training-loss experiments whose scope is narrower than the final benchmark claims. e-probleme-architecturee-interfacee-sfte-text-vqae-queriese-attention
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Foundational work |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
This table preserves the labels recorded at reading time. The current major category is Components of WAMs. View the current classification.
3.1 Evidence-based assessment
Supports the recorded classification
The foundational Language & VLM backbones classification is supported. A vision encoder, adapter and language generator perform perception and grounding. Joint multimodal training supplies neither joint future/action prediction nor inverse dynamics, so the catalog's architecture, prediction-paradigm and quadrant fields remain Not applicable. e-architecturee-interfacee-multitaske-future
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Why compression and coordinates must be considered together
The image pathway has two competing demands. Fine detail benefits from a denser patch grid, but a long visual sequence burdens the language model. Qwen-VL separates these concerns: increasing input resolution expands the ViT sequence, while the adapter holds the sequence entering QwenLM at 256 features. The source adds 2D absolute positions to query-key pairs because compression could otherwise discard spatial detail. Coordinates then become a language target through <box> and <ref>, so localization depends on what this compressed representation preserves. A reader inference is that query count, positional encoding and image resolution should be studied jointly: enough tokens for a caption may still be inadequate for a small word or tightly specified region. Figure 7 motivates the chosen count through optimization, but does not independently test that spatial-information hypothesis. e-architecturee-interfacee-queries
5.2 How the same sequence format teaches opposite spatial tasks
Appendix B makes the learning signal concrete. In referring grounding, the image and phrase belong to the unsupervised prefix, while the box coordinates form the supervised continuation. Grounded captioning reverses the direction: coordinates are supplied and the model predicts a region description. OCR adds text linked to quadrilateral coordinates. These examples explain how text-token training can teach both where an entity is and what a region contains, without defining robot actions. The three stages then control which parameters absorb the signal. Image-text pre-training adapts vision and the adapter to a frozen language model; multi-task training updates the entire model; supervised dialogue tuning freezes vision and changes the adapter and language model. The ChatML example marks supervised special tokens and answers separately from role names and question prompts. e-formate-interfacee-pretraininge-multitaske-sft
5.3 Separate capability evidence from explanations of why it works
The benchmark tables establish that the complete trained system can caption, answer visual questions and generate useful referring regions. They do not separate the contributions of resolution, data cleaning, OCR supervision and the adapter. The appendix experiments narrow the question to convergence and iteration time, which is useful engineering evidence but a different outcome from held-out task accuracy. The language-retention claim needs a further distinction: Qwen-VL starts from an intermediate Qwen-7B checkpoint. Table 11 improves on that initialization for MMLU and C-Eval, while the final released language model is stronger. Because the intermediate CMMLU score is absent and the paper does not provide a no-text-mixing control, neither universal preservation nor a causal attribution to text mixing follows. A reproduction should keep checkpoint identity, training controls and evaluation metrics explicit. e-caption-vqae-text-vqae-groundinge-queriese-attentione-text-retention
5.4 Training and inference
During training
Stage 1 freezes the LLM and trains vision plus adapter at 224×224 using text-token cross-entropy. The reported cleaned pool is 1.4B pairs, 77.3% English and 22.7% Chinese. Training processes approximately 1.5B samples over 50,000 steps with global batch 30,720. e-pretraininge-hyperparameters
Stage 2 trains the entire model at 448×448. Captioning, VQA, grounded captions, referring grounding, grounded captioning, OCR and pure-text autoregression share the text objective. Table 3 includes 24.8M OCR and 7.8M pure-text samples. Same-task examples are packed into sequences of length 2048. e-multitask
Stage 3 yields Qwen-VL-Chat from 350k instruction examples mixing multimodal and pure-text dialogue. Vision is frozen; adapter and LLM train. ChatML marks turns, image-order strings distinguish pictures, and supervision covers answers and special tokens rather than role names or questions. e-sfte-format
Table 8 specifies AdamW and bfloat16; stages 2/3 run 19k/8k steps with global batches 4096/128. Peak learning rates across stages are 2×10⁻⁴, 5×10⁻⁵ and 10⁻⁵, with cosine decay. e-hyperparameters
During inference
Images pass through the encoder and adapter before language generation. Caption evaluation uses greedy decoding with the printed prompt “Descripe the image in English:”. VQAv2, OKVQA, GQA and VizWiz use open-ended greedy answers; ScienceQA instead selects the highest-confidence allowed option. e-architecturee-caption-vqa
Few-shot examples are randomly sampled, without RICES retrieval. SEED-Bench video questions use four sampled frames. These are comprehension evaluations; no future-image rollout, action decoder or executed control loop is specified. e-fewshote-instructionse-interface
5.5 Implementation flow
- Encode and preserve location
ViT-bigG, initialized from OpenCLIP, extracts stride-14 patch features. A randomly initialized single-layer cross-attention adapter uses learnable queries and image-feature keys; 2D absolute position encodings enter query-key pairs. Table 1 reports 1.9B vision, 0.08B adapter, 7.7B language and 9.6B total parameters. e-architecture
- Expose visual information to language
The adapter compresses features to 256 positions between <img> and </img>. The language initialization is an intermediate Qwen-7B checkpoint, not its final release. A larger patch grid therefore does not directly lengthen the compressed visual sequence. e-architecturee-interfacee-text-retention
- Serialize spatial tasks
Coordinates are normalized into [0,1000) and tokenized as text inside <box> delimiters; <ref> marks associated descriptions. Referring grounding predicts coordinates from a phrase, while grounded captioning predicts a description given coordinates. OCR examples use <quad> for quadrilaterals. e-interfacee-format
6. Experiments & results
Qwen-VL turns a language model into a multilingual image-understanding system through a visual encoder, a position-aware compression adapter, and three training stages. It produces captions, answers and textual bounding boxes. Strong generalist benchmark results coexist with specialist gaps and limited causal evidence for individual design choices.
6.1 Read the original evidence
Table 4. Strong generalist scores come with task-dependent effects of dialogue tuning. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the two Qwen rows, then compare vertically within a single column. The caption columns report CIDEr; the VQA columns use their respective benchmark scores. Only selected headers explicitly say zero-shot, so that label should not be extended across the table. Appendix D identifies Flickr30K as karpathy-test and VQAv2 as test-dev. Section 4.1 also separates greedy open-ended answers from constrained option selection on ScienceQA. The gray specialist row retains useful context, even when it weakens a broad superiority claim. Dashes indicate missing reported entries and cannot be treated as zero performance. e-caption-vqae-benchmark-protocolse-sft
What it supports. Qwen-VL scores 85.8 CIDEr on Flickr30K, against 82.8 for InstructBLIP with Vicuna-13B and 67.2 for Flamingo-80B. Its VQAv2 score is 79.5 versus Shikra's 77.36, although the listed specialist reaches 86.1. Qwen-VL-Chat falls to 81.0 on Flickr30K and 78.2 on VQAv2.
Where the evidence stops. These are reported comparisons across models with different training histories. They do not isolate the adapter or control compute and data. CIDEr and VQA scores are different measurements and should not be averaged into one capability score.
Table 5. Reading capability varies across natural images, documents, charts, diagrams and book covers. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow the columns as distinct evaluation problems. TextVQA uses a validation VQA Score; DocVQA uses test ANLS; ChartQA uses test relaxed exact match; AI2D and OCR-VQA use test exact match. Those definitions come from Table 9 and are absent from this compact result table. Compare each Qwen row with the relevant baseline in the same column. The gray PALI-X row explicitly describes single-task fine-tuning without an OCR pipeline, which limits direct attribution to model architecture. Qwen-VL is the stage-2 checkpoint and Qwen-VL-Chat is its instruction-tuned successor, so their difference is useful but not uniform. e-text-vqae-benchmark-protocolse-multitaske-sft
What it supports. Qwen-VL reaches 63.8 on TextVQA versus InstructBLIP's 50.7 and DocOwl's 52.6. On DocVQA, however, its 65.1 remains below Pix2Struct-Large's 76.6. Chat tuning raises ChartQA from 65.7 to 66.3 while lowering the other four displayed Qwen scores, illustrating a mixed benchmark tradeoff.
Where the evidence stops. These datasets overlap by name with the multi-task training sources; the table is not evidence of zero-shot transfer to unseen dataset families. The results also do not isolate higher resolution, OCR data or positional encoding as the cause of improvement.
Table 6. Generating a region as text supports competitive visual localization. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the nested headers before comparing scores: RefCOCO and RefCOCO+ each have validation, test-A and test-B columns, while RefCOCOg has validation and test. The final GRIT column is a separate benchmark. Each value measures the referring-expression task: identify an image region from language. Section 2.2 explains that Qwen-VL expresses that region as normalized coordinate text, rather than using a separate positional vocabulary. Compare within one split, retaining the gray specialist rows. The missing GRIT score for Qwen-VL-Chat remains missing; other split results cannot fill it. Table 9 labels the metric Accuracy without supplying its geometric acceptance rule. e-interfacee-groundinge-benchmark-protocolse-multitask
What it supports. On RefCOCO test-B, Qwen-VL reports 85.34 compared with Shikra-13B's 81.81, while the specialist UNINEXT-H reaches 91.46. The GRIT column supplies another boundary: Qwen-VL's 78.22 is slightly below Unified-IO's 78.61. Competitive grounding therefore coexists with visible exceptions to being the best generalist.
Where the evidence stops. RefCOCO-family data are part of multi-task training, and the compared systems are not controlled for training data. These scores concern localization in images; they provide no measurement of whether a robot can reach, grasp or manipulate the identified object.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Zero-shot image captioning on Flickr30K Karpathy-test; greedy caption generation after multi-task pre-training. | Qwen-VL 85.8; Qwen-VL-Chat 81.0. CIDEr, higher is better | InstructBLIP (Vicuna-13B) 82.8; Flamingo-80B 67.2. A strong reported captioning result; the comparison does not match training data or compute. e-caption-vqae-benchmark-protocols |
| General visual question answering on VQAv2 Test-dev; open-ended greedy answer generation. | Qwen-VL 79.5; Qwen-VL-Chat 78.2. VQA Score, higher is better | Shikra (Vicuna-13B) 77.36; specialist PALI-X-55B 86.1. The reported generalist gain does not surpass the listed specialist. e-caption-vqae-benchmark-protocols |
| Text-oriented VQA on TextVQA Validation split; evaluated Qwen-VL and its SFT variant. | Qwen-VL 63.8; Qwen-VL-Chat 61.5. VQA Score, higher is better | InstructBLIP 50.7; mPLUG-DocOwl 52.6. Supports text-reading capability; TextVQA also appears in multi-task training, so this is not an unseen-dataset claim. e-text-vqae-benchmark-protocolse-multitask |
| Document VQA on DocVQA Test split; document-image questions. | Qwen-VL 65.1; Qwen-VL-Chat 62.6. ANLS, higher is better; paper's reported scale | Pix2Struct-Large 76.6; mPLUG-DocOwl 62.2. A clear exception to universal generalist superiority. e-text-vqae-benchmark-protocols |
| Referring-expression grounding on RefCOCO Test-B; predict a target region from its description. | Qwen-VL 85.34; Qwen-VL-Chat 84.51. Accuracy, higher is better | Shikra-13B 81.81; specialist UNINEXT-H 91.46. Shows localization rather than action execution; the supplied benchmark summary does not specify its geometric acceptance threshold. e-groundinge-benchmark-protocols |
| Bilingual instruction following on TouchStone English and Chinese benchmark evaluations of Qwen-VL-Chat. | English 645.2; Chinese 401.2. GPT-4 Score, higher is better; not a percentage | mPLUG-Owl English 605.4; VisualGLM Chinese 247.1. The compared language coverage is sparse; these aggregate judged scores are not physical task-success rates. e-instructionse-benchmark-protocols |
| Window versus global attention training cost Appendix E.3 resolution/attention ablation; hardware and batch configuration not specified for timings. | 448×448: global 10, window 9; 896×896: global 60, window 25. Seconds per training iteration, lower is faster | Figure 8 shows lower loss for global attention at each matched resolution. The selected 448 global configuration trades a small timing penalty for lower loss. Cross-resolution speed differences are much larger; downstream accuracy is not ablated. e-attention |
| Pure-text capability after multimodal training Table 11; exact split and prompting/shot protocol unspecified. | Qwen-VL 50.7 / 51.1. Reported MMLU / C-Eval benchmark scores | Intermediate initialization 49.9 / 48.5; final released Qwen-7B 58.2 / 63.5. Improvement is relative to the actual intermediate initialization. Its CMMLU baseline is missing, and no universal preservation claim follows. e-text-retention |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 7. An initially lower loss does not settle the visual-token budget. Original paper, p. 22 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use the legend to interpret L64, L144, L256 and L400 as compressed feature lengths, set by the number of learned queries. The left panel covers initialization and the first 50 training steps; the right examines a later interval with a tightly zoomed loss axis. Do not compare vertical distances across the panels. The caption states that these curves are smoothed. Appendix E.2 specifies ViT-L/14 at 224×224 for this experiment, yielding 256 patch features before compression. Its discussion then considers the final higher-resolution setting, which produces 1024 patch features, when choosing the adapter's 256-query budget. e-queriese-architecture
What it supports. Fewer queries begin with lower loss, but L256 shows favorable later convergence in the displayed experiment. The authors select 256 queries while also considering information loss at the higher-resolution training stage. This supports a practical optimization rationale; it does not directly quantify how much small-text or spatial information survives compression.
Where the evidence stops. The ablation uses a different visual backbone from the final ViT-bigG model. It reports smoothed training loss without downstream task scores or uncertainty intervals, and it does not test whether positional encoding is responsible for the observed ordering.
Figure 8. Attention choices must be judged against elapsed time as well as optimization steps. Original paper, p. 23 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Match colors through the legend: red and cyan compare window and global attention at 448×448; blue and green compare them at 896×896. Lower loss is better, but the horizontal axis counts steps rather than elapsed time. Appendix E.3 describes the window alternative as mostly local attention with occasional global layers. Read Table 10 on the preceding page alongside the curves: 448 global/window iterations take 10/9 seconds, whereas 896 global/window iterations take 60/25 seconds. A lower curve at higher resolution therefore comes with a substantial time cost. These are training-loss comparisons rather than downstream VQA or grounding results. e-attention
What it supports. The paper chooses global attention at 448×448: its loss is better than the matched window alternative, with only a small reported per-iteration timing difference. Although 896 global attention obtains the lowest late loss, it is substantially slower. The 896 window option also costs 2.5 times the reported iteration time of 448 global attention.
Where the evidence stops. Appendix E.3's statement that attention speeds are similar fits the 448 comparison, not the 896 table entries. Hardware and timing configuration are unspecified, so the absolute seconds cannot be transferred to another setup without measurement.
7. Analysis & limitations
7.1 What the evidence leaves open
The query-count experiment uses ViT-L/14 at 224×224 and smoothed training-loss curves. It motivates 256 queries, but does not isolate their downstream benefit in the final ViT-bigG model or ablate positional encoding. e-queriese-architecture
Cross-model results lack matched data/compute and uncertainty estimates. SFT improves some columns but lowers others; Figure 4 also does not show uniformly increasing few-shot performance. e-caption-vqae-text-vqae-groundinge-fewshot
The conclusion leaves speech/video integration, higher resolution and image/speech generation to future work. Four-frame video question answering does not establish a predictive dynamics model. e-futuree-instructions
7.2 Questions for discussion
- Would query counts selected by training loss remain optimal for small-text reading and grounding at fixed compute?
- Does pure-text mixing preserve capabilities beyond the few benchmarks and intermediate checkpoint reported here?
8. Reproducibility audit
8.1 Requirements and known gaps
Reconstruction requires the intermediate Qwen checkpoint, OpenCLIP initialization, private corpus components, task mixture and Table 8 schedule. The paper supplies optimizer/parallelism settings but no GPU model/count, software versions or complete data-release specification. e-text-retentione-architecturee-pretraininge-multitaske-hyperparameters
Appendix A documents cleaning categories, greedy removal of recursive grounding labels, and synthetic OCR construction. Numeric aspect-ratio, size, CLIP-score and text-length thresholds remain unstated. Appendix E.1 calls stage 1 one epoch; exact reconciliation with rounded pool/processed-sample totals is unavailable. e-cleaninge-ocre-convergencee-pretraining
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Does positional compression preserve fine detail?
Reader-proposed check: train four matched variants with 64 versus 256 queries and with versus without the adapter's 2D positional encoding. Fix the backbone initialization, 448×448 resolution, data order, optimizer and number of examples; separately report wall time and memory. Evaluate RefCOCO test-B accuracy and TextVQA validation VQA Score, with predeclared small-region and small-text subsets and repeated seeds. A larger benefit of positions or extra queries on these subsets would support the information-preservation explanation. Unchanged downstream scores despite lower loss would weaken the use of Figure 7 as a selection criterion. This is a proposed controlled study, not a reported or executed reproduction. e-architecturee-queriese-interfacee-benchmark-protocols
Check 2: Does pure-text mixing prevent forgetting relative to the actual initialization?
Reader-proposed check: preserve the exact intermediate Qwen-7B checkpoint and record its MMLU, CMMLU and C-Eval scores under one fixed prompt/shot protocol. Compare multimodal training with the reported pure-text mixture against a run with that component removed, holding multimodal examples and update count fixed and reporting the resulting token-budget difference. Add a compute-matched control to separate extra text exposure from additional training. Measure visual tasks as well as language tasks before and after each stage. Consistently smaller language-score declines in the mixed run would support the stated retention mechanism; equal retention without mixing would weaken that attribution. The missing intermediate CMMLU value must be measured rather than inferred. e-text-retentione-multitaske-sfte-hyperparameters
8.3 Reading coverage
Visual audit: The title/author/version page, every original figure (1–8), every table (1–11), method/training pages, colored Appendix B supervision examples, and relevant appendix conclusions were rendered and visually inspected. Six final original crops were also individually inspected, preserving axes, legends, headers and specialist rows. Figure 3 arrow directions and freeze/train symbols were checked against Sections 2–3; the Appendix B color mask was checked against its prose. All retained claim-supporting method, numerical, protocol and reproduction pages are declared here. Reference-only pages 11–15 were read in the complete text but not rendered; separate supplements and linked resources were outside the supplied material.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 16, 17, 18, 19, 20, 21, 22, 23, 24. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract; Sections 1–2.2: introduction, architecture and interfaces
- Sections 3–3.3: all three training stages
- Sections 4–4.5: captioning, general/text VQA, grounding, few-shot and instruction-following evaluation
- Sections 5–6: related work, conclusion and future work
- References, PDF pp. 10–15
- Appendix A.1–A.4: image-text, VQA, grounding and OCR data
- Appendix B.1–B.2: pre-training and dialogue formats
- Appendices C–D: hyperparameters and evaluation protocols
- Appendix E.1–E.4: convergence, query count, attention and pure-text evaluation
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- The supplied artifact is arXiv:2308.12966v3, dated 13 October 2023. Its title and all nine authors match the catalog. The catalog submission date is 24 August 2023; earlier revisions were not supplied, so changes between editions were not compared.
- The acquisition noted that text extraction does not reconstruct figure images. Direct PDF inspection addressed this omission for all eight figures and all eleven tables.
- Separate supplemental material availability has not been fully verified; no separate supplement was supplied.
- Code, linked resources and underlying datasets were not inspected; no experiments were reproduced. References were read as part of this paper, not as independently verified sources.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e-identityPDF p. 1, title block, author credits and arXiv margin
Exact catalog title; Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou and Jingren Zhou; Alibaba Group; v3, 13 October 2023. First three authors contribute equally; Chang Zhou is corresponding author.
Go to primary source ↓e-problemPDF pp. 1–3, Abstract and Section 1
Motivates multilingual, multi-image, fine-grained visual understanding beyond coarse descriptions; distinguishes Qwen-VL from instruction-tuned Qwen-VL-Chat.
Go to primary source ↓e-architecturePDF pp. 3–4, Section 2.1, Table 1 and Figure 3
OpenCLIP ViT-bigG with stride 14; randomly initialized single-layer cross-attention; 256 learned-query outputs; 2D absolute query-key positions; 9.6B reported total parameters.
Go to primary source ↓e-interfacePDF p. 4, Section 2.2
Image feature delimiters; boxes normalized within [0,1000), represented as two coordinate pairs and tokenized as text; box and reference delimiters associate regions and descriptions.
Go to primary source ↓e-pretrainingPDF p. 5, Section 3.1 and Table 2
5B original and 1.4B cleaned pairs; English/Chinese shares 77.3%/22.7%; frozen LLM, trainable vision/adapter, 224 resolution, text cross-entropy, about 1.5B processed samples.
Go to primary source ↓e-multitaskPDF pp. 5–6, Section 3.2 and Table 3
Seven tasks; 448 resolution; all modules train; 2048-token same-task packing. Captioning 19.7M, VQA 3.6M, grounding 3.5M, referring/grounded-caption tasks 8.7M each, OCR 24.8M, text 7.8M; TextVQA is listed.
Go to primary source ↓e-sftPDF p. 6, Section 3.3 and evaluation preamble; p. 4, Figure 3
350k instruction data; mix multimodal and text dialogue; freeze visual encoder, update adapter/LLM; Qwen-VL denotes stage 2 and Qwen-VL-Chat stage 3.
Go to primary source ↓e-formatPDF pp. 18–19, Appendix B.1–B.2 and colored format boxes
Black prefixes are not supervised, blue target labels are; grounding and inverse description examples; quadrilateral OCR labels; Picture id and ChatML turns; answer/special-token SFT supervision.
Go to primary source ↓e-hyperparametersPDF p. 20, Appendix C and Table 8
AdamW, bfloat16, cosine schedule; step/batch/LR settings for all stages; optimizer sharding, no activation checkpointing, model parallelism 2 in stages 2/3. No GPU model/count or software versions appear.
Go to primary source ↓e-caption-vqaPDF p. 7, Table 4, Flickr30K/VQAv2 and other columns; Section 4.1
Flickr30K: Qwen-VL 85.8, Chat 81.0, InstructBLIP 82.8, Flamingo-80B 67.2. VQAv2: 79.5/78.2, Shikra-13B 77.36, PALI-X 86.1. Specifies greedy captions/answers and constrained ScienceQA choices.
Go to primary source ↓e-text-vqaPDF pp. 7–8, Section 4.2 and Table 5
TextVQA 63.8/61.5 vs InstructBLIP 50.7 and DocOwl 52.6. DocVQA 65.1/62.6 vs Pix2Struct 76.6 and DocOwl 62.2. ChartQA 65.7/66.3, AI2D 62.3/57.7, OCR-VQA 75.7/70.5.
Go to primary source ↓e-groundingPDF pp. 7–8, Section 4.3 and Table 6
RefCOCO test-B Qwen-VL 85.34, Chat 84.51, Shikra-13B 81.81, UNINEXT-H 91.46. GRIT Qwen-VL 78.22 vs Unified-IO 78.61; Chat entry absent.
Go to primary source ↓e-fewshotPDF p. 8, Section 4.4 and Figure 4
Randomly sampled exemplars, no RICES; curves for Flickr30K, OKVQA, VizWiz and TextVQA show task-dependent changes and lack error bars.
Go to primary source ↓e-instructionsPDF p. 9, Section 4.5 and Table 7
TouchStone Chat English/Chinese 645.2/401.2; mPLUG-Owl English 605.4, VisualGLM Chinese 247.1. SEED-Bench uses four sampled video frames; distinguishes image/video scores and MME aggregates.
Go to primary source ↓e-benchmark-protocolsPDF p. 21, Appendix D, Table 9, relevant task/split/metric rows
Flickr30K karpathy-test/CIDEr; VQAv2 test-dev/VQA Score; TextVQA val/VQA Score; DocVQA test/ANLS; ChartQA test/Relaxed EM; AI2Diagram and OCRVQA test/EM; referring splits/Accuracy; TouchStone/GPT-4 Score.
Go to primary source ↓e-cleaningPDF p. 16, Appendix A.1–A.3; p. 18, Appendix A.4 continuation
Lists image/text/CLIP filtering without numeric thresholds, longest-caption selection, maximum-confidence VQAv2 answer selection, recursive grounding cleanup, PDF/HTML render-and-text extraction filters.
Go to primary source ↓e-ocrPDF pp. 16–17, Appendix A.4 and Figure 5
SynthDoG-style synthetic OCR uses COCO train2017/unlabeled2017 backgrounds, 41 English and 11 Chinese fonts and quadrilateral labels; document renderings add OCR supervision.
Go to primary source ↓e-convergencePDF p. 21, Appendix E.1 and Figure 6
Reports one epoch, declining training loss and fluctuating improvement on captioning and zero-shot VQAv2; no VQA data in stage 1.
Go to primary source ↓e-queriesPDF pp. 21–22, Appendix E.2 and Figure 7
ViT-L/14, 224 resolution and 64/144/256/400 queries; smoothed early and later loss curves. Gives patch counts 256 and 1024 and chooses 256 queries for final training.
Go to primary source ↓e-attentionPDF pp. 22–23, Appendix E.3, Table 10 and Figure 8
Window attention uses 224-sized windows in most layers with occasional global attention. At 448, global/window speed is 10/9 s per iteration; at 896, 60/25. Figure 8 compares training loss; authors select 448 global attention.
Go to primary source ↓e-text-retentionPDF pp. 23–24, Appendix E.4 and Table 11
Intermediate Qwen-7B initialization is distinct from final release. MMLU/C-Eval: intermediate 49.9/48.5, Qwen-VL 50.7/51.1, final Qwen-7B 58.2/63.5; intermediate CMMLU missing. Authors attribute retention to mixed pure-text data.
Go to primary source ↓e-futurePDF p. 10, Section 6
Future directions include more modalities, higher resolution/model/data scale and image/speech generation.
Go to primary source ↓8.5 Primary sources
Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond ↗
PDF · 8,462 extracted words
Source fingerprint
b4eca60ae9d3c1548242da02ece0e2f35ce23d3a983255830d98212aa92e3167