Mixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Models
1. Paper overview
In one sentence: MoT separates modality-specific transformer weights while retaining shared attention, improving multimodal learning efficiency at the cost of extra stored parameters and implementation-sensitive overhead. e-routinge-parameter-boundarye-chameleon-resulte-speeche-text-boundarye-systems
| At a glance | What to know |
|---|---|
| Research problem | Source description A dense multimodal transformer must accommodate different token statistics and objectives using shared weights. MoT asks whether deterministic modality specialization can improve learning efficiency while preserving cross-modal interaction and comparable activated computation. Its hypothesis concerns parameter allocation, rather than a new image tokenizer or action-learning objective. e-routinge-architecture |
| Core mechanism | Source description The architectural contribution extends modality untying beyond feed-forward networks to attention projections and normalization. Grouping and restoring tokens preserves their sequence relationships for shared attention. e-routing |
| A key reported result | Chameleon 7B text/image validation efficiency: MoT: 55.8% of the dense budget. Training FLOPs needed for comparable validation loss, lower is better.. Pretraining from scratch; Obelisc/Shutterstock held-out sets and COCO/Flickr30k Karpathy test splits. Dense final checkpoint: 100%; Figure 5 separately reports training-loss matching slope 0.455. The 55.8%-checkpoint supports the validation claim; the training slope is a different summary. These are predictive losses, not human image-quality scores. e-chameleon-setupe-chameleon-result |
| Reading caution | Reader analysis Expert Choice validation can leak future information and can also deteriorate under distribution shift. The MoE comparison therefore does not cleanly rank causal generation systems. e-moe-caveat |
Core contributions
- Source description
The architectural contribution extends modality untying beyond feed-forward networks to attention projections and normalization. Grouping and restoring tokens preserves their sequence relationships for shared attention. e-routing
- Source description
Experiments cover discrete text/image autoregression, added discrete speech, and autoregressive text with image diffusion. Component ablations and modality-sharing controls test where specialization helps; a preliminary text-MoE hybrid explores complementary sparsity. e-chameleon-resulte-speeche-transfusion-smalle-ablatione-looe-hybrid
Figure 3. Separate weights process each modality; shared attention keeps the sequence connected. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read panel (a) from bottom to top. Yellow T, pink S and blue I tokens first enter modality indexing, then their respective embeddings and attention projections. The sequence-reordering buffer restores the interleaved stream before the green joint-attention block. Tokens are grouped again for modality-specific processing and then returned to the output queue. Algorithm 1 makes the post-attention output projection explicit, although the drawing compresses that operation. Panels (b) and (c) change the image prediction target: one discrete next token versus the next denoising state of continuous image tokens. The faded branches illustrate selective activation; they do not establish permanently frozen parameters. e-routinge-architecturee-transfusion-objectivee-inference
What it supports. MoT specializes token processing without removing cross-modal communication. A token uses the weights associated with its known modality, while attention can still incorporate permitted context from other modalities. The same architectural rule supports both autoregressive image tokens and image diffusion, so sparsity and the prediction objective are separate design choices.
Where the evidence stops. Panel (c) says 1,000 diffusion steps; Section 3.4.1 specifies 250 at inference. The diagram does not show the attention mask: Appendix A.2 requires causal ordering across the sequence and bidirectional attention within an image. These details should be implemented from the text, preserving the original drawing.
2. Motivation
2.1 The problem and the proposed response
A dense multimodal transformer must accommodate different token statistics and objectives using shared weights. MoT asks whether deterministic modality specialization can improve learning efficiency while preserving cross-modal interaction and comparable activated computation. Its hypothesis concerns parameter allocation, rather than a new image tokenizer or action-learning objective. e-routinge-architecture
2.2 What this reading follows
Imagine processing a caption, its image and a speech segment in one sequence. A dense transformer applies the same weights to every position. MoT instead uses the token’s known modality to select its projections, feed-forward network and normalization, then lets the sequence communicate through global attention. The experiments ask whether this specialization reaches useful predictive performance with less training computation. The answer is strongest for image and speech modeling, with smaller benefits for text under Transfusion. This reading follows the original architecture, a configuration table, quantitative comparisons and a component ablation, while separating activated FLOPs, held-out quality and measured training time. e-routinge-parameter-boundarye-chameleon-resulte-speeche-text-boundarye-systems
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Foundational work |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
This table preserves the labels recorded at reading time. The current major category is Components of WAMs. View the current classification.
3.1 Evidence-based assessment
Supports the recorded classification
The recorded foundational-backbone and training-optimization categories fit. MoT studies multimodal generation, with neither action extraction nor environment-conditioned control. Consequently the world-action architecture, prediction-paradigm and quadrant fields remain Not applicable; modality-specific branches do not establish a One Model world/action classification. Distillation is not a demonstrated contribution within the combined catalog subcategory. e-routinge-inferencee-transfusion-objective
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Follow one token without severing its context
Consider an image token following a text caption. Algorithm 1 first groups it with other image tokens so the image-specific projections can be computed efficiently. Grouping does not authorize changing the sequence: the projected tokens return to their original order before attention. The image query can therefore receive information from allowed caption positions, even though the caption and image used different projection matrices. After attention, the image position again selects image-specific output and feed-forward parameters. Reader interpretation: the architecture changes how a modality processes information while preserving an interface for cross-modal exchange. The leave-one-out experiment supports this motivation conditionally: merging image and text weights removes much of their benefit, whereas merging image and speech hurts speech more strongly. Those asymmetric effects caution against treating all modality pairs as interchangeable. e-routinge-loo
Table 4. The Transfusion training budget is specified separately from model width and depth. Original paper, p. 21 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each row as one model-scale configuration. The left columns determine the transformer geometry; the middle columns describe sequence length and distributed batch formation; the right columns report training exposure. The 760M row uses hidden dimension 1536, 24 layers and 24 attention heads, with sequence length 4096, batch size four per GPU and 128 GPUs. All rows list 250k steps and 0.524 trillion tokens. Section 3.4.1 additionally holds the U-Net parameter count at 0.27B across scales. That fixed count describes an architectural choice, not a frozen-module training stage, and must be considered when interpreting the abbreviated model-size labels. e-transfusion-tablee-transfusion-setupe-transfusion-small
What it supports. This table makes the 760M-versus-1.4B comparison more informative than the size labels alone: training exposure is held approximately constant while the activated transformer configuration changes. It also gives a concrete starting point for reconstruction, but contains no benchmark scores and cannot itself establish a quality improvement.
Where the evidence stops. The 7B Tokens/Batch cell literally reads 2.10N; no correction is supplied. The prose rounds 0.524T to 0.5T and calls four listed scales five. Preserve these discrepancies and confirm configuration details before exact replication.
5.2 Separate the image objective from the routing rule
The Transfusion experiment changes what an image position represents. Instead of predicting the next discrete image token, the model receives noisy continuous patches and learns to predict their added noise. Appendix A compresses local VAE patch windows through a U-Net into 256 transformer tokens per image. The combined objective adds language-modeling loss to diffusion loss weighted by lambda, set to five. MoT supplies modality-specific transformer weights within this setup; it does not invent the diffusion objective. The attention mask is crucial: patches can communicate bidirectionally within their own image, while the surrounding sequence retains causal order. At generation time, BOI starts iterative denoising and EOI returns to text sampling. Reader interpretation: same-model media generation requires compatible representations, masks and decoding transitions, not merely summing two losses. e-transfusion-objectivee-inferencee-routing
5.3 Ask which budget and which outcome improved
Begin with Figure 5’s stopped checkpoint: it is evidence about validation loss at a reduced training budget. Then inspect Figure 11, where a smaller activated MoT model improves generation and captioning metrics, but retain that figure’s numbers because Figure 12 disagrees on FID. Finally use Figure 19 to ask whether the computational saving survives communication and indexing overhead. Its training and validation slopes differ, and its hardware setting matters. Reader interpretation: these comparisons form complementary tests rather than a universal speedup multiplier. The Transfusion text-only results provide a counterexample to uniform benefit, while the preliminary text-MoE hybrid suggests a way to improve that branch. A faithful reproduction should decide its target metric and budget first, then evaluate all alternatives with that same definition. e-chameleon-resulte-transfusion-smalle-numeric-discrepanciese-wall-clocke-systemse-text-boundarye-hybrid
5.4 Training and inference
During training
Models are pretrained from scratch, with dense and sparse baselines controlled by activated FLOPs. Chameleon uses roughly equal text/image tokens and a pretrained image tokenizer. Speech uses a 500-entry, 25-Hz in-house tokenizer and a speech-to-Chameleon sampling ratio of 1:6. e-chameleon-setupe-speech
Transfusion samples text and image tokens/patches equally from Llama 2 data and licensed Shutterstock images. Images are resized to 256×256; captions precede images 80% of the time. Table 4 reports 250k steps and 0.524T tokens, rounded to 0.5T in prose. The extra U-Net size is fixed across scales, which does not mean frozen weights. e-transfusion-setupe-transfusion-table
A later fine-tuning stage uses internal aesthetic data and image-editing examples. Its illustrated outcomes are qualitative and do not establish a population-level success rate. e-finetuning
During inference
Discrete generation repeatedly samples the next token. In Transfusion, BOI switches generation to denoising Gaussian image patches; after decoding, EOI returns the model to language generation. The setup specifies 250 diffusion steps, despite Figure 3’s 1,000-step label. This is media generation without environment feedback or action execution. e-architecturee-inference
5.5 Implementation flow
- Route by known modality
For each token, its modality selects Q/K/V projections. No learned router chooses among modalities. Restore the original ordering before attention so grouping is an implementation operation, not a change to the modeled sequence. e-routing
- Communicate, then specialize again
Global attention mixes the allowed sequence positions. Each output subsequently uses its own modality’s output projection, normalization and FFN, with residual connections. Separate weights do not make independent models: attention still couples the branches. e-routing
- Choose representation and objective
Chameleon predicts discrete tokens autoregressively. Transfusion represents each image with 256 continuous tokens after VAE/U-Net processing; it uses causal inter-sequence attention and bidirectional attention within an image. These image tokens predict diffusion noise, not robot actions. e-architecturee-transfusion-objective
6. Experiments & results
Mixture-of-Transformers (MoT) assigns modality-specific weights throughout transformer layers while exchanging information through global attention. It accelerates multimodal pretraining, especially image and speech modeling; benefits depend on the objective and evaluation protocol. This is a generative backbone study, with no action policy or executed-control evaluation.
The paper’s five tables describe datasets or training configurations; it has no tabular benchmark-results comparison. This edition therefore includes original Table 4 for quantitative configuration details and original result plots for measured outcomes. Fine-tuned generation/editing is illustrated qualitatively without an aggregate success table. e-chameleon-setupe-speeche-transfusion-tablee-ablatione-finetuning
6.1 Read the original evidence
Figure 5. A checkpoint-level comparison supports the 55.8% validation-compute claim. Original paper, p. 11 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the bottom two rows, where the dark-red bars represent MoT stopped at 55.8% of the dense training budget. Compare each dark-red bar with its pale-blue dense counterpart within the same dataset and modality. These are the observations behind the headline validation efficiency claim. The upper curves explain why such a checkpoint is plausible: MoT reaches lower losses earlier. The upper-right matching plot compares required steps for corresponding loss levels; its displayed 0.455 slope summarizes global training loss, while the smaller image and text plots describe separate losses. Axes and vertical ranges vary across panels, so bar heights cannot be compared across datasets. e-chameleon-resulte-chameleon-setupe-moe-caveat
What it supports. The early MoT checkpoint achieves comparable or lower validation losses on the displayed image and text evaluations. With the controlled per-step FLOPs used here, 55.8% of the training budget corresponds to 44.2% fewer FLOPs. This result supports faster attainment of the tested predictive-loss levels at 7B activated scale.
Where the evidence stops. The 0.455 training-loss slope and the 55.8% validation checkpoint are different measurements. The paper does not show uncertainty across repeated training seeds here. Expert Choice routing also complicates interpretation of the MoE baseline’s validation loss.
Figure 11. Figure 11 reports stronger image generation and captioning from a smaller activated MoT backbone. Original paper, p. 20 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow the three bottom panels separately. Higher CLIP scores indicate better prompt alignment, lower FID is the reported generation-quality criterion, and higher CIDEr evaluates captions. The dark-red 760M MoT bars beat the blue 1.4B dense bars in each panel. The lighter-red 163M model provides a useful boundary: reducing activated size further does not preserve all evaluation quality. The top row concerns diffusion training loss and required steps, so it should not be used as a substitute for the bottom evaluations. Generation uses 30,000 COCO validation prompts, while captioning uses the Karpathy test split; the default numerical protocol omits classifier-free guidance. e-transfusion-smalle-transfusion-evale-transfusion-setupe-numeric-discrepanciese-transfusion-7be-text-boundary
What it supports. In this figure, 760M MoT obtains CLIP 0.214 versus 0.206, FID 21.145 versus 24.688, and CIDEr 0.320 versus 0.286. The authors associate this comparison with half the training/inference FLOPs. The improvement spans image generation and captioning, but does not establish an equivalent gain in text-only language modeling.
Where the evidence stops. Figures 11 and 12 report conflicting FID values for corresponding model scales without a clear reconciliation. The values here belong specifically to Figure 11. Figure 10’s caption repeats this smaller-model comparison despite its 7B plots; neither discrepancy is silently repaired.
Figure 19. The FLOP advantage becomes a training-time advantage in the reported A100 environment. Original paper, p. 34 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the left plots as loss against elapsed training cost and the adjacent plots as time needed to match the dense model’s loss. The diagonal in each matching plot marks equal time; slopes below one favor the alternative. Image and text training occupy the upper two rows. The small bottom row repeats the comparison for held-out Obelisc data, which yields its own ratios. Section 6 specifies 256 A100 GPUs on AWS p4de.24xlarge instances, full-shard FSDP and PyTorch 2 Compiler where applicable. The horizontal axes are normalized, so the figure supports relative training-time comparisons rather than an absolute number of hours. e-wall-clocke-systemse-scaling-control
What it supports. The image training-loss slope is 0.472 and the text slope is 0.756 for MoT. Held-out image/text slopes are instead 0.584 and 0.749. MoE-4x’s corresponding training slopes exceed one. This demonstrates practical value for the authors’ implementation and cluster, while retaining the distinction between training loss and validation.
Where the evidence stops. These are training measurements, not serving throughput or perceptual-quality tests. Section 6 warns that relative performance can vary by cluster. Its separate GPU-scaling experiment also changes global batch and total tokens, so that experiment cannot isolate hardware count alone.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Chameleon 7B text/image validation efficiency Pretraining from scratch; Obelisc/Shutterstock held-out sets and COCO/Flickr30k Karpathy test splits. | MoT: 55.8% of the dense budget. Training FLOPs needed for comparable validation loss, lower is better. | Dense final checkpoint: 100%; Figure 5 separately reports training-loss matching slope 0.455. The 55.8%-checkpoint supports the validation claim; the training slope is a different summary. These are predictive losses, not human image-quality scores. e-chameleon-setupe-chameleon-result |
| Chameleon+Speech 7B speech validation efficiency Three-modality training; held-out LL60K and PPL30K. | MoT: 0.313 on LL60K; 0.372 on PPL30K. Validation-loss step-matching slope, with controlled per-step FLOPs. | Dense identity slope: 1; MoE-4x: 0.979 and 0.840 respectively. The headline 37.2% concerns speech token prediction, not intelligibility or acoustic quality. e-speech |
| Transfusion 760M image generation and captioning Figure 11; COCO 30k validation prompts for generation, Karpathy test split for captions; default without classifier-free guidance. | MoT: CLIP 0.214; FID 21.145; CIDEr 0.320. CLIP and CIDEr higher; FID lower. | 1.4B dense: 0.206; 24.688; 0.286. Authors report half the training/inference FLOPs for 760M MoT. A smaller activated backbone wins this comparison. Figure 12 supplies conflicting FID values, so these figures are not pooled. e-transfusion-smalle-transfusion-evale-numeric-discrepancies |
| Transfusion 7B image-loss convergence Figure 10; image training loss and held-out CC12M diffusion loss. | MoT: training 0.309; validation 0.374. Step-matching slope at controlled per-step FLOPs. | MoE-4x: 0.784 and 1.042; dense identity slope 1. The plotted validation ratio exceeds one third. Figure 10’s caption describes a different scale comparison; the panel labels and Section 3.4.2 establish the 7B setting. e-transfusion-7be-transfusion-eval |
| Attention untying beyond FFN specialization 880M Chameleon ablation; held-out Obelisc and Shutterstock. | About 33.3% for images and 10% for text, as reported in prose. Estimated additional FLOP saving for matched Obelisc performance. | FFN+Q/K/V untying versus FFN-only untying; further norm untying contributes little. This tests incremental attention specialization; it does not isolate the standalone effect of normalization. e-ablation |
| Chameleon 7B wall-clock convergence 256 A100 GPUs on AWS p4de.24xlarge; full-shard FSDP, PyTorch 2 Compiler where applicable. | MoT: 0.472 image training loss; 0.756 text training loss. Normalized training-time matching slope, lower is better. | MoE-4x: 1.707 and 1.070. MoT Obelisc validation slopes are separately 0.584 and 0.749. Efficiency transfers to this system configuration. Training-loss ratios should not be presented as measured perceptual-quality or serving-latency gains. e-systemse-wall-clock |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 14. Attention specialization improves on FFN-only untying; norm untying adds little in this combination. Original paper, p. 25 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use the legend in the rightmost panel for all four plots. Blue is the dense baseline; orange, labeled MoMa, unties only the feed-forward module. Green uses MoT attention and feed-forward specialization while retaining shared normalization. Red is full MoT. Each vertical axis is perplexity, with lower values better, and horizontal axes show training steps. First compare orange with blue to locate the contribution of FFN untying, then compare green with orange to locate the additional attention contribution. Finally compare red with green. Their near overlap matters because the full architecture includes normalization specialization, but this particular evaluation does not show a substantial incremental benefit from it. e-ablation
What it supports. At the 880M Chameleon setting, both feed-forward and attention specialization improve held-out perplexity. Section 3.5 estimates that adding Q/K/V untying to FFN untying saves about 33.3% of image FLOPs and 10% of text FLOPs on Obelisc at matched performance. The curves support a stronger role for attention specialization than for the final normalization change.
Where the evidence stops. These are nested ablations, not a full factorial test of every component. Footnote 7 explicitly leaves standalone normalization untying unresolved. The plots also lack repeated-seed uncertainty, so small separations should not be treated as established effects.
7. Analysis & limitations
7.1 What the evidence leaves open
Expert Choice validation can leak future information and can also deteriorate under distribution shift. The MoE comparison therefore does not cleanly rank causal generation systems. e-moe-caveat
Transfusion gains are concentrated in images and captioning; text-only improvement is small. Hard prompts still defeat both fine-tuned models, and the paper provides selected examples rather than systematic editing success rates. e-text-boundarye-finetuning
Source discrepancies remain unresolved: Figure 10’s caption/plots, Figures 11–12 FID values, the 1,000/250 diffusion-step labels, and ambiguous total-versus-active parameter wording. This report preserves figure-specific values rather than inventing a harmonized result. e-transfusion-7be-numeric-discrepanciese-architecturee-parameter-boundary
Hardware scaling also increases batch size and token exposure. It cannot isolate GPU count as the cause of improved learning efficiency. MoT still has extra stored parameters and routing overheads despite comparable activated FLOPs. e-scaling-controle-systems
7.2 Questions for discussion
- How much benefit survives matching total stored parameters as well as activated FLOPs?
- Would causal MoE routing change the relative ranking on generation and held-out loss?
- Does attention specialization remain useful under strongly skewed modality mixtures?
8. Reproducibility audit
8.1 Requirements and known gaps
Reconstruct modality grouping, sequence restoration and attention masks first. Transfusion’s documented optimizer uses AdamW, learning rate 3e-4, 4k warmup and cosine decay to 1.5e-5, with weight decay 0.1 and gradient clipping 1.0. Fix the preprocessing, token mix, diffusion steps and guidance before comparing losses or FID. e-routinge-transfusion-setupe-transfusion-objectivee-transfusion-eval
Exact replication needs the reused Chameleon corpus, in-house speech tokenizer, licensed image data and internal fine-tuning data. The paper does not supply a complete standalone recipe for these artifacts or exact software patch versions. Table 4’s 2.10N cell also needs clarification. e-chameleon-setupe-speeche-transfusion-setupe-transfusion-tablee-finetuninge-systems
Proposed checks: repeat the FFN/attention/norm ablation with fixed token order and repeated seeds; then compare time-to-held-out-loss under a fixed global token budget and identical hardware/software. Report total and activated parameters separately. These test mechanisms and practical efficiency without assuming exact replication of private data. e-ablatione-scaling-controle-systems
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Does Q/K/V untying add a repeatable benefit beyond FFN untying?
Reader-proposed check: reproduce the four Figure 14 variants using the 880M configuration and a fixed text/image training sequence, tokenizer, optimizer and FLOP budget. Run at least three seeds and evaluate separate image/text perplexities on identical held-out data at prespecified checkpoints. Add a normalization-only variant to test the component not isolated by the paper. Report active and stored parameters separately. The attention hypothesis is weakened if FFN+Q/K/V no longer reaches the FFN-only target loss earlier beyond seed variability. If the original corpus is unavailable, label an explicitly documented public-data substitute as a mechanism test, not exact replication. e-ablatione-chameleon-setupe-parameter-boundary
Check 2: Does faster convergence survive a fixed token budget and identical systems controls?
Reader-proposed check: compare dense and MoT on the same A100 hardware, data order, global batch, total tokens and held-out loss target. Repeat at two GPU counts, using gradient accumulation to preserve global batch rather than increasing training exposure. Match FSDP and compilation settings, then record time per step, peak memory, synchronization overhead and total time to the target. MoT’s practical advantage is falsified for that setting if reduced steps fail to reduce end-to-end time. Report both image and text outcomes; the paper’s differing ratios make either modality alone an incomplete test. e-systemse-wall-clocke-scaling-control
8.3 Reading coverage
Visual audit: Visually inspected the title/author/version page, routing equations and Algorithm 1, Figures 3, 5–8, 10–12, 14–15, 17–22 and 26, Tables 1–5, and the method, evaluation, systems and appendix pages supporting retained claims. Inspected each of the six final original crops; the corrected Figure 5 crop was inspected again. Cross-checked Figure 3’s processing order against Algorithm 1 and Appendix A.2’s mask and inference rules. Preserved the 1,000/250 diffusion-step discrepancy, Figure 10 caption/scale mismatch, Figures 11–12 numerical differences and Table 4’s 2.10N cell. Figure 15’s duplicated Image label and Appendix B’s reversed qualitative figure references remain source discrepancies. All 48 pages were covered by the 13 text chunks; figures on other pages were not visually reviewed. No code, external supplements or reproduction runs were inspected.
PDF pages inspected for this edition: 1, 6, 8, 9, 11, 12, 13, 14, 15, 16, 17, 19, 20, 21, 22, 23, 25, 26, 27, 28, 30, 31, 32, 33, 34, 42, 43, 44, 48. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Title, abstract and contents (pp. 1–3)
- Introduction and method, including Algorithm 1 (pp. 4–9)
- Experiments: Chameleon, speech, Transfusion, fine-tuning and component ablation (pp. 10–25)
- Leave-one-out analysis and hybrid MoT/MoE (pp. 26–30)
- Systems, deployment, related work and conclusion (pp. 31–36)
- References (pp. 36–41)
- Appendices A–C: diffusion, inference, fine-tuning examples and supplementary plots (pp. 42–48)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- Recovered-edition limitation: arXiv v2 author-deposited publication-formatted copy. Page 1 says Published in Transactions on Machine Learning Research (04/2025) and explicitly links the catalog OpenReview forum Nu6N69i8SB. Byte-for-byte equivalence to the blocked OpenReview PDF was not checked.
- All 13 supplied text chunks were read individually. Original PDF images were inspected on the pages listed in the illustrated edition; visual coverage is selective, not every figure in the paper.
- Code, linked resources and separate supplements were not inspected; no experiments were reproduced.
- Identity notes: reviewed arXiv 2411.04996v2 dated 8 May 2025, carrying the TMLR 04/2025 header and catalog OpenReview link. This is the supplied author-deposited edition; equivalence to the inaccessible venue PDF remains unchecked. Title line-wrap hyphenation is preserved in source metadata. The title-page author Srinivasan Iyer corresponds to Srini Iyer in the catalog BibTeX.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e-identityPDF p. 1, title, author block, publication header, arXiv margin and OpenReview link
The title matches the catalog after joining the line-broken word Architecture. The eleven listed authors match the catalog author string; the title page uses Srinivasan Iyer, whereas the supplied catalog BibTeX abbreviates this as Srini Iyer. Affiliations are Stanford University and AI at Meta. The page identifies arXiv:2411.04996v2, 8 May 2025, TMLR (04/2025), and OpenReview forum Nu6N69i8SB.
Go to primary source ↓e-routingPDF p. 6, Section 2.2, Eqs. (1)–(2); p. 9, Algorithm 1 lines 3–16 and Eq. (3)
MoT groups tokens by modality, applies modality-specific Q/K/V projections, restores sequence order for global attention, and then applies modality-specific output projections, normalization and feed-forward networks with residual connections.
Go to primary source ↓e-architecturePDF p. 8, Figure 3(a–c) and caption; p. 20, Section 3.4.1, last sentence
Figure 3 depicts text, speech and image parameter branches with shared attention, and discrete autoregression versus continuous image diffusion. Panel (c) labels 1,000 diffusion timesteps, while the experimental setup states 250 inference steps. This difference is not explained.
Go to primary source ↓e-chameleon-setupPDF pp. 11–12, Section 3.2.1, Data and Pre-processing; p. 13, Table 1 and footnote 4
Chameleon reuses the earlier work’s data and pretrained tokenizer, with roughly equal text/image token counts. Evaluation includes Obelisc, Shutterstock, and Karpathy test splits of COCO and Flickr30k. The 7B row specifies hidden size 4096, 32 layers, 32 heads, sequence length 4096, 384 GPUs, 120k steps and 0.377T tokens. These are relative architecture comparisons, not a scaling-law study.
Go to primary source ↓e-chameleon-resultPDF p. 11, Figure 5(a–n), especially (g–n), and caption; p. 14, Section 3.2.2 continuation
Figure 5 compares dense, MoT and MoE-4x under controlled FLOPs. Its 55.8%-checkpoint has comparable or lower image/text validation losses than the dense final model on four datasets. Global training-loss matching is separately summarized by slope 0.455. Sparse size labels denote activated parameters.
Go to primary source ↓e-moe-caveatPDF p. 13, Section 3.2.1, Mixture-of-Experts Implementation; p. 21, Section 3.4.2
Expert Choice routing is used for MoE training and validation perplexity, maintaining isoFLOP evaluation but permitting future-token leakage and sensitivity to evaluation distribution shifts. The authors exclude MoE caption scores because of possible leakage.
Go to primary source ↓e-speechPDF p. 15, Section 3.3.1 and Tables 2–3; p. 17, Figure 8(c–f) and caption
Speech training uses SpiRit-LM data and an in-house DinoSR variant with 500 semantic tokens at 25 Hz. Speech data is sampled against Chameleon data at 1:6. Figure 8 reports speech validation matching slopes 0.313 on LL60K and 0.372 on PPL30K; corresponding MoE-4x slopes are 0.979 and 0.840. The 0.229 in panel (b) concerns MoT training loss.
Go to primary source ↓e-transfusion-setupPDF p. 20, Section 3.4.1, Data and Pre-processing and Model Hyperparameters
Transfusion uses Llama 2 text data/tokenization and 380 million licensed Shutterstock image-caption pairs; images are center-cropped/resized to 256 by 256. Text and image tokens/patches are sampled 1:1, with captions preceding images 80% of the time. AdamW uses betas 0.9/0.95, epsilon 1e-8, learning rate 3e-4, 4k warmup steps, cosine decay to 1.5e-5, weight decay 0.1 and gradient clipping 1.0. The extra U-Net parameter count is held at 0.27B across configurations; this is not a statement that its weights are frozen.
Go to primary source ↓e-transfusion-tablePDF p. 21, Table 4, all rows; p. 20, Section 3.4.1, Model Hyperparameters
Table 4 lists 163M, 760M, 1.4B and 7B configurations, all with sequence length 4096, 250k steps and 0.524T tokens. The 760M row has hidden size 1536, 24 layers, 24 heads, batch size/GPU 4 and 128 GPUs. The 7B Tokens/Batch cell literally reads 2.10N. Prose rounds the token budget to 0.5T and calls the four listed scales five.
Go to primary source ↓e-transfusion-evalPDF p. 21, Section 3.4.1, Evaluation Benchmarks and footnote 6
Text evaluation uses 20 million held-out Wikipedia/C4 tokens. Image diffusion loss uses held-out CC12M. Generation FID and CLIP use 30,000 randomly selected COCO validation prompts; caption CIDEr uses the COCO Karpathy test split. Unless noted, quantitative generation omits classifier-free guidance; qualitative 7B examples use guidance 5.
Go to primary source ↓e-transfusion-smallPDF p. 20, Figure 11(c–e) and caption; p. 22, Section 3.4.2 continuation
Figure 11 reports 760M MoT versus 1.4B dense: CLIP 0.214 versus 0.206, FID 21.145 versus 24.688, CIDEr 0.320 versus 0.286, with the authors stating half the training/inference FLOPs. The 163M MoT bars show CLIP 0.195, FID 26.920 and CIDEr 0.232.
Go to primary source ↓e-transfusion-7bPDF p. 19, Figure 10(b–h); p. 21, Section 3.4.2
The plotted 7B comparison reports MoT matching slopes 0.309 for image training loss and 0.374 for image validation loss; MoE-4x slopes are 0.784 and 1.042. MoT/dense metrics are CLIP 0.227/0.222, FID 18.862/19.502 and CIDEr 0.406/0.315. The Figure 10 caption instead repeats the 760M-versus-1.4B comparison. Section 3.4.2 writes the CIDEr values as 40.6 versus 31.5; this report retains the plotted scale.
Go to primary source ↓e-numeric-discrepanciesPDF p. 20, Figure 11(d); p. 22, Section 3.4.3; p. 23, Figure 12(6,9,14,15,24)
Figures 11 and 12 give different FID values for apparently corresponding scales: 760M MoT 21.145 versus 15.749, and 1.4B dense 24.688 versus 19.318. The 163M MoT FID is 26.920 in Figure 11 versus 21.586 in Figure 12. Section 3.4.3 also gives dense 760M CLIP 0.202 versus panel (14)’s 0.196, and dense 163M CIDEr 0.147 versus panel (9)’s 0.142. The supplied text does not reconcile these differences.
Go to primary source ↓e-ablationPDF p. 25, Section 3.5, Figure 14, Table 5 and footnote 7
The 880M Chameleon ablation compares dense, FFN-only untying, FFN plus Q/K/V untying with shared norms, and full MoT. Held-out Obelisc and Shutterstock perplexities improve with FFN untying and further attention untying. Prose estimates additional Obelisc FLOP savings of about 33.3% for images and 10% for text relative to FFN-only untying. Norm untying adds little in this combination; its standalone effectiveness is untested.
Go to primary source ↓e-looPDF p. 26, Figure 15(a–n) and caption; p. 27, Sections 4.1–4.2
At 443M activated parameters, two-tower variants isolate one modality while sharing the other two. Sharing image/text removes most of their gains; isolating speech retains speech benefits. Sharing image/speech hurts speech more than images. Panel (a) mistakenly labels two blocks Image; the caption and Section 4.1 identify image, text and speech. The claims of universally lowest isolated-modality losses are stronger than the plotted bars, which can favor full MoT.
Go to primary source ↓e-hybridPDF p. 28, Section 5.2; p. 30, Figure 17(a–n)
The preliminary Transfusion hybrid replaces the text FFN with MoE-4x while retaining the MoT image branch. It improves text validation loss relative to plain MoT, while retaining broadly similar image benefits. Its scalability and generalizability remain future work.
Go to primary source ↓e-systemsPDF p. 31, Sections 6.1–6.2; p. 32, Sections 6.2.2–6.3
MoT adds stored parameters through modality untying and incurs indexing, synchronization and uneven modality-batch overheads. Experiments use AWS p4de.24xlarge A100 instances, full-shard FSDP and PyTorch 2 Compiler where applicable. Wall-clock comparison uses 256 GPUs. Deployment suggestions include batching and padding, rather than measured serving latency.
Go to primary source ↓e-wall-clockPDF p. 34, Figure 19(a–h) and caption; p. 32, Section 6.2.2
Chameleon 7B matching-line slopes for MoT are 0.472 for image training loss and 0.756 for text training loss. Obelisc validation slopes are 0.584 and 0.749. MoE-4x training slopes are 1.707 and 1.070. These are normalized GPU training-time comparisons in the reported environment.
Go to primary source ↓e-scaling-controlPDF p. 32, Section 6.2.1; p. 33, Figure 18 and caption
The 443M hardware-scaling experiment changes GPU count from 16 to 256 while proportionally increasing global batch size and total training tokens at fixed steps. Thus it changes both hardware and training exposure.
Go to primary source ↓e-transfusion-objectivePDF p. 42, Appendix A.1–A.2 and Eq. (4); p. 43, Appendix A.2 continuation
The image denoiser predicts added Gaussian noise under squared-error DDPM loss. U-Net modules compress 2 by 2 VAE patch windows, yielding 256 continuous tokens per image. Attention is causal across the sequence but bidirectional within each image. The combined language-modeling plus diffusion loss uses balancing coefficient lambda=5.
Go to primary source ↓e-inferencePDF p. 43, Appendix A.2, inference paragraph; p. 20, Section 3.4.1, final sentence
Inference samples text autoregressively, switches at BOI to denoising appended Gaussian image patches, then appends EOI and resumes text generation; a VAE decoder produces the image. The experimental setup specifies 250 diffusion steps.
Go to primary source ↓e-finetuningPDF p. 22, Section 3.4.4; pp. 43–44, Appendix B, Figures 20–22
Fine-tuning uses an internal visually appealing dataset and 8k image-editing examples. Qualitative comparisons include hard prompts that both models fail to follow fully. Appendix B prose reverses the apparent easy/better figure references relative to Figures 20–21 captions; no aggregate success rate is provided.
Go to primary source ↓e-text-boundaryPDF p. 48, Appendix C, Figure 26(1–15); pp. 21–22, Section 3.4.2
Transfusion MoT shows little text-only training/validation improvement compared with dense models across the displayed scales, despite better caption CIDEr. MoE training-loss gains do not consistently transfer to held-out text loss.
Go to primary source ↓e-parameter-boundaryPDF p. 16, Section 3.3.3, final paragraph; p. 9, Algorithm 1; p. 31, Section 6.1, Communication Volume
Page 16’s wording about nearly identical non-embedding parameter counts is ambiguous against Algorithm 1’s separate modality weights and Section 6.1’s explicitly added MoT parameters. Sparse figure sizes should be read as activated parameters, without claiming identical total stored parameter counts.
Go to primary source ↓8.5 Primary sources
Mixture-of-Transformers: A Sparse and Scalable Architec- ture for Multi-Modal Foundation Models ↗
PDF · 22,958 extracted words
Source fingerprint
79eb51c94c39c824257c210b3d1975c4997e011b9cb3f1ee11aedd498c2f7af6