RESEARCH PAPERYear 2025

Mixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Models

Weixin Liang; Lili Yu; Liang Luo; Srinivasan Iyer; Ning Dong; Chunting Zhou; Gargi Ghosh; Mike Lewis; Wen-tau Yih; Luke Zettlemoyer; Xi Victoria Lin

Read paper PDF

Classification

View four quadrants

Major category: Components of WAMs
Quadrant: Not applicable
Architecture: Not applicable
Prediction paradigm: Not applicable
Subcategories: Language & vision-language backbones Language & VLM backbones
Source review status: Verified from primary sources

Category review. Mixture-of-Transformers defines modality-specific projections/normalization/FFNs coupled through shared attention. This is a reusable multimodal backbone architecture, explicitly referenced by the manuscript for WAM information flow; it is not itself a robot control method. Reading evidence

AT A GLANCE

Contribution

A contribution summary has not been added yet.

Abstract

An abstract has not been added yet.

Affiliations

Not listed in the collection.

BibTeX

@article{mot,
  author       = {Weixin Liang and
                  Lili Yu and
                  Liang Luo and
                  Srini Iyer and
                  Ning Dong and
                  Chunting Zhou and
                  Gargi Ghosh and
                  Mike Lewis and
                  Wen{-}tau Yih and
                  Luke Zettlemoyer and
                  Xi Victoria Lin},
  title        = {Mixture-of-Transformers: {A} Sparse and Scalable Architecture for
                  Multi-Modal Foundation Models},
  journal      = {TMLR},
  volume       = {2025},
  year         = {2025},
  
}

Classification

Contribution

Abstract

Affiliations

BibTeX

More in this category

V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning

DINOv3

World Simulation with Video Foundation Models for Physical AI