Scalable Diffusion Models with Transformers
1. Paper overview
In one sentence: DiT makes a transformer the latent-diffusion denoiser, improving ImageNet generation through adaptive conditioning and increased model computation while retaining a pretrained convolutional VAE. e02e04e06e10e11e12e14
| At a glance | What to know |
|---|---|
| Research problem | Source description Can a standard transformer replace diffusion's entrenched U-Net backbone while retaining scalable image quality? The paper studies this question inside a latent-diffusion pipeline and evaluates architecture complexity through forward-pass Gflops, because parameter count alone misses the computation introduced by spatial resolution and token count. e02e05 |
| Core mechanism | |
| A key reported result | Class-conditional ImageNet generation at 256×256: 2.27 FID; 4.60 sFID; 278.24 IS; precision 0.83; recall 0.57. FID-50K ↓; sFID ↓; Inception Score ↑; precision/recall ↑. DiT-XL/2, 7M training steps, classifier-free guidance scale 1.50; FID-50K with 250 DDPM sampling steps and ADM evaluation suite. The supplied text does not explicitly identify the ImageNet reference split. LDM-4-G at guidance 1.50: FID 3.60, precision 0.87, recall 0.48. StyleGAN-XL: FID 2.30, sFID 4.02. Unguided DiT: FID 9.62, recall 0.67. DiT has the lowest FID in this table, but does not win every metric. Guidance improves FID while reducing recall. Cross-paper entries do not establish equal training compute. e09e12 |
| Reading caution | Reader analysis The evidence concerns class-conditioned still images. Text-conditioned generation is proposed as future work; temporal prediction, action generation and robot execution are untested. The pipeline also retains a convolutional VAE, so replacing the denoising backbone does not make the entire generator convolution-free. e04e15 |
Core contributions
Figure 3. DiT predicts latent noise and covariance; conditioning modulates each residual block. Original paper, p. 3 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the left pipeline from bottom to top. The noised latent is patchified, while timestep t and label y enter an embedding branch; the prose calls the class label c. Repeated DiT blocks feed normalization, linear decoding and spatial rearrangement to produce Noise and Σ. In the highlighted block, input tokens pass through normalization, conditioning-dependent scale/shift, attention, and a gate before the residual addition. The feedforward sublayer repeats this pattern. The conditioning MLP supplies γ and β for normalization modulation and α for residual scaling. The two muted alternatives insert cross-attention or concatenate conditioning tokens. The arrows and gate placement agree with Section 3.2. e03e04e05e06e07
What it supports. The useful change is inside the denoiser: a patch sequence can replace the U-Net's spatial hierarchy. adaLN-Zero starts each residual block as an identity function by zeroing its residual scales, then learns how timestep and class information should modulate token processing. The separate outputs are diffusion statistics, rather than two independently generated images.
Where the evidence stops. This diagram shows one denoising pass, not the full sampling loop. The VAE decoder is outside the pictured transformer. Its convolutional architecture remains part of the image generator, and no action-execution interface is defined.
2. Motivation
2.1 The problem and the proposed response
Can a standard transformer replace diffusion's entrenched U-Net backbone while retaining scalable image quality? The paper studies this question inside a latent-diffusion pipeline and evaluates architecture complexity through forward-pass Gflops, because parameter count alone misses the computation introduced by spatial resolution and token count. e02e05
2.2 What this reading follows
The central experiment in DiT is architectural: keep latent diffusion, replace its U-Net denoiser with a transformer, and ask which design choices improve generated images. The answer involves both how conditioning enters each block and how much computation the denoiser performs. Follow the architecture upward from noised latent patches to noise and covariance predictions, then use the conditioning and scaling plots to understand why the authors choose adaLN-Zero and small patches. The benchmark table shows the benefit of long training and classifier-free guidance. The final diagnostic separates a stronger model from simply taking more sampling steps. All findings here concern class-conditioned still-image generation. e02e04e06e10e11e12e14
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Foundational work |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
This table preserves the labels recorded at reading time. The current major category is Components of WAMs. View the current classification.
3.1 Evidence-based assessment
Classification assessment not applicable
The recorded foundational-work category fits this diffusion-backbone study. The WAM architecture, prediction-paradigm and quadrant axes are not applicable: the network estimates image-latent noise/covariance, without future/action joint prediction or inverse dynamics. A unified denoiser is insufficient evidence for a One Model world-action system. e03e04e06e07e15
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Move diffusion into latent patches without changing its target
DiT begins from latent diffusion's separation of representation learning and generative modeling. The pretrained encoder compresses an image into a smaller spatial tensor and is frozen while the diffusion model is trained. Corruption supplies a noised latent and a timestep; the transformer is asked to predict the added noise and the reverse-process covariance. Patchification changes the denoiser's interface into a sequence, but the final linear projection rearranges that sequence back into spatial predictions. This explains why the architecture can remain close to a Vision Transformer while serving a diffusion objective. The noise predictor uses squared-error loss, whereas covariance learning needs the variational objective. At inference, successive reverse-process samples yield a latent that the VAE decoder converts into an image. No observed action or executed control command enters this chain. e03e04e05e07e08
5.2 Understand why adaLN-Zero is more than a conditioning label
Timestep information tells the model what corruption regime it is processing, and class information tells it which image distribution to generate. The architectural question is how those signals affect each token update. In-context conditioning lets ordinary attention discover the interaction; cross-attention installs a separate interaction layer. adaLN instead regresses scale and shift from the summed conditioning embeddings and applies the same modulation function across tokens. adaLN-Zero additionally gates each attention and feedforward residual update, starting those gates at zero. A reader interpretation is that this gives optimization an initially simple identity path while learning conditioned departures from it. Figure 5 supports the combined design's effectiveness, but it does not prove that interpretation: introducing gates and changing their initialization together leaves their separate contributions unresolved. e06e10
5.3 Separate model computation, training computation and sampling computation
The paper uses three related but different budgets. Forward-pass Gflops describe a single denoiser evaluation; changing backbone size or token count changes that cost. Total training compute also depends on batch size, update count and backward work, which Figure 9 estimates rather than measures directly. Sampling compute depends on how many denoising evaluations generate an image, which Figure 10 varies after fixing the training checkpoint. Keeping these budgets separate explains the experimental logic: the scaling sweep asks which denoisers learn well, while the sampling sweep asks whether a weaker denoiser can catch up by iterating longer. Finally, Table 2 combines a large denoiser, much longer training and classifier-free guidance. Reader analysis: its excellent FID is a result of that full configuration, not evidence that a backbone swap alone guarantees the same gain. e05e11e12e14
5.4 Training and inference
During training
The noise head uses squared-error denoising loss; learned covariance uses the full variational objective. Final output weights are zero-initialized. Training uses AdamW, constant learning rate 10⁻⁴, zero weight decay, batch size 256, horizontal flips and EMA decay 0.9999. No learning-rate warmup is used; the same recipe is retained across model and patch sizes. e03e08
Diffusion uses 1000 forward timesteps with a linear variance schedule from 10⁻⁴ to 0.02 and ADM's covariance parameterization. The best 256×256 model is trained for 7M steps; the separate 512×512 model for 3M. Reported evaluations use EMA weights. e08e12e13
During inference
Start with Gaussian latent noise, repeatedly predict reverse-process statistics, sample the next denoised latent and finally decode through the VAE. Classifier-free guidance combines conditional and null-conditioned predictions; random class dropout during training supplies the null embedding. Standard comparisons use 250 DDPM sampling steps. No actions, execution feedback or planning loop are defined. e03e04e09
5.5 Implementation flow
- Compress and corrupt
An off-the-shelf Stable Diffusion VAE provides the latent representation. Its encoder is frozen: a 256×256 RGB image becomes a 32×32×4 latent. Training adds Gaussian noise at a sampled diffusion timestep; DiT learns reverse-process statistics in this latent space. e03e04e08
- Convert space to tokens
Linearly embed non-overlapping p×p latent patches and add sine-cosine positional embeddings. Patch sizes 2, 4 and 8 change sequence length substantially while leaving downstream parameter counts essentially unchanged. e05
- Condition residual computation
adaLN-Zero regresses normalization scale and shift from summed timestep/class embeddings and adds dimension-wise gates before the attention and feedforward residual additions. Initializing those gates to zero makes each block an identity function. Alternatives append conditioning tokens, add cross-attention, or use ordinary adaLN. e06
- Decode predictions
After N transformer blocks, final normalization and a linear decoder map each token to p×p×2C values, where C is latent channel count. Rearrangement reconstructs noise and covariance tensors. DiT-XL uses 28 blocks, hidden width 1152 and 16 attention heads; /2 denotes latent patch size 2. e07
6. Experiments & results
DiT replaces the U-Net denoiser in latent image diffusion with a transformer over spatial latent patches. Conditioning design and the amount of computation per denoising pass both matter: adaLN-Zero works best among the tested blocks, and larger backbones or smaller patches improve ImageNet generation. Its strongest 256×256 result uses long training and classifier-free guidance; the experiments establish image-generation performance, not action-conditioned dynamics or control.
The supplied main-paper PDF ends after References. Its text refers to appendix Figure 12 and additional generated samples, but no appendix or separate supplement was supplied. Those additional plots and implementation details cannot be assessed here; the five visuals cover the available architecture, quantitative results and diagnostics. e11e15
6.1 Read the original evidence
Figure 6. Depth, width and token count provide distinct ways to improve the denoiser. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the three upper panels. Within each, the patch-size suffix stays fixed while S, B, L and XL change the transformer configuration. Then read the four lower panels: each fixes the backbone and compares /8, /4 and /2. The bold colored lines are the active comparisons; faint lines retain the wider sweep as background context. Every horizontal axis measures training steps and every vertical axis measures FID-50K. The lower row matters because a smaller patch creates more tokens without materially increasing the downstream parameter count. Figure 4 defines this relation, while Table 1 defines the backbone configurations used here. e05e07e09e11
What it supports. Across this sweep, larger backbones and smaller patches yield lower FID through the displayed training trajectories. The patch-size comparison shows why parameter count cannot describe the result by itself: a similar-sized set of weights performs more computation when processing a longer token sequence. The evidence therefore motivates examining computation alongside model size.
Where the evidence stops. Equal training iterations do not imply equal training cost. Figure 9 on inspected PDF page 8 separately plots estimated training compute, using an approximate backward-pass multiplier. Neither view establishes a universal law beyond these architectures and this ImageNet setup.
Table 2. The best guided DiT has the lowest tabulated FID, with a visible precision–recall tradeoff. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the three bottom rows together before comparing model families. They use the same DiT-XL/2 architecture without guidance and with classifier-free guidance scales 1.25 and 1.50. FID and sFID decrease in the preferred direction; Inception Score, precision and recall increase in their preferred direction. At scale 1.50, DiT reports FID 2.27 and recall 0.57, while the unguided row has FID 9.62 and recall 0.67. The LDM-4-G row at the same guidance scale gives FID 3.60 and recall 0.48. These are the 7M-step DiT results described on page 8, evaluated under the paper's FID-50K and 250-step convention. e09e12e13
What it supports. The strongest DiT row leads this table in FID and Inception Score, but the other columns prevent an indiscriminate claim of superiority. StyleGAN-XL reports a lower sFID, and LDM-4-G at scale 1.50 reports higher precision. Within DiT, stronger guidance improves FID while reducing recall, exposing a change in distribution coverage.
Where the evidence stops. Dashes are unreported metrics, not zeros. The table supplies neither equal training budgets nor uncertainty across seeds. Its 256×256 ranking also does not transfer wholesale to Table 3: at 512×512, StyleGAN-XL has a lower FID than DiT.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Class-conditional ImageNet generation at 256×256 DiT-XL/2, 7M training steps, classifier-free guidance scale 1.50; FID-50K with 250 DDPM sampling steps and ADM evaluation suite. The supplied text does not explicitly identify the ImageNet reference split. | 2.27 FID; 4.60 sFID; 278.24 IS; precision 0.83; recall 0.57. FID-50K ↓; sFID ↓; Inception Score ↑; precision/recall ↑ | LDM-4-G at guidance 1.50: FID 3.60, precision 0.87, recall 0.48. StyleGAN-XL: FID 2.30, sFID 4.02. Unguided DiT: FID 9.62, recall 0.67. DiT has the lowest FID in this table, but does not win every metric. Guidance improves FID while reducing recall. Cross-paper entries do not establish equal training compute. e09e12 |
| Class-conditional ImageNet generation at 512×512 Separate DiT-XL/2 model, 3M training iterations, guidance scale 1.50; standard FID-50K/250-step protocol. Precision/recall use 1000 real samples, following the cited baseline convention. | FID 3.04; precision 0.84; recall 0.54. FID-50K ↓; precision/recall ↑ | ADM-G with ADM-U: FID 3.85. StyleGAN-XL: FID 2.41. The result improves over the listed diffusion baselines, not over every generative model. Its 64×64×4 latent yields 1024 tokens and 524.6 Gflops per forward pass. e09e13 |
| Conditioning mechanism ablation Four DiT-XL/2 variants, ImageNet 256×256, up to 400K training steps; unguided FID-50K under the shared evaluation recipe. | The authors report adaLN-Zero's 400K-step FID as nearly half the in-context model's; it also beats vanilla adaLN and cross-attention. FID-50K ↓ | Forward costs: adaLN and adaLN-Zero 118.6 Gflops; in-context 119.4; cross-attention 137.6. Conditioning and initialization materially affect performance. The vanilla-adaLN comparison changes residual gating as well as initialization, so it does not isolate initialization alone. e06e09e10 |
| Model capacity versus sampling compute Models trained for 400K steps; sampling sweep over 16, 32, 64, 128, 256 and 1000 steps. This diagnostic uses FID-10K. | DiT-XL/2 at 128 steps: FID 23.7 with 15.2 Tflops. FID-10K ↓; per-image sampling Tflops ↓ | DiT-L/2 at 1000 steps: FID 25.9 with 80.7 Tflops. Extra denoising steps fail to close this capacity gap in the tested setting. These FID-10K values must not be compared directly with the long-trained FID-50K benchmark. e14 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 5. Residual modulation and initialization change quality even within the XL/2 configuration. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Every curve uses the XL/2 backbone and latent patch size, so read vertical separation at the same training step as a conditioning-design comparison. The horizontal axis ends at 400K steps; the vertical axis is FID-50K, where lower is better. Blue denotes adaLN-Zero, green ordinary adaLN, orange cross-attention and red in-context conditioning. Section 4 supplies the shared evaluation convention: EMA models, 250 DDPM sampling steps and no guidance unless stated. Section 5 gives the computation behind the curves: both adaLN versions cost 118.6 Gflops per pass, versus 119.4 for in-context and 137.6 for cross-attention. The plot does not show uncertainty bands. e06e08e09e10
What it supports. adaLN-Zero stays below the other tested conditioning strategies throughout the displayed training trajectory. The authors describe its FID at 400K steps as nearly half the in-context result. Cross-attention's additional computation does not compensate for its weaker result in this class-label setting, motivating the use of adaLN-Zero for the subsequent scaling experiments.
Where the evidence stops. The adaLN-to-adaLN-Zero comparison changes residual gating as well as initialization. It supports the combined design, but does not isolate zero initialization as the sole cause. These short conditioning inputs also differ from long text sequences.
Figure 10. More sampling steps do not erase the tested gap between denoiser capacities. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Here the horizontal axis changes from training steps to computation spent sampling one image. All models are evaluated after 400K training iterations. Each curve belongs to one backbone/patch-size combination, and its points sweep 16, 32, 64, 128, 256 and 1000 denoising steps. Follow one curve rightward to see the return from additional sampling steps, then compare different curves at similar horizontal positions to judge the cost of image quality. The vertical axis is FID-10K, unlike the FID-50K benchmark table. Section 5.2 provides a numerical example comparing L/2 with 1000 steps against XL/2 with 128 steps; individual points do not print their exact values. e14
What it supports. XL/2 at 128 steps achieves FID-10K 23.7 with 15.2 Tflops per image, compared with L/2 at 1000 steps achieving 25.9 with 80.7 Tflops. Thus, in this comparison, the larger denoiser is both better in FID and cheaper to sample from. Increasing iteration count does not replace the stronger learned denoising model.
Where the evidence stops. The conclusion is bounded by the tested checkpoints, step schedules and model family. These FID-10K results cannot be numerically merged with the long-trained FID-50K results, and sampling Gflops are not a measurement of end-to-end deployment latency.
7. Analysis & limitations
7.1 What the evidence leaves open
The evidence concerns class-conditioned still images. Text-conditioned generation is proposed as future work; temporal prediction, action generation and robot execution are untested. The pipeline also retains a convolutional VAE, so replacing the denoising backbone does not make the entire generator convolution-free. e04e15
Scaling plots support a trend within the tested design space, not an unrestricted scaling law. Fixed-iteration comparisons spend different amounts of compute; training-compute plots estimate cost using a factor of three for forward plus backward work. Tables and curves supply no confidence intervals or multi-seed uncertainty. e10e11e12e13e14
7.2 Questions for discussion
- How much of adaLN-Zero's gain comes from residual gating versus its zero initialization?
- Does the smaller-patch advantage persist when measured wall-clock training budgets and evaluation uncertainty are controlled?
8. Reproducibility audit
8.1 Requirements and known gaps
Reproduction requires ImageNet, the compatible pretrained VAE, ADM diffusion/evaluation conventions and substantial training compute. The authors use JAX and TPU-v3 pods, reporting roughly 5.7 iterations/second for XL/2 on a TPU v3-256 pod at global batch 256. e08e09
The supplied setup leaves exact VAE checkpoint identity, software versions, random seeds, class-dropout probability and the explicit ImageNet reference split unspecified. It delegates covariance and embedding details to ADM. Recovering these choices would be necessary before claiming an exact replication. e04e08e09
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Separate residual gating from zero initialization
Reader-proposed check: train XL/2 with ordinary adaLN, the reported adaLN-Zero block, and an otherwise identical gated block whose residual gates start nonzero. Hold the VAE, data order, class dropout, optimizer, output-layer initialization and sampling settings fixed; use paired random seeds and multiple independent runs. Compare unguided EMA FID-50K after 400K updates with 250 DDPM steps, and report computation alongside quality. The decisive comparison is between the two gated variants, which have the same structure. If zero initialization consistently improves that comparison beyond sampling and training variability, it supports an initialization-specific contribution. If they tie while both outperform ordinary adaLN, gating is a plausible explanation requiring further tests. e06e08e09e10
Check 2: Test whether the sampling-compute gap survives matched evaluation
Reader-proposed check: evaluate 400K-step L/2 and XL/2 checkpoints across the paper's six sampling-step counts using matched class-label draws, starting noise seeds and the same evaluator. First test the stated 1000-step L/2 versus 128-step XL/2 comparison with repeated FID-10K sample sets. Then report a separate FID-50K comparison to assess estimator sensitivity, and compare the quality frontiers at approximately matched sampling Gflops. Preserve guidance settings and exclude VAE costs consistently from the paper-style Gflops comparison; measure end-to-end latency separately. A reversal that persists across repeated sample sets or a matched-budget frontier favoring L/2 would weaken the proposed capacity advantage in this setting. This is a proposed check, not a reproduced result. e04e09e14
8.3 Reading coverage
Visual audit: All 11 supplied PDF pages were rendered and visually inspected, including the title/author block and watermark, Figures 1–10, Tables 1–3, equations, training/evaluation/compute details, conclusion and reference pages. The five final crops were separately viewed with axes, legends and table headers intact. Figure 3's arrows and residual gate placement were cross-checked against Section 3.2; its label y corresponds to class c in the prose. Page 8 supplies the uncropped training-compute evidence and page 9 supplies the uncropped 512×512 table and its 1000-real-sample precision/recall note. No appendix is present in the supplied PDF; its referenced Figure 12 and additional samples remain uninspected.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11. Appendix coverage: not present.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract
- 1. Introduction
- 2. Related Work
- 3. Diffusion Transformers
- 3.1. Preliminaries
- 3.2. Diffusion Transformer Design Space
- 4. Experimental Setup
- 5. Experiments
- 5.1. State-of-the-Art Diffusion Models
- 5.2. Scaling Model vs. Sampling Compute
- 6. Conclusion
- Acknowledgements
- References
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- Identity and edition: the inspected title and authors exactly match the catalog: Scalable Diffusion Models with Transformers, William Peebles and Saining Xie. The title-page watermark identifies the ICCV open-access version as identical to the accepted version except for the watermark, and distinguishes it from the final IEEE proceedings version. No numbered revision or revision date is stated; no other edition was inspected.
- Bibliographic discrepancy: the supplied catalog BibTeX gives pages 4172–4182, whereas the inspected PDF prints 4195–4205. This report follows the supplied PDF without altering the catalog.
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout. This acquisition omission was addressed by visually inspecting all 11 PDF pages and five final original crops.
- Separate supplemental material availability has not been fully verified. The supplied 11-page PDF ends with References and contains no appendix, despite references to appendix Figure 12 and additional samples. Those materials were not read.
- Code and project links were not opened; implementation files, external references and other editions were not inspected. No experiments were reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1 (printed p. 4195), title, author block, watermark and author footnote; PDF p. 11 footer (4205)
Title and authors match the catalog. Peebles is affiliated with UC Berkeley and Xie with New York University; the footnote locates Peebles's internship at Meta AI, FAIR. The watermark identifies the open-access accepted version.
Go to primary source ↓e02PDF pp. 2–3, Sections 1–2, architecture motivation and Architecture complexity
The work replaces the U-Net diffusion backbone with a transformer and studies computation versus image quality, explaining why parameter count misses resolution-dependent cost.
Go to primary source ↓e03PDF p. 3, Section 3.1, Diffusion formulation
Gaussian corruption, learned reverse mean/covariance, noise squared-error training and separate variational covariance training define the denoising problem and sampling sequence.
Go to primary source ↓e04PDF p. 4, Section 3.1, Classifier-free guidance and Latent diffusion models
Conditional/null prediction mixing uses scale s; random condition dropout learns a null embedding. The frozen encoder supplies latents, and a learned decoder maps sampled latents to pixels; the pipeline combines a convolutional VAE with transformer diffusion.
Go to primary source ↓e05PDF p. 4, Section 3.2, Patchify and Figure 4
Latent patches become tokens with sine-cosine positions. For latent side I and patch size p, T=(I/p)^2; tested p values are 2, 4 and 8.
Go to primary source ↓e06PDF p. 3, Figure 3; PDF pp. 4–5, Section 3.2, four DiT block designs
The diagram and text place normalization modulation before attention/feedforward and gates before residual additions. adaLN-Zero predicts zero-initialized residual scales; alternative conditioning uses appended tokens, cross-attention or adaLN.
Go to primary source ↓e07PDF p. 5, Table 1 and Section 3.2, Model size and Transformer decoder
Table 1 lists S/B/L/XL configurations; XL has 28 layers, width 1152 and 16 heads. The decoder produces p×p×2C values per token for spatial noise and diagonal covariance outputs.
Go to primary source ↓e08PDF pp. 5–6, Section 4, Training, Diffusion and Compute
Reports AdamW, learning rate 10⁻⁴, batch 256, no weight decay, horizontal flips, EMA 0.9999, a Stable Diffusion VAE with factor-eight downsampling, 1000-step linear variance schedule, JAX and TPU v3-256 throughput of about 5.7 iterations/second for XL/2.
Go to primary source ↓e09PDF p. 6, Section 4, Evaluation metrics and Training
Uses EMA models, FID-50K, 250 DDPM sampling steps and ADM's TensorFlow evaluation suite; guidance is absent unless stated. Secondary metrics include sFID, IS and precision/recall. The explicit ImageNet reference split is not identified here.
Go to primary source ↓e10PDF p. 5, Figure 5; PDF p. 6, Section 5, DiT block design
adaLN-Zero's curve is below the alternatives. At 400K steps its FID is described as nearly half the in-context result. Text gives forward costs of 118.6, 119.4 and 137.6 Gflops for the relevant variants.
Go to primary source ↓e11PDF p. 6, Figure 6; PDF p. 8, Section 5 and Figures 8–9
The 12-model sweep crosses four sizes and three patch sizes. Larger backbones and smaller patches improve FID. Training compute is estimated as model Gflops × batch size × steps × 3; the appendix Figure 12 is referenced but not supplied.
Go to primary source ↓e12PDF p. 8, Section 5.1, 256×256 ImageNet; PDF p. 9, Table 2 and continuation
After 7M steps, XL/2-G at cfg=1.50 reports FID 2.27, sFID 4.60, IS 278.24, precision 0.83 and recall 0.57. LDM-4-G at cfg=1.50 has FID 3.60; StyleGAN-XL has FID 2.30 and sFID 4.02. Unguided DiT has FID 9.62 and recall 0.67.
Go to primary source ↓e13PDF p. 9, Section 5.1, 512×512 ImageNet; Table 3 including caption
The separate 3M-step XL/2 model uses 1024 tokens and 524.6 Gflops. Guided FID is 3.04 versus ADM-G/ADM-U 3.85 and StyleGAN-XL 2.41. DiT precision/recall are 0.84/0.54; the caption specifies 1000 real samples for these two metrics.
Go to primary source ↓e14PDF p. 9, Figure 10, caption and Section 5.2
At 400K training steps the six sampling-step counts are 16–1000. XL/2 with 128 steps obtains FID-10K 23.7 using 15.2 Tflops; L/2 with 1000 obtains 25.9 using 80.7 Tflops.
Go to primary source ↓e15PDF p. 8, appendix Figure 12 reference; PDF p. 9, Section 6 and appendix-sample reference; PDF pp. 10–11, References
Text-to-image use is proposed as future work. The source refers to appendix metrics and samples but the supplied PDF concludes with references and has no appendix or dedicated limitations section.
Go to primary source ↓8.5 Primary sources
Scalable Diffusion Models with Transformers ↗
PDF · 6,960 extracted words
Source fingerprint
29cc26d5497bb7d40ff2d02fdf6e6c9cdebd1167a8b7c3b1ecf012336768a6bf