CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer
1. Paper overview
In one sentence: CogVideoX combines temporal video compression, modality-specific normalization and full spatiotemporal attention to improve text-conditioned motion generation, at the cost of expensive attention and a substantial data pipeline. e-probleme-architecturee-vaee-transformere-ablationse-benchmarkse-human
| At a glance | What to know |
|---|---|
| Research problem | Source description The authors target temporally consistent videos with substantial motion and multi-event narratives. High-dimensional video makes long sequences expensive, while weak captions and separated spatial/temporal processing can obstruct semantic alignment and object consistency. The paper asks how to build a scalable text-to-video generator that addresses these problems through representation, architecture, training and data preparation. e-probleme-transformer |
| Core mechanism | Source description The proposed backbone combines an 8 × 8 × 4 causal video VAE with an expert transformer. Text and visual tokens share computation, but receive distinct timestep-conditioned normalization. The design targets coherent visual generation, without an action decoder. e-architecturee-vaee-transformer |
| A key reported result | Text-to-video benchmark comparison: CogVideoX-5B: 70.95; 69.5; 3.36. Multiple Objects; Dynamic Quality; GPT4o-MTScore (higher is better). Table 3: five selected VBench dimensions plus Dynamic Quality and GPT4o-MTScore; benchmark-specific sample counts, seeds and matched generation settings are not fully specified. LaVie-2: 64.88 Multiple Objects; OpenSora V1.2: 63.7 Dynamic Quality; VideoCrafter-2.0: 2.68 GPT4o-MTScore. These are separate metric comparisons, not an aggregate success rate. Table 3 gives 5B four column wins; 2B wins Dynamic Degree at 66.39 versus 62.22 for 5B. The prose’s five-of-seven claim for 5B conflicts with the table. e-benchmarks |
| Reading caution | Reader analysis Full attention has a computational cost: Table 8 reports 9.60 seconds per DiT forward step versus 4.17 for 2D+1D at 768 × 1360 for five seconds on H800. The collapse of the separated-attention baseline in Figures 8 and 10 supports this tested configuration, not a universal impossibility result. e-runtimee-ablationse-timestep-loss |
Core contributions
- Source description
The proposed backbone combines an 8 × 8 × 4 causal video VAE with an expert transformer. Text and visual tokens share computation, but receive distinct timestep-conditioned normalization. The design targets coherent visual generation, without an action decoder. e-architecturee-vaee-transformer
- Source description
The training recipe combines variable-duration frame packing, progressive resolution increases, explicit uniform timestep sampling and dense recaptioning. The authors announce 2B and 5B releases, including image-to-video variants; that announcement does not establish artifact usability in this reading. e-probleme-traininge-samplinge-captions
Figure 3. Separate modality modulation surrounds a shared transformer. Original paper, p. 3 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start at the bottom: the text encoder feeds the green text tokens, while the VAE feeds the purple visual tokens from the opposite side. Both streams move upward through a shared 3D Full Attention block and a shared Feed Forward block. The central timestep t drives two Expert AdaLN branches. Follow their side arrows to Scale & Shift and Gate operations; the dashed paths terminate at residual additions. Section 2.2 confirms that these branches handle the two modalities independently while processing a concatenated sequence. The drawing shows the transformer core, so use Section 2 for the subsequent unpatchifying and VAE decoding. e-architecturee-transformere-traininge-ablations
What it supports. The expert distinction lies in how text and visual hidden states are modulated. Shared attention can exchange information across both modalities, while separate normalization accommodates their different feature distributions. This is an architectural reason to test Expert AdaLN against a model with shared normalization and against MMDiT.
Where the evidence stops. Figure 3 does not depict the noise-injection or iterative sampling procedure. Its video input is part of the architecture explanation, not evidence that text-to-video inference requires a source clip. No action-output branch appears.
2. Motivation
2.1 The problem and the proposed response
The authors target temporally consistent videos with substantial motion and multi-event narratives. High-dimensional video makes long sequences expensive, while weak captions and separated spatial/temporal processing can obstruct semantic alignment and object consistency. The paper asks how to build a scalable text-to-video generator that addresses these problems through representation, architecture, training and data preparation. e-probleme-transformer
2.2 What this reading follows
A video generator must preserve what an object is while allowing where it is—and what it does—to change. CogVideoX addresses that problem at several levels: a causal VAE compresses redundant frames, an expert transformer mixes textual and visual information, and a training pipeline supplies variable-length clips with detailed captions. Read the architecture together with the ablations: the paper’s argument depends on representation quality, optimization stability and semantic alignment, rather than one component alone. The results establish improvements in video-generation metrics and human ratings. They leave open how these representations would support action-conditioned prediction or control. e-probleme-architecturee-vaee-transformere-ablationse-benchmarkse-human
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Foundational work |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
This table preserves the labels recorded at reading time. The current major category is Components of WAMs. View the current classification.
3.1 Evidence-based assessment
Supports the recorded classification
The recorded foundational/video-generation-backbone classification is supported. The architecture predicts visual latents conditioned on text, with an optional image condition; it does not jointly predict executable actions or infer actions through inverse dynamics. Thus the catalog’s architecture, prediction-paradigm and quadrant fields remain Not applicable for the world-action taxonomy. e-architecturee-transformere-i2v
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 1. Reduce the visual sequence before paying for full attention
The first bottleneck is the amount of video that the transformer must represent. CogVideoX’s VAE reduces spatial dimensions by eight and the temporal dimension by approximately four, preserving a special first-frame offset. This creates a compact grid for patchification. The VAE is learned with reconstruction, perceptual and regularization objectives, then a discriminator loss, so compression quality is an empirical constraint rather than a free reduction in cost. Table 1 makes that constraint visible: more channels can improve PSNR, while more aggressive compression is difficult to optimize. Reader interpretation: the value of full attention depends partly on whether the compressed representation preserves the details that attention must connect. The causal VAE and bidirectional denoiser solve different parts of this problem. e-vaee-vae-ablatione-architecturee-transformere-training
Figure 4(a–b). Temporal compression and causal context exchange make long clips more tractable. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Panel (a) separates two stages that downsample both space and time from a stage that downsamples space only. The decoder mirrors these operations. Read the shape labels carefully: the input is (4f + 1) × 8h × 8w, while the latent grid is (f + 1) × h × w. The first frame creates an offset around the nominal temporal compression factor. Panel (b) explains distributed causal convolution. Rank 0 supplies first-frame padding; later ranks receive the preceding rank’s tail. The marked tail and padding lengths are the temporal kernel size minus one, matching Section 2.1. e-vaee-transformer
What it supports. The VAE reduces the grid that the transformer must process while learning reconstruction across time. Its causal structure also makes temporal partitioning straightforward: a rank needs a bounded segment from its predecessor. This supports long-clip VAE training without requiring the full clip on one device.
Where the evidence stops. Causality here describes the VAE convolution. It does not impose a causal attention mask on the diffusion transformer, which uses full spatiotemporal attention. This diagram alone also says nothing about action-conditioned physical accuracy.
5.2 2. Normalize modalities separately, then let them exchange information
Text embeddings and video latents need to share information, but their numerical distributions need not match. Figure 3 routes the diffusion timestep into separate text and vision modulation branches, then processes the concatenated sequence through shared attention and feed-forward blocks. This is the specific meaning of expert processing in this architecture. Section 2.2 motivates full spatiotemporal attention with a moving-object example: separated spatial and temporal operations force some information to travel indirectly through other patches. Figure 8 tests both ideas, including a parameter-matched MMDiT comparison and an attention-pattern replacement. Reader interpretation: these experiments support the proposed combination under the displayed training conditions, while Table 8’s runtime increase explains why the more connected attention pattern still requires a cost-benefit decision. e-architecturee-transformere-ablationse-runtime
5.3 3. Align the data pipeline, optimization and inference prompt
The generator’s conditioning distribution is constructed deliberately. Training clips receive detailed descriptions through a captioning pipeline, and the inference prompt is expanded by a language model to resemble those descriptions. Mixed-duration frame packing admits images and videos of different lengths, while progressive training increases resolution before a final high-quality fine-tuning stage. Explicit uniform sampling addresses a different issue: uneven timestep coverage across data-parallel ranks can make batches differ in loss scale. Table 9 evaluates the resulting models at fixed timesteps, which is more informative than comparing only aggregate curve smoothness. Reader interpretation: a faithful reproduction must preserve both the semantic pipeline and the optimizer’s sampling distribution. Changing captions, prompt expansion and timestep sampling together would make the cause of a quality difference difficult to identify. e-captionse-upsamplinge-traininge-finetunee-samplinge-timestep-loss
5.4 Training and inference
During training
The VAE is trained at 256 × 256 on 17-frame clips sampled at 8 or 16 fps, then fine-tuned on 161-frame clips using context parallelism. Both stages combine L1 reconstruction, LPIPS and KL losses; a 3D-discriminator GAN loss is introduced after a few thousand steps. Exact loss weights and the introduction step are unspecified. e-vae
The diffusion model uses v-prediction, zero SNR and the LDM noise schedule. Images are treated as single-frame videos; frame packing accommodates mixed durations and resolutions. Table 5 lists four stages with 400k, 220k, 120k and 10k steps. Final fine-tuning uses a higher-quality 20% data subset, with an acknowledged slight semantic degradation. e-traininge-confige-finetune
Training uses approximately 35M filtered single-shot clips averaging six seconds and 2B aesthetic-filtered images. Six Video-LLaMA filters are learned from 20,000 labeled videos. Dense captions combine Panda70M and CogVLM descriptions, GPT-4 summaries distilled into Llama 2, and subsequently CogVLM2-Caption. Appendix G specifies one frame every two seconds and 50,000 summary-model training examples. e-datae-captionse-upsampling
Explicit Uniform Sampling partitions diffusion timesteps 1 through T into n intervals, one per data-parallel rank. Each rank samples uniformly inside its interval. This addresses finite-batch timestep imbalance; Equation 1 is cited as background noise-prediction notation, while the actual training setting specifies v-prediction. e-samplinge-training
During inference
A language model expands the prompt to resemble the detailed training captions, then text-conditioned diffusion produces visual latents for decoding. Table 7 measures 50 inference steps on H800, reporting 113 seconds and 26 GB for 5B at 480 × 720 for six seconds. Its precision label is only “bf”; sampler and guidance settings are not specified there. e-upsamplinge-architecturee-runtime
5.5 Implementation flow
- Compress time and space
The VAE applies 3D convolutions with temporal padding placed before the current input. Variant B uses 16 latent channels and 8 × 8 × 4 compression. Temporal context parallelism sends the preceding segment's last k − 1 elements to the next rank, where k is the temporal kernel size. e-vaee-vae-ablation
- Build a multimodal sequence
Patchified video latents form z_vision; T5 supplies z_text. Concatenation is along the sequence dimension. The transformer’s described 3D RoPE allocates 3/8, 3/8 and 2/8 of channels to the two spatial coordinates and time. Table 6 specifically lists sinusoidal encoding for 2B and RoPE for 5B. e-architecturee-transformere-config
- Mix information with modality-specific modulation
Diffusion timestep t drives separate text and vision Expert AdaLN branches, which modulate scale, shift and gates around shared 3D full attention and feed-forward blocks. Full attention connects visual positions across space and time; the VAE's causal convolutions do not make the entire denoiser autoregressive. e-architecturee-transformere-training
- Restore video and support conditional variants
The output visual sequence is unpatchified and decoded by the VAE. Image-to-video fine-tuning adds an image encoded by the VAE along the noisy input’s channel dimension, with strong conditioning-image noise during training. Appendix I instead performs video-to-video generation through an intermediate caption. e-architecturee-i2ve-v2v
6. Experiments & results
CogVideoX generates videos from text using a temporally compressing VAE and a diffusion transformer that jointly processes text and visual tokens. Its distinguishing choice is modality-specific adaptive normalization around shared attention and feed-forward computation. Full spatiotemporal attention supports information exchange across moving objects, while frame packing, progressive resolution training and dense captions address data efficiency and alignment. The inspected revision reports generation up to 10 seconds at 16 fps and 768 × 1360 pixels. Experiments support several design choices and competitive video quality, with important protocol gaps and internal reporting discrepancies. The output is video; no action policy or environment-control loop is provided.
6.1 Read the original evidence
Table 3. The benchmark table supports specific strengths rather than uniform dominance. Original paper, p. 10 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Compare models vertically within a column; the seven columns are different metrics and should not be averaged casually. The first five are selected VBench dimensions. Dynamic Quality combines dynamics with quality assessment, while GPT4o-MTScore measures metamorphic changes using GPT-4o. A dash is an unreported result, not zero. The 5B row leads Human Action, Multiple Objects, Dynamic Quality and GPT4o-MTScore. The 2B row has the strongest Dynamic Degree, while T2V-Turbo leads Scene and VideoCrafter-2.0 leads Appearance Style. The source’s bold entries make these winners visible; retained green outlines are original PDF citation-link borders. e-benchmarks
What it supports. CogVideoX-5B scores 70.95 on Multiple Objects, 69.5 on Dynamic Quality and 3.36 on GPT4o-MTScore. Scaling from 2B to 5B improves those three entries, but Dynamic Degree falls from 66.39 to 62.22. The table supports a multidimensional improvement with a visible exception.
Where the evidence stops. The adjacent prose claims five wins for 5B, whereas the table shows four; five applies to the family when 2B’s Dynamic Degree win is included. Matched generation settings, uncertainty and benchmark-specific sample counts are not fully documented.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Text-to-video benchmark comparison Table 3: five selected VBench dimensions plus Dynamic Quality and GPT4o-MTScore; benchmark-specific sample counts, seeds and matched generation settings are not fully specified. | CogVideoX-5B: 70.95; 69.5; 3.36. Multiple Objects; Dynamic Quality; GPT4o-MTScore (higher is better) | LaVie-2: 64.88 Multiple Objects; OpenSora V1.2: 63.7 Dynamic Quality; VideoCrafter-2.0: 2.68 GPT4o-MTScore. These are separate metric comparisons, not an aggregate success rate. Table 3 gives 5B four column wins; 2B wins Dynamic Degree at 66.39 versus 62.22 for 5B. The prose’s five-of-seven claim for 5B conflicts with the table. e-benchmarks |
| Video VAE reconstruction WebVid validation videos, 256 × 256 resolution and 17 frames; Table 2. | CogVideoX VAE: 29.1 PSNR and 85.5 Flickering. PSNR (higher); adjacent-frame L1 Flickering (lower) | Open-Sora: 28.5 and 92.4; Open-Sora-Plan: 27.6 and 90.2. Reconstruction improves on these metrics, but the competitors use fewer latent channels. This does not isolate architecture at a matched latent bitrate or measure text-to-video generation. e-vae-comparisone-vae-ablation |
| Human-rated video generation against Kling 100 crafted prompts; Kling version 2024.7 versus CogVideoX-5B; four aspects rated 0, 0.5 or 1, with a separate total described on a 0–5 scale. | CogVideoX-5B: 0.495; 0.667; 2.74. Instruction Following; Physics Simulation; Total Score | Kling: 0.367; 0.561; 2.17. Reported ratings favor CogVideoX, but they are not pairwise win percentages. Appendix J caps total scores at 2 for instruction failures without fully defining aggregation. Rater count and uncertainty are absent; physics ratings are perceptual judgments. e-humane-human-protocol |
| Explicit uniform timestep sampling ablation Table 9, validation loss after 40k training steps at diffusion timesteps 100, 300, 500, 700 and 900; validation dataset is not named in that table. | With explicit uniform sampling: 0.216, 0.126, 0.116, 0.129, 0.157. Validation loss (lower is better) | Without: 0.222, 0.130, 0.119, 0.133, 0.161. All five listed timesteps improve. Figure 8 adds WebVid test-set generation diagnostics on 500 videos, but the table alone neither establishes every timestep nor quantifies seed variability. e-timestep-losse-ablations |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 1. The selected VAE balances temporal compression, channel count and reconstruction. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read down each column to keep the representation and its metrics together. The SDXL baseline compresses only space and uses four latent channels. Variants A, B and C keep the same 8 × 8 × 4 compression while increasing channels from 8 to 16 to 32. Variants D and E change compression further, so they are not pure channel-count comparisons. The source caption identifies B as the pretraining choice. Lower Flickering and higher PSNR are preferred; the caption defines Flickering as the L1 difference between adjacent frames. These numbers belong to this ablation, separately from Table 2’s cross-model reconstruction comparison. e-vae-ablatione-vaee-vae-comparison
What it supports. B reports Flickering 86.3 and PSNR 28.7, compared with 93.2 and 28.4 for the baseline. C reaches the highest PSNR, 30.5, but uses twice B’s latent channels. The selected configuration therefore does not simply maximize the reconstruction score shown in the table.
Where the evidence stops. Adjacent-frame L1 can reflect genuine motion as well as unwanted flicker; interpret that metric with reconstruction fidelity. The table does not give uncertainty or a complete evaluation protocol, and the baseline comparison changes several representation properties.
Figure 8(a–d). The diagnostic curves connect architectural choices to quality and stability. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow training steps horizontally. Panels (a), (b) and (c) report FVD, where lower is better; panel (d) reports CLIP4Clip, where higher is better. The source caption specifies 500 WebVid test videos. In the architecture panels, MMDiT1 matches Expert AdaLN’s parameter count, while MMDiT2 matches depth and uses twice as many parameters. Panel (b) has a much larger vertical scale and an early-step inset because separated 2D+1D attention deteriorates sharply. Panel (c) changes timestep sampling. Keep these comparisons distinct: architecture, attention pattern and sampling are separate interventions, even though their curves appear in one figure. e-ablationse-transformere-samplinge-timestep-loss
What it supports. Expert AdaLN ends with better FVD and CLIP4Clip than the same-parameter MMDiT1 and the no-expert baseline. The tested separated-attention model suffers a marked FVD increase, while full attention remains stable. Explicit uniform sampling improves the later FVD trajectory; these diagnostics support the paper’s mechanism choices.
Where the evidence stops. The curves contain no seed-level uncertainty, and the model/compute controls are incompletely specified. MMDiT2 is not parameter-matched. The observed 2D+1D collapse should not be generalized to every possible separated-attention implementation or training recipe.
Table 9. Fixed-timestep validation tests more than the smoothness of a training curve. Original paper, p. 17 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. The columns are diffusion timesteps, not training iterations. Both rows are evaluated after 40k training steps, as specified in the table caption. Compare the two rows within each column: the upper row omits explicit uniform sampling and the lower row uses it. Section 3.3 explains that the intervention partitions the timestep range among data-parallel ranks; it does not replace diffusion with a new predictive task. Looking at fixed timesteps helps distinguish lower conditional validation loss from the simpler effect of changing which noise levels happen to appear in a batch. e-timestep-losse-samplinge-ablations
What it supports. All five reported losses decrease: for example, timestep 100 changes from 0.222 to 0.216, and timestep 900 from 0.161 to 0.157. This supports an optimization benefit beyond a visually smoother aggregate loss curve, though it does not establish a gain at every possible noise level.
Where the evidence stops. The table does not identify its validation dataset, uncertainty or seed count. Figure 8’s 500-video WebVid protocol should not automatically be assigned to these loss measurements. Generation quality requires its own evaluation.
7. Analysis & limitations
7.1 What the evidence leaves open
Full attention has a computational cost: Table 8 reports 9.60 seconds per DiT forward step versus 4.17 for 2D+1D at 768 × 1360 for five seconds on H800. The collapse of the separated-attention baseline in Figures 8 and 10 supports this tested configuration, not a universal impossibility result. e-runtimee-ablationse-timestep-loss
Configuration reporting needs reconciliation. Table 5 lists 700k sequence length for later stages, while Table 6 gives an 82k maximum. Section 3.2 describes a 512px intermediate stage, whereas Table 5 lists 480 × 720. These differences are retained without guessing a correction or assigning them to an unverified revision. e-traininge-config
The selected qualitative examples, including caption-mediated regeneration, cannot establish general physical fidelity. Human physics scoring concerns perceived realism, and the paper supplies neither action-conditioned rollout tests nor executed-control experiments. High-quality fine-tuning also trades some semantic ability for cleaner visuals. e-examplese-v2ve-human-protocole-finetunee-architecture
7.2 Questions for discussion
- Does Expert AdaLN retain its advantage when parameter count and training compute are both controlled?
- How much of instruction-following performance depends on recaptioning and inference-time prompt expansion?
8. Reproducibility audit
8.1 Requirements and known gaps
A faithful training attempt needs the filtered clip inventory, captions, VAE configuration and staged diffusion setup. Some parameters are given, including BF16 training and a lowest aesthetic value of 4.5; missing items include the evolving optical-flow/aesthetic threshold schedule, complete data composition, peak learning rate, software versions and total training hardware/compute. Encoder freezing and exact VAE-to-diffusion training boundaries are not fully specified. e-datae-captionse-vaee-confige-filtering
Reader-proposed checks should first isolate timestep sampling at matched data and compute, and then compare Expert AdaLN with the parameter-matched MMDiT1 baseline. Preserve the frozen evaluation prompt list, model configuration, seeds and caption-upsampled inputs for each comparison. These checks test mechanisms more directly than trying to reproduce the full headline benchmark without its complete protocol. e-samplinge-ablationse-upsamplinge-benchmarks
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Does stratified timestep sampling improve fixed-timestep generalization?
Reader-proposed experiment: train two otherwise identical small configurations with the same initial weights, global batch, data order, optimizer and compute budget. Change only ordinary per-rank uniform timestep sampling versus the paper’s rank-partitioned sampling, keeping the global timestep marginal uniform. Repeat with several matched seeds. At 40k steps, measure held-out loss at 100, 300, 500, 700 and 900 and evaluate the same generated prompt set. Support would be consistent paired reductions across the fixed-timestep losses, not merely a smoother training curve. If the curves smooth but held-out losses do not improve, the claimed optimization benefit would be weakened. This experiment has not been run. e-samplinge-traininge-timestep-losse-ablations
Check 2: Is Expert AdaLN’s advantage robust to the comparison budget?
Reader-proposed experiment: compare Expert AdaLN, the no-expert baseline and parameter-matched MMDiT1 using the same VAE, captions, full attention, position encoding, diffusion schedule and evaluation inputs. First match training steps and examples, then repeat the comparison at matched total training compute; report both budgets explicitly. Use the Figure 8 protocol of 500 WebVid test videos for FVD and CLIP4Clip, with repeated seeds and unchanged prompt processing. The mechanism is supported if the expert model’s advantage persists across both budgeting choices. A reversal under compute matching would show that the displayed parameter comparison is insufficient to establish efficiency. This is a proposed test, not a reproduced result. e-ablationse-transformere-captionse-upsamplinge-config
8.3 Reading coverage
Visual audit: All six supplied text chunks were read individually, covering the 30-page paper and Appendices A–K. The declared pages were rendered and actually inspected, including the title/author/version page, every numbered figure, all tables, appendix training/runtime details, human-rating criteria and the image-only page 27. Every final crop was separately viewed. Figure 3’s upward data flow, modality-specific scale/shift/gates and residual additions were checked against Section 2.2; Figure 4’s first-frame padding and predecessor-to-successor tail exchange were checked against Section 2.1. Figure 8 legends, metric directions and MMDiT controls were checked against its caption. Pages 11–14 (references) and 18 (related-work prose) were read as text but are not asserted to have been visually inspected. Static qualitative grids do not substitute for playable video inspection. The Table 3 win-count discrepancy and the Tables 5–6 configuration discrepancies remain explicit in the report.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 15, 16, 17, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract and Sections 1–5 (PDF pp. 1–10)
- Acknowledgments and references (pp. 11–14)
- Appendix A: Training Details (pp. 15–16)
- Appendices B–E: losses, examples, image-to-video and related works (pp. 17–22)
- Appendices F–H: prompt upsampling, dense caption generation and examples (pp. 23–25)
- Appendix I: video-to-video via captions, including the image-only page 27 (pp. 26–27)
- Appendices J–K: human evaluation and filtering details (pp. 28–30)
Outside the original text pass
- Text extraction does not reconstruct figure pixels or preserve equation/table layout; downstream reading must inspect the retained original PDF.
- Qualitative sample imagery on PDF page 27 is preserved in the PDF but is not reconstructed in reading text.
- Availability of external supplements, code, datasets and videos beyond the retained PDF has not been fully verified.
- Version scope: the inspected artifact is arXiv:2408.06072v3, dated 26 March 2025, and marked Published as a conference paper at ICLR 2025. Its title and all 18 authors match the supplied catalog. Earlier revisions and a separate venue PDF were not supplied for comparison.
- The supplied extraction does not reconstruct figure pixels or preserve equation/table layout. Original PDF images were inspected separately; page 27 contains qualitative image grids rather than a missing text section.
- External supplements, code, datasets and playable videos beyond the retained PDF were not inspected or their availability verified. No experiments were reproduced. Static PDF frames cannot establish continuous-video playback quality.
- No separate supplemental artifact was supplied. References were read as bibliography entries, not as independently inspected research sources.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e-identityPDF p. 1, title block and arXiv margin
Title, all 18 author identities and Tsinghua University/Zhipu AI affiliations are visible; arXiv:2408.06072v3 is dated 26 March 2025 and marked ICLR 2025.
Go to primary source ↓e-problemPDF pp. 1–3, Abstract, Section 1 and contribution list
The paper targets long, coherent text-to-video generation, reports up to 10 seconds at 16 fps and 768 × 1360, and announces 2B/5B text- and image-conditioned models.
Go to primary source ↓e-architecturePDF p. 3, Section 2 and Figure 3
VAE visual latents and T5 text embeddings are concatenated; timestep-conditioned expert modulation surrounds shared attention and feed-forward blocks; output visual latents are decoded.
Go to primary source ↓e-vaePDF pp. 4–5, Section 2.1 and Figure 4(a–b)
Causal convolution, 8 × 8 × 4 grid compression, first-frame shape offsets, k − 1 temporal context exchange, 17-to-161-frame training and reconstruction/perceptual/KL/GAN objectives are described.
Go to primary source ↓e-vae-ablationPDF p. 4, Table 1, all columns and caption
Variant B has 16 channels, 8 × 8 × 4 compression, Flickering 86.3 and PSNR 28.7; baseline values are 93.2/28.4, and variant C achieves PSNR 30.5. B is used for pretraining; Flickering is adjacent-frame L1.
Go to primary source ↓e-transformerPDF p. 5, Section 2.2 and Figure 5
Patchification, 3D RoPE channel allocation, separate modality AdaLN and full text-video attention are described. Figure 5 illustrates the indirect information path under separated spatial/temporal attention.
Go to primary source ↓e-trainingPDF pp. 6–7, Figure 6 and Sections 3, 3.1–3.2
The model uses image/video mixing, v-prediction, zero SNR, LDM scheduling, mixed-duration frame packing and progressively increased resolutions; the prose names 256px, 512px and 768px.
Go to primary source ↓e-samplingPDF p. 7, Section 3.3, Equation 1 and following paragraph
A background noise-prediction loss motivates uniform timesteps. Explicit sampling divides 1…T into n rank-specific intervals to reduce finite-batch imbalance.
Go to primary source ↓e-dataPDF p. 7, Section 3.4, Data and Video Filtering
Approximately 35M clips averaging six seconds and 2B images are used. Six Video-LLaMA filters use 20,000 labels; optical-flow and aesthetic thresholds change during training without a full schedule.
Go to primary source ↓e-captionsPDF p. 8, Section 3.4, Figure 7 and Video Captioning
Panda70M short captions and CogVLM frame captions feed GPT-4 summarization, distilled into Llama 2; generated captions subsequently supervise CogVLM2-Caption.
Go to primary source ↓e-ablationsPDF pp. 8–9, Section 4.1 and Figure 8(a–d), including caption
WebVid test diagnostics use 500 videos. MMDiT1 matches Expert AdaLN parameter count; MMDiT2 has equal depth and twice the parameters. Panels a–c show FVD, panel d CLIP4Clip; full attention avoids the displayed separated-attention collapse.
Go to primary source ↓e-vae-comparisonPDF p. 9, Section 4.2.1, VAE Reconstruction Effect and Table 2
On WebVid validation 256 × 256, 17-frame videos, Ours scores Flickering 85.5/PSNR 29.1 versus Open-Sora 92.4/28.5 and Open-Sora-Plan 90.2/27.6; competitors use fewer channels.
Go to primary source ↓e-benchmarksPDF pp. 9–10, Section 4.2.1 and Table 3, all rows/columns
5B leads Human Action 96.8, Multiple Objects 70.95, Dynamic Quality 69.5 and GPT4o-MTScore 3.36. 2B leads Dynamic Degree 66.39; T2V-Turbo leads Scene 55.58 and VideoCrafter-2.0 Appearance Style 25.13. The prose calls 5B best in five metrics, conflicting with four visible wins.
Go to primary source ↓e-humanPDF p. 10, Section 4.2.2 and Table 4
Against Kling (2024.7), 5B has sensory/instruction/physics/cover scores 0.722/0.495/0.667/0.712 and total 2.74; Kling has 0.638/0.367/0.561/0.668 and 2.17.
Go to primary source ↓e-finetunePDF p. 15, Appendix A, High-Quality Fine-Tuning and Figure 9
Final fine-tuning uses a 20% higher-quality subset, improving cleanliness with slight semantic degradation. RoPE extrapolation favors local details while interpolation favors global structure in the shown initial generations.
Go to primary source ↓e-configPDF p. 16, Appendix A, Tables 5–6
Four stages list 400k/220k/120k/10k steps and later 700k sequences; Table 6 instead gives maximum 82k. 2B/5B use 30/42 layers, sinusoidal/RoPE positions; BF16 training and minimum aesthetic value 4.5 are listed, but peak learning rate and total training compute are absent.
Go to primary source ↓e-runtimePDF p. 16, Appendix A, Tables 7–8 and captions
Table 7 reports H800 inference labeled bf with 50 steps: 5B at 480 × 720/6s takes 113s and 26GB. Table 8 reports one-forward-step times of 9.60s (3D) versus 4.17s (2D+1D) at 768 × 1360/5s.
Go to primary source ↓e-timestep-lossPDF p. 17, Appendix B, Figure 10(a–d) and Table 9
Loss curves compare positions, attention, expert architecture and sampling. At 40k steps, explicit sampling lowers validation loss at the five listed timesteps from 0.222/0.130/0.119/0.133/0.161 to 0.216/0.126/0.116/0.129/0.157.
Go to primary source ↓e-i2vPDF p. 17, Appendix D
Image-to-video fine-tuning concatenates a VAE-encoded conditioning image with noisy inputs along channels and adds strong condition noise to reduce the first-frame versus real-image distribution gap.
Go to primary source ↓e-upsamplingPDF p. 23, Appendices F–G
LLM prompt expansion is used at inference; image-to-video can use a vision-language model. Caption training samples frames every two seconds and uses 50,000 examples to fine-tune the summary model.
Go to primary source ↓e-examplesPDF pp. 19–22, Figures 11–14; pp. 24–25, Appendix H and Figure 15
Selected text-/image-to-video frame grids and detailed caption comparisons are shown. Figure 11 and Figure 13 captions explicitly say displayed prompts are upsampled before model input.
Go to primary source ↓e-v2vPDF pp. 26–27, Appendix I and unnumbered source/caption/generated-video panels
CogVLM2-Caption first describes a source video; CogVideoX generates from that caption. The selected wizard, village and eye examples preserve semantic themes but have visibly different image details.
Go to primary source ↓e-human-protocolPDF pp. 28–29, Appendix J and Tables 10–13
100 prompts, aspect criteria and a 0–5 overall score are specified. Instruction failures cap total at 2; complete aggregation and rater count are not given. Physics criteria concern visible realism, lighting, interactions and fluid motion.
Go to primary source ↓e-filteringPDF pp. 29–30, Appendix K, Table 14 and Figure 16
Filtering is evaluated on a random 10% labeled test subset. Six classifier rows and example negative categories are shown; a complete evolving video-selection threshold schedule is not supplied.
Go to primary source ↓8.5 Primary sources
CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer ↗
PDF · 9,251 extracted words
Source fingerprint
699795d3cfdf7acef250e0e28043b001e31ae7ab101f35fd079370c43f6473f3