Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets
1. Paper overview
In one sentence: Curated video pretraining gives a latent diffusion backbone a transferable motion representation, but its strongest evidence remains short-video preference and image-based multi-view evaluation. e02e07e09e13e15e16
| At a glance | What to know |
|---|---|
| Research problem | Source description Video diffusion research had emphasized layer design while leaving data selection and training stages less systematically examined. The paper holds a baseline architecture and training recipe fixed within its curation studies, asking whether image initialization, curated video pretraining and subsequent high-quality finetuning each improve the resulting generator. e02e07 |
| Core mechanism | |
| A key reported result | Curation benefits after high-quality finetuning: At 10k steps: curated +127, uncurated +89. At 50k: curated +103, uncurated +70. Human-preference Elo improvement relative to image initialization. Identical eight-frame models initialized from image weights or video pretraining on curated/uncurated 50M clips; common 250K-clip Stage III. Image initialization is zero at each checkpoint; curated initialization remains highest. Figure 4(e) supports persistence of a pretraining advantage. These are relative Elo points, not preference percentages or an absolute cross-checkpoint performance scale. e07e17 |
| Reading caution | Source description The authors report expensive joint-frame generation, slow sampling, high VRAM requirements, limited suitability for long videos and occasional insufficient motion. They also identify misuse and bias concerns requiring investigation. e20 |
Core contributions
Figure 11. Detect gradual scene transitions before treating adjacent frames as a training sequence. Original paper, p. 16 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each row from left to right as consecutive source frames, then compare the two detection columns. Both methods detect the abrupt changes in the first two rows. In the lower rows, one scene fades into another; only the cascaded detector receives a checkmark. Appendix C explains the mechanism as three detectors operating at different frame rates and thresholds, followed by clipping to nearby keyframes that do not cross a detected boundary. This is a visual example of the data-processing method, not a neural-network architecture diagram. It shows why a generative model needs cleaner temporal boundaries than raw video metadata alone provides. e04e03
What it supports. The illustrated cascade catches gradual transitions missed by the displayed single-rate baseline. Together with the broader cut-count diagnostic in Figure 2, the example motivates cleaning temporal discontinuities before video pretraining, so those transitions are less likely to become part of the distribution the generator learns.
Where the evidence stops. The four rows are selected demonstrations, not a measured cut-detection recall benchmark. The paper does not supply the three precise frame-rate/threshold pairs, and these checkmarks do not establish performance on every kind of transition.
2. Motivation
2.1 The problem and the proposed response
Video diffusion research had emphasized layer design while leaving data selection and training stages less systematically examined. The paper holds a baseline architecture and training recipe fixed within its curation studies, asking whether image initialization, curated video pretraining and subsequent high-quality finetuning each improve the resulting generator. e02e07
2.2 What this reading follows
Stable Video Diffusion is easiest to understand as a study of what a video generator should see before its final finetuning. An image model supplies spatial representations; temporal layers learn from a large collection of processed video clips; a smaller, higher-quality set then refines the output. The crucial experiment asks whether the earlier data still matters after that last stage. It does in the reported preference study. This edition follows the data pipeline, the persistence of that advantage, and transfer to novel object views. The model predicts visual sequences. Camera-motion conditioning and realistic-looking videos should be interpreted within that generative scope. e02e07e09e13e15e16
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Foundational work |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
This table preserves the labels recorded at reading time. The current major category is Components of WAMs. View the current classification.
3.1 Evidence-based assessment
Supports the recorded classification
The recorded foundational video-generation-backbone category is supported. A temporal UNet jointly predicts visual frames and transfers to novel views; it does not predict or extract executable actions. Camera conditioning controls generated imagery. Consequently the catalog’s action-model architecture, prediction-paradigm and quadrant fields remain Not applicable; joint video generation alone does not establish a One Model world-action system. e02e11e13e15
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Why data cleaning must be tested after finetuning
A cut or fade creates a temporal transition that is valid in an edited video but unhelpful when learning how a single scene evolves. The cascade in Figure 11 addresses that problem before training; flow, aesthetics, caption alignment and text-area filters then change which clips remain. My interpretation is that the paper’s decisive design choice is to evaluate these earlier choices after a common final finetuning stage. Otherwise, a cleaner pretraining set might merely give a temporary advantage that high-quality data later removes. Figure 4(e) tests that possibility using the same final dataset and training duration. The curated initialization remains ahead. This supports investing in pretraining curation, while the independent filter sweeps leave interactions between selection criteria unresolved. e04e05e06e07
Table 1. Filtering reduces the collection substantially while preserving a large video pretraining set. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Compare each unfiltered column with its filtered partner before looking across unrelated datasets. LVD has 577M clips in this table, while LVD-F has 152M; the smaller experimental subset contracts from 9.8M to 2.3M. Clip duration is measured in seconds, whereas total duration is measured in years. The lower rows distinguish average frame counts, clips per source video, and availability of motion annotations. Those annotation checkmarks describe information available for curation, not supervision for a robot action model. Dashes in the WebVid and InternVid frame-count cells are source omissions and should stay missing rather than being reconstructed from other rows. e03e06e07
What it supports. The filtering recipe leaves a much smaller but still substantial training collection. For LVD, reported total duration changes from 212.09 to 50.64 years and mean clip duration from 11.58 to 10.53 seconds. These statistics establish scale; the preference experiments are needed to establish whether this distribution is better for generation.
Where the evidence stops. The surrounding prose rounds the original count to 580M, while the table reports 577M. Dataset columns differ in provenance and processing; their sizes alone cannot isolate why one trained model is preferred or establish comparable data coverage.
5.2 Follow the conditioning into a joint video denoiser
The scaled model starts from SD 2.1 spatial weights and adds temporal convolution and attention, then trains both kinds of layers. Denoising therefore operates on a sequence whose frames can exchange information. For text-to-video, the condition is a text embedding. For image-to-video, that route is replaced by a CLIP image embedding, while a second route concatenates the noise-augmented image encoding to every noisy frame. This separates the conditioning image from the unknown frames being generated. Training uses the clean-sample denoising objective; inference repeatedly combines conditional and unconditional predictions. The printed guidance implementation increases that combination’s strength across frame positions. It does not schedule increasing strength over denoising iterations. The paper relates this choice to the tension between consistency with the input image and oversaturated output. e02e08e09e11e12
5.3 Read multi-view transfer as adaptation evidence
An orbital view sequence resembles a video in which the camera moves while the object stays fixed. The authors exploit that relationship by adapting the image-to-video model to 21 Objaverse views and adding elevation conditioning. Compare video initialization with image initialization and random initialization in Figure 9 before invoking the stronger phrase ‘3D understanding.’ All three are evaluated against target frames from unseen GSO objects, and the video-initialized model obtains better reported metrics and earlier progress. My interpretation is that temporal video training supplies a useful initialization for jointly representing related views. The result does not isolate every cause of that advantage: the priors differ in training exposure and preparation. Nor does it measure explicit reconstructed geometry. The short reported adaptation time must also be separated from the cost of the inherited prior. e15e16e09
5.4 Training and inference
During training
Scaled training first adapts SD 2.1 to continuous EDM preconditioning for 31k image iterations: only the time embedding trains for the first 1k, followed by all layers. Video pretraining uses 14 frames, 150k iterations at 256×384 with batch 1536, then 100k at 320×576 with batch 768. AdamW uses learning rate 10⁻⁴; text dropout is 15%. e09
The noise distribution shifts upward as resolution and conditioning increase. High-resolution text-to-video training uses approximately 1M clips, 576×1024, 50k iterations, batch 768 and learning rate 3×10⁻⁵. Image-to-video first adapts at base resolution for 50k iterations, then each high-resolution variant trains for another 50k; its final log-noise mean and standard deviation are 1.0 and 1.6. e09e10e11
Keep the curation experiments separate: they use eight 256×256 frames, 40k steps, batch 256, the discrete SD 2.1 schedule with offset noise and v-parameterization, and 10% text dropout. Their Stage III uses 250K high-quality clips, eight 512×512 frames and 50k steps. These are not the scaled EDM configurations. e07e06
During inference
Sampling iteratively denoises a joint frame sequence, with classifier-free guidance toward its condition. Image-to-video guidance increases from low to high across frame position, not across denoising time, balancing conditioning fidelity against oversaturation. Frame-rate and motion-score conditioning permit motion variation; there is no interaction feedback or action-execution loop. e02e08e09e12
The appendix also probes separate spatial-content and temporal-motion prompts with a fixed seed. This is a qualitative inference observation rather than a quantified disentanglement result. Camera LoRAs instead receive additional training: rank 16 for 5k iterations. e13e21
5.5 Implementation flow
- Construct temporally usable clips
Cascade three cut detectors at different frame rates and thresholds, then snap boundaries to keyframes without crossing detected cuts. Annotate motion with Farnebäck flow at 2 FPS; high-quality finetuning uses more accurate RAFT flow. CoCa describes the middle image, VideoBLIP describes the clip, and an LLM combines their captions. CLIP scores, aesthetics and OCR provide further filtering signals. e04e05
- Calibrate the training distribution
Separate experiments remove 12.5%, 25% or 50% along each annotation axis. The selected recipe removes the lowest 25% for motion and aesthetics, lowest 50% for CLIP similarity, and highest 25% for text area. Appendix C also specifies a 7% text-area cutoff. Caption sampling favors CoCa with probabilities 0.5/0.25/0.25 for CoCa/VideoBLIP/LLM. e05e06
- Extend the spatial denoiser
Insert temporal convolutions and attention after corresponding spatial layers in SD 2.1. Video training updates spatial and temporal weights together. The scaled UNet adds 656M parameters, reaching 1,521M; it predicts clean samples from noisy latents under diffusion conditioning. Temporal mixing couples the jointly generated frames. e02e08e09
- Change conditioning for image animation
Replace text embeddings with the conditioning image’s CLIP embedding. Noise-augment the image, encode it with SD 2.1, and concatenate the encoder-distribution mean to every noisy frame input. The image is repeated along time without an image-to-video mask. Dedicated high-resolution variants generate 14 or 25 frames. e11
- Adapt the representation
Camera LoRAs modify temporal attention. Interpolation conditions on two endpoints plus masked intermediate positions and predicts three intervening frames. Objaverse adaptation converts the 25-frame image-to-video model to 21 jointly generated orbital views, adding a sine/cosine embedding of camera elevation to the UNet’s vector condition. e13e14e15
6. Experiments & results
Stable Video Diffusion turns an image diffusion backbone into a reusable video generator through curated video pretraining and high-quality finetuning. Its strongest methodological lesson is that pretraining data quality continues to matter after finetuning. Text-to-video, image-to-video and multi-view experiments test generative representations; they do not establish an action policy or closed-loop world model.
The supplied PDF describes the UNet architecture in prose rather than a complete network schematic; the original mechanism crop therefore illustrates the cut-detection method that prepares training sequences. Camera-LoRA, interpolation quality and temporal-prompting findings are primarily qualitative and cannot be assigned unreported numerical gains. Static PDF samples also cannot replace video playback. e02e04e09e13e14e21e22
6.1 Read the original evidence
Table 2. A strong reported distributional video score, with a literature-comparison boundary. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the downward arrow beside FVD first: lower is better, and the bold final row is SVD at 242.02. The closest listed score is PYOCO at 355.20. Appendix E.4 supplies the evaluation behind SVD’s row: generate 13,320 videos with the action-category frequency distribution of UCF-101, condition directly on category names, and use 16 frames at 25 FPS with guidance strength 7. Generated 320×576 frames are resized to 240×432 and center-cropped to 240×320 before the referenced I3D feature computation. These details belong to this benchmark; they are not the image-to-video preference-study settings. e18
What it supports. SVD records the lowest FVD among the listed methods. The result supports the usefulness of its base video representation for this zero-shot generation benchmark. It measures similarity between video distributions through the stated feature pipeline; it does not measure whether a generated action can be executed successfully.
Where the evidence stops. The table caption explicitly says baseline numbers come from literature. The paper does not show matched reruns for all methods or uncertainty intervals. Its ranking applies to this set of reported results, with baseline preprocessing and sampling differences unresolved.
Figure 9(b). Video initialization gives an early and sustained advantage during the reported multi-view adaptation. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. This crop isolates Figure 9(b). Begin with the legend: red denotes SVD-MV, blue SD2.1-MV, and green Scratch-MV, corresponding to video, image and random initialization. Follow training iterations along the horizontal axis. Solid curves use the left CLIP-score axis, while broken curves use the right PSNR axis, as specified in the original caption. Compare the same line style across colors; the two vertical scales do not permit direct comparisons between solid and broken heights. Section 4.5 evaluates corresponding generated and target frames on 50 unseen GSO objects after Objaverse training. The plot therefore tracks adaptation progress on that test set, rather than measuring the cost of acquiring the preceding priors. e15e16
What it supports. At the displayed checkpoints, the red curves remain above the corresponding blue and green curves for both metrics. This supports an early and sustained benefit from video initialization in the reported multi-view experiment. The exact endpoint table results remain in the base report; this panel provides complementary evidence about progress during finetuning.
Where the evidence stops. The adjacent Figure 9(a) table has a partly obscured CLIP-S header in the original PDF; this crop presents the intact plot. Its caption and legible table values support the retained numerical results. No uncertainty is shown, and the curves exclude prior-training cost and do not measure reconstructed geometry.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Curation benefits after high-quality finetuning Identical eight-frame models initialized from image weights or video pretraining on curated/uncurated 50M clips; common 250K-clip Stage III. | At 10k steps: curated +127, uncurated +89. At 50k: curated +103, uncurated +70. Human-preference Elo improvement relative to image initialization | Image initialization is zero at each checkpoint; curated initialization remains highest. Figure 4(e) supports persistence of a pretraining advantage. These are relative Elo points, not preference percentages or an absolute cross-checkpoint performance scale. e07e17 |
| Zero-shot text-to-video generation on UCF-101 13,320 generated videos matching the action-category distribution; raw category prompts, 16 frames at 25 FPS, guidance 7; 320×576 generation resized and center-cropped to 240×320 for I3D features. | 242.02 FVD ↓ | PYOCO 355.20; Make-A-Video 367.23; Video LDM 550.61. Baseline scores are imported from literature. SVD is best among Table 2’s listed scores. The table is not a controlled rerun under demonstrably identical baseline protocols; no uncertainty is reported. e18 |
| Single-image multi-view generation on GSO Finetune on 150K curated Objaverse objects; evaluate corresponding generated and ground-truth frames on 50 unseen GSO objects. | 0.14 / 16.83 / 0.89 LPIPS ↓ / PSNR ↑ / CLIP-S ↑ | SD2.1-MV: 0.18 / 15.06 / 0.83; Scratch-MV: 0.22 / 14.20 / 0.76; SyncDreamer: 0.18 / 15.29 / 0.88. Video initialization improves the reported multi-view metrics. The 12k-step, approximately 16-hour adaptation cost excludes video pretraining; it is not an end-to-end compute comparison. e15e16 |
| Image-to-video visual preference against Gen-2 and Pika 64 SDXL-generated conditioning images; SVD’s 25-frame model sampled for 28 frames, then interpolated. Pair-specific duration, resolution and watermark matching are described in Appendix E.3. | SVD is preferred in both Figure 6 comparisons; exact percentages are not tabulated. Pairwise human visual-quality preference | Gen-2 comparison uses 3.89-second clips; Pika comparison uses 3-second clips. This supports preference under the stated historical sampling settings, not prompt-following superiority or universal performance on real photographs. e19e17 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 4(a–e). Better pretraining data retains an advantage after all models see the same high-quality finetuning set. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Panels (a) and (b) compare the curated small subset against WebVid and InternVid. Panel (c) holds dataset size at 50M while changing curation; panel (d) compares larger and smaller curated collections. Their vertical axes are user-preference fractions. Switch units when reading panel (e): it plots Elo improvement relative to image initialization at two finetuning checkpoints. Green is that reference, orange is uncurated video pretraining, and blue is curated video pretraining. Section 3.4 and Appendix E.2.3 specify a shared 250K-clip finetuning set and eight-frame, higher-resolution training. The experiment asks whether common final training erases the initialization advantage. e07e17e09
What it supports. At 10k finetuning steps, curated and uncurated initialization gain 127 and 89 Elo points over image initialization. At 50k, the gains remain 103 and 70. Thus the curated model retains the larger relative advantage in this experiment, supporting the distinction between video pretraining and final high-quality finetuning.
Where the evidence stops. Each checkpoint uses its own image-initialized reference. The smaller relative Elo gaps at 50k do not mean absolute model quality fell. No uncertainty intervals are displayed, and these eight-frame discrete-schedule experiments differ from the scaled EDM training recipe.
Figure 17. The chosen curation recipe balances objectives instead of mechanically choosing the highest aggregate bar. Original paper, p. 23 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Treat each panel as its own experiment. In the four threshold panels, legend values 0.125, 0.25 and 0.5 are fractions of examples removed, not raw optical-flow, aesthetic or similarity score cutoffs. Bars separate prompt alignment, visual quality and their aggregate. The captioning panel instead compares caption sources and the option labeled Mixed-U. Appendix E.2.2 specifies the shared eight-frame, 40k-step setup and explains the selections. Inspect the motion panel especially carefully: the authors choose 25% removal even though unfiltered data ranks better on the aggregate. Their rationale prioritizes visual quality while retaining competitive prompt following. e06e05e17
What it supports. The final choices remove 25% for low motion, 25% for low aesthetics, 50% for low CLIP similarity and 25% for high text area. The caption study favors CoCa overall, motivating a final training mixture that samples CoCa half the time and each other caption source one quarter of the time.
Where the evidence stops. The sweeps change one annotation axis at a time. They do not measure every interaction in the combined filter. The final weighted caption mixture is specified in prose but is not separately plotted; exact Elo values and uncertainty should not be invented from bar heights.
7. Analysis & limitations
7.1 What the evidence leaves open
The authors report expensive joint-frame generation, slow sampling, high VRAM requirements, limited suitability for long videos and occasional insufficient motion. They also identify misuse and bias concerns requiring investigation. e20
Filter selection is a quality–alignment tradeoff: the chosen motion filter does not maximize aggregate preference. Independent single-axis sweeps do not establish a globally optimal joint filter. Figure 17 labels a Mixed-U caption option but does not separately report the final CoCa-weighted sampling recipe. e06
Multi-view evidence concerns image metrics on a small object test set and selected examples, not explicit geometric reconstruction or controllable physical dynamics. MVImgNet uses roughly 200K training and 900 test videos without pose conditioning; its shown examples do not supply a comparable quantitative table. e15e16e22
7.2 Questions for discussion
- Does the chosen motion threshold still help on unseen prompts when quality and alignment receive equal importance?
- How much of the multi-view gain comes from temporal video learning versus additional training exposure and changed noise preconditioning?
8. Reproducibility audit
8.1 Requirements and known gaps
Exact training needs the curated data and SD 2.1 initialization, not only the architecture. Multi-view adaptation specifies eight 80GB A100 GPUs, batch 16 and learning rate 10⁻⁵. The paper does not identify the full LVD membership, exact three cut-detector settings, the caption-combining LLM, or total scaled-training hardware/time. e03e04e05e09e15
Resolve interpolation ambiguities before implementation: Section 4.4 says high-resolution text-to-video initialization, whereas Appendix D.5 says the base model. D.5 prints latent height 52 and input shape (5,9,52,128) alongside 576×1024 training resolution without explaining that dimensional mapping. Preserve these statements rather than silently substituting a corrected shape. e14
A proposed minimal mechanism check would compare curated and equally sized uncurated pretraining under the same eight-frame recipe, followed by identical finetuning and held-out preference evaluation. Retain visual quality and prompt alignment separately to test whether any gain survives the final stage. e06e07e17
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Does the motion-filter choice survive a matched-data transfer test?
Reader-proposed check, not performed: build equal-sized subsets of one available video corpus, either randomly sampled or excluding the lowest motion-score quartile, while matching caption strategy and the remaining filters. Train the paper’s eight-frame 256×256 discrete-schedule recipe for 40k steps with identical seeds and batch sizes, then finetune both on the same high-quality set using its eight-frame 512×512 protocol. Evaluate held-out prompts with randomized pair ordering, reporting visual quality and prompt alignment separately before and after finetuning, with uncertainty across prompts and training seeds. If the filtered model’s quality advantage disappears after finetuning, or depends entirely on a large alignment loss, that would weaken the rationale for this particular motion threshold. e06e07e17
Check 2: Does framewise guidance outperform a fair constant-guidance control?
Reader-proposed check, not performed: hold the image-to-video checkpoint, input images, initial noise, denoising sampler, frame count and motion conditions fixed. Compare the Figure 15 linear guidance ramp against constant guidance at the ramp’s mean, minimum and maximum. Measure conditioning-image fidelity at the beginning, oversaturation over frame position, generated motion, and blinded visual preference. Include low-motion and high-motion inputs. The claim predicts that the ramp improves the consistency–oversaturation tradeoff while retaining useful motion. If a constant control matches or improves all these outcomes, or the apparent benefit comes only from generating less motion, the ramp’s claimed advantage is not established. e11e12e20
8.3 Reading coverage
Visual audit: The title and author block, method and training descriptions, diffusion equations, Figure 15’s printed guidance implementation, Tables 1–2 and Figures 1–25 were visually read on the declared pages. This includes architecture descriptions on pages 3 and 19–20, quantitative and evaluation protocols on pages 8 and 21–25, and the multi-view hardware specification on page 21. All six final original crops were individually inspected. Figure 9(b) is shown separately because the adjacent table’s CLIP-S header is partly obscured by the plot in the source PDF; the table’s numerical rows and metric definition remain readable on inspected page 8. The final plot and UCF-101 table were cropped from 600-DPI PDF renders. References and contents on pages 9–14 were read in the complete text pass. No external supplement, repository, model execution or video playback was inspected.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract
- 1. Introduction
- 2. Background
- 3. Curating Data for HQ Video Synthesis
- 3.1. Data Processing and Annotation
- 3.2. Stage I: Image Pretraining
- 3.3. Stage II: Curating a Video Pretraining Dataset
- 3.4. Stage III: High-Quality Finetuning
- 4. Training Video Models at Scale
- 4.1. Pretrained Base Model
- 4.2. High-Resolution Text-to-Video Model
- 4.3. High Resolution Image-to-Video Model
- 4.3.1 Camera Motion LoRA
- 4.4. Frame Interpolation
- 4.5. Multi-View Generation
- 5. Conclusion
- Acknowledgements
- References
- Appendix
- A. Broader Impact and Limitations
- B. Related Work
- C. Data Processing
- D. Model and Implementation Details
- D.1. Diffusion Models
- D.2. Base Model Training and Architecture
- D.3. High-Resolution Text-to-Video Model
- D.4. High-Resolution Image-to-Video Model
- D.4.1 Linearly Increasing Guidance
- D.4.2 Camera Motion LoRA
- D.5. Interpolation Model Details
- D.6. Multi-view generation
- E. Experiment Details
- E.1. Details on Human Preference Assessment
- E.1.1 Experimental Setup
- E.1.2 Elo Score Calculation
- E.2. Details on Experiments from Section 3
- E.2.1 Architectural Details
- E.2.2 Calibrating Filtering Thresholds
- E.2.3 Finetuning Experiments
- E.3. Human Eval vs SOTA
- E.4. UCF101 FVD
- E.5. Additional Samples
- E.5.1 Additional Text-to-Video Samples
- E.5.2 Additional Image-to-Video Samples
- E.5.3 Additional Camera Motion LoRA Samples
- E.5.4 Temporal Prompting via Temporal Cross-Attention Layers
- E.5.5 Additional Samples on Multi-View Synthesis
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- All nine supplied text chunks were read individually, including references and appendices. PDF pages 1–8 and 15–30 were visually inspected; reference and contents pages 9–14 were read as text.
- Identity/version: the inspected title page is arXiv:2311.15127v1 [cs.CV], 25 November 2023. Its title and all twelve authors match the catalog. Stability AI is the observed affiliation; the first three authors share equal-contribution marks. No different revision was supplied, so revision-to-revision changes cannot be assessed.
- External repositories, model weights, training datasets and separate supplements were not inspected. Figure 15’s printed implementation was read as paper content only. No paper code was executed and no experiments were reproduced.
- The visual pass covers the original static PDF figures, not playback of generated videos; temporal smoothness cannot be independently judged from these still-image sequences.
- Source layout defects remain unresolved: on PDF page 8, Figure 9(b) overlaps the end of Figure 9(a)’s CLIP-S header, although the metric definition and numerical rows are readable; page 19 contains an incomplete introductory cross-reference and empty numbered equation lines (9) and (14). No missing header content, cross-reference or equation is reconstructed.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title, author/affiliation block, equal-contribution footnote and arXiv margin
The exact observed title and twelve catalog authors appear with Stability AI; Blattmann, Dockhorn and Kulal have equal-contribution marks. The margin identifies arXiv:2311.15127v1 [cs.CV], 25 Nov 2023.
Go to primary source ↓e02PDF pp. 2–3, Sections 1–2 and Section 3 stage definitions
The paper studies data selection with fixed video baselines, distinguishes image pretraining, video pretraining and high-quality finetuning, and inserts temporal convolution/attention after spatial layers while training the full model. Outputs are generated visual sequences.
Go to primary source ↓e03PDF p. 4, Table 1 and Section 3.1 continuation; p. 3, Figure 2
Table 1 reports LVD/LVD-F sizes 577M/152M and LVD-10M/LVD-10M-F sizes 9.8M/2.3M. LVD durations are 11.58/10.53 seconds and total durations 212.09/50.64 years. Prose rounds LVD to 580M. Figure 2 reports 2.65 versus 11.09 clips per video and a peak of low-motion clips.
Go to primary source ↓e04PDF p. 16, Appendix C, Figure 11 and Cascaded Cut Detection; p. 17, Keyframe-Aware Clipping
The method cascades three PySceneDetect detectors with different frame rates and thresholds, then snaps cuts to non-crossing keyframes. Figure 11 shows sudden transitions detected by both variants and gradual transitions detected only by the cascade in the displayed examples; precise detector settings are not supplied.
Go to primary source ↓e05PDF pp. 17–18, Appendix C, Optical Flow, Synthetic Captioning, Caption similarities and Aesthetics, Text Detection; Figures 12–14
Farnebäck flow is computed at 2 FPS and reduced to a 16-pixel shortest side; RAFT uses 800×450 for the smaller finetuning set. CoCa, VideoBLIP and an unnamed lightweight LLM produce captions. CLIP and OCR annotate three frames; CRAFT text-area coverage above 7% is excluded.
Go to primary source ↓e06PDF p. 23, Appendix E.2.1–E.2.2 and Figure 17
Single-axis sweeps compare 0%, 12.5%, 25% and 50% removal. Eight-frame 256×256 models train 40k steps with batch 256 and are evaluated on 64 prompts. Chosen removal fractions are 25% motion, 25% aesthetics, 50% CLIP and 25% text. Motion filtering is selected despite lower aggregate ranking. Final caption mixture probabilities are 0.5/0.25/0.25; Figure 17 instead labels its plotted mixture Mixed-U.
Go to primary source ↓e07PDF p. 4, Figure 3; p. 5, Figure 4(a–e) and Sections 3.3–3.4; pp. 23–24, Appendix E.2.1 and E.2.3
Image initialization and curated datasets improve the studied preferences. Figure 4 covers dataset comparisons and 50M-scale curation; panel (e) labels relative Elo improvements 127/89 at 10k and 103/70 at 50k for curated/uncurated pretraining. Common finetuning uses 250K clips and eight 512×512 frames. The ablations use discrete v-parameterization with offset noise, AdamW at 10⁻⁴, batch 256, 10% conditioning dropout, and 50-step DDIM at guidance 12.
Go to primary source ↓e08PDF p. 18, Appendix D.1, Equations (1)–(3); p. 19, Equations (10)–(13)
The diffusion description defines iterative denoising, the weighted clean-sample reconstruction objective, conditioning c and Gaussian noise n, and preconditions the trainable network in the EDM framework.
Go to primary source ↓e09PDF p. 6, Section 4.1; pp. 19–20, Appendix D.2
SD 2.1 is adapted for 31k image iterations, freezing all but time embeddings for the first 1k. The UNet adds 656M parameters to reach 1521M. Video training uses 14 frames, 150k steps at 256×384/batch 1536, then 100k at 320×576/batch 768, AdamW learning rate 10⁻⁴ and 15% text dropout. Log-noise mean shifts from −1.2 to 0. Frame rate and motion score are conditions. Page 19 also contains an incomplete cross-reference and empty numbered equation lines (9) and (14).
Go to primary source ↓e10PDF p. 6, Section 4.2; p. 20, Appendix D.3
High-resolution text-to-video finetuning uses approximately 1M clips, 576×1024, 50k iterations, batch 768 and learning rate 3×10⁻⁵, with noise mean/std 0.5/1.4 and EMA decay 0.9999; checkpoints use visual inspection and human evaluation.
Go to primary source ↓e11PDF p. 7, Section 4.3; p. 20, Appendix D.4
Image conditioning replaces text with CLIP image embeddings and concatenates the noise-augmented encoded frame across time without masking. Conditioning log-noise is normal with mean −3 and standard deviation 0.5. Base and high-resolution adaptation each use 50k iterations; final variants generate 14 or 25 frames with noise mean/std 1.0/1.6.
Go to primary source ↓e12PDF p. 19, Equation (4); p. 20, Appendix D.4.1; p. 21, Figure 15, lines 25–35
Classifier-free guidance combines conditional and unconditional predictions. The printed implementation linearly spaces guidance strengths over the frame axis and applies unconditional + scale × (conditional − unconditional); the prose relates this to inconsistency at low guidance and oversaturation at high guidance.
Go to primary source ↓e13PDF p. 7, Section 4.3.1 and Figure 7; p. 20, Appendix D.4.2; p. 27, Figure 20
Camera-specific LoRAs are trained in temporal attention for horizontal, zooming and static camera classes. Appendix D specifies rank 16 and 5k iterations. The figures provide qualitative controlled-generation examples.
Go to primary source ↓e14PDF p. 7, Section 4.4; pp. 20–21, Appendix D.5
Interpolation conditions on endpoint encodings, three learned mask embeddings and a binary presence mask, with endpoint CLIP features for cross-attention. It predicts three middle frames and trains about 10k iterations. Section 4.4 names the high-resolution initializer; D.5 names the base initializer and prints latent height 52 and input (5,9,52,128) alongside 576×1024 training without resolving the discrepancy.
Go to primary source ↓e15PDF pp. 7–8, Section 4.5, Datasets/Metrics/Training; p. 21, Appendix D.6
Objaverse training uses 150K curated CC-licensed objects, 21-frame orbital renders and camera elevations −5° to 30°; evaluation uses 50 unseen GSO objects. MVImgNet uses approximately 200K/900 train/test videos without pose conditions. D.6 specifies 576×576 renders, elevation embeddings, 12k iterations, batch 16, learning rate 10⁻⁵ and eight 80GB A100 GPUs.
Go to primary source ↓e16PDF p. 8, Figure 9(a), all rows and columns; Figure 9(b); Section 4.5 Results
SVD-MV LPIPS/PSNR/CLIP-S are 0.14/16.83/0.89; SD2.1-MV 0.18/15.06/0.83; Scratch-MV 0.22/14.20/0.76; SyncDreamer 0.18/15.29/0.88; Zero123 0.18/14.87/0.87; Zero123XL 0.20/14.51/0.87. The curve compares CLIP-S and PSNR during finetuning; approximately 16 hours is the reported adaptation time. The end of the table’s CLIP-S header is obscured by the adjacent plot in the PDF; the Metrics paragraph identifies CLIP similarity, and the numeric cells remain legible. Figure 9(b) uses red/blue/green for SVD-MV/SD2.1-MV/Scratch-MV and solid/broken curves for left-axis CLIP-S/right-axis PSNR.
Go to primary source ↓e17PDF pp. 21–22, Appendix E.1–E.1.2 and Figure 16
Preference evaluation uses 64 text prompts, randomized prompt/model ordering, attention checks and on average three annotator votes per pair per task. Elo starts at 1000, uses K=1 and aggregates 1000 randomized vote-order calculations. Quality and prompt following are distinct evaluation axes.
Go to primary source ↓e18PDF p. 6, Table 2; p. 24, Appendix E.4; p. 25, footnote 3
Table 2 reports SVD FVD 242.02 and literature baselines including PYOCO 355.20, Make-A-Video 367.23 and Video LDM 550.61. SVD generates 13,320 category-conditioned videos, 16 frames at 25 FPS, guidance 7, with native 320×576 outputs resized to 240×432 and center-cropped to 240×320 for the referenced pretrained I3D extractor. Footnote 3 specifies rescale=True, resize=True and return_features=True.
Go to primary source ↓e19PDF p. 6, Figure 6; p. 24, Appendix E.3
SVD preference bars exceed the Gen-2 and Pika bars, without tabulated percentages. Evaluation uses 64 SDXL-generated images. The 25-frame SVD model is sampled for 28 frames and interpolated; Gen-2 is matched at 3.89 seconds, Pika at 3 seconds, with spatial and watermark adjustments. Historical baseline settings are specified.
Go to primary source ↓e20PDF p. 15, Appendix A, Broader Impact and Limitations
The authors discuss misinformation, harm and potential bias, and acknowledge expensive joint-keyframe generation, insufficient motion in some samples, slow diffusion sampling and high VRAM demands, especially for long videos.
Go to primary source ↓e21PDF p. 25, Appendix E.5.4; p. 28, Figure 21 and caption
The text-to-video model can be probed with separate spatial and temporal prompts. The displayed fixed-seed experiment varies temporal prompts among empty, panning, rotating and zooming; no quantitative disentanglement or control metric is supplied.
Go to primary source ↓e22PDF p. 8, Figure 10; pp. 25–30, Appendix E.5 and Figures 18–25
The appendix presents selected text-to-video, image-to-video, camera-control and multi-view still-image sequences, including GSO comparisons and a qualitative MVImgNet comparison. These panels are illustrative samples, not playback or additional quantitative benchmarks.
Go to primary source ↓8.5 Primary sources
Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets ↗
PDF · 17,441 extracted words
Source fingerprint
654ef597e183c0544cd753494cb442125bc42b17dd2477481b6799114482a92e