Towards Accurate Generative Models of Video: A New Metric & Challenges
1. Paper overview
In one sentence: FVD uses action-recognition features to compare whole-video distributions, gaining sensitivity to temporal defects while retaining dependence on representation, sampling and protocol. e03e04e07e09e11e15
| At a glance | What to know |
|---|---|
| Research problem | Source description Framewise PSNR and SSIM compare a prediction against a particular reference, which penalizes alternative plausible futures and does not directly evaluate a distribution of videos. The authors seek an evaluation that includes appearance, temporal coherence and diversity, alongside controllable scenarios that reveal failures hidden by visually complicated real-world data. e02e06 |
| Core mechanism | |
| A key reported result | Human agreement when PSNR cannot distinguish models: FVD: 81.0% Agreement with human judgment (%). BAIR; ten models selected to have equal PSNR near the best quartile. Three generated videos per selected model; pairwise ratings by two raters, with a third resolving disagreement. KVD: 73.8%; averaged FID: 52.0%; SSIM: 44.6%; agreement among raters: 84.4%. Supports discrimination among models tied by PSNR under this selection protocol. PSNR/SSIM selection uses the best frame-averaged score among 100 generations per context; FVD uses distribution statistics. e10e11 |
| Reading caution | Reader analysis The human study is on BAIR and selected model families. Table 1 does not show universal dominance: averaged FID exceeds FVD for spread-SSIM selections (62.7% versus 58.4%), and KVD narrowly exceeds it for spread-PSNR (63.8% versus 63.5%). Generalization to other video domains remains unestablished here. e10e11 |
Core contributions
Figure 2. SCV makes the intended temporal behavior explicit through four controlled scenarios. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each row from left to right, then move downward through the four tasks: MUtB, CMS, Brawl and RTwM. The first asks for a unit's movement and animation to remain coherent. The second links a unit reaching a mineral shard to that shard disappearing. The third adds many interacting units and attack effects. In the bottom row, the medivac must pick up units, visit beacons and unload the same cargo. These are example scenario videos, rather than generated baseline outputs. Appendix B explains that deterministic agents create the replays while scenario initialization controls randomness. e07e19e20
What it supports. The benchmark supplies observable temporal requirements beyond attractive frames. In particular, RTwM makes identity and memory concrete: the number and types of unloaded units should match those picked up. This motivates checking semantic consistency alongside the distribution-level FVD reported for generated videos.
Where the evidence stops. Selected still frames cannot establish every intervening event. The main text describes the Terran army as left-sided, while Appendix B.1 randomizes the two armies' sides; the figure should not be read as defining a fixed spatial rule.
2. Motivation
2.1 The problem and the proposed response
Framewise PSNR and SSIM compare a prediction against a particular reference, which penalizes alternative plausible futures and does not directly evaluate a distribution of videos. The authors seek an evaluation that includes appearance, temporal coherence and diversity, alongside controllable scenarios that reveal failures hidden by visually complicated real-world data. e02e06
2.2 What this reading follows
A video can contain plausible frames while telling an impossible story: objects vanish too early, motion changes abruptly, or passengers fail to reappear after transport. This paper tackles both how to score generated videos and how to expose such failures. Fréchet Video Distance embeds clips with a pretrained I3D network and compares Gaussian feature statistics. StarCraft 2 Videos supplies controlled scenarios with progressively harder interactions and memory requirements. The figures below connect the benchmark's intended behavior to corruption tests, human preferences and baseline results. Read the score as an empirical distribution comparison: the paper also shows why sample count and task-specific inspection remain necessary. e03e04e07e09e11e15
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Evaluation metrics |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
3.1 Evidence-based assessment
Supports the recorded classification
The recorded metrics classification is supported: the contribution evaluates video distributions and introduces a benchmark. Pretrained I3D action recognition supplies features; deterministic agents collect SCV videos. Neither constitutes a proposed action-generating world model, so architecture, prediction-paradigm and quadrant labels remain not applicable. e03e04e19
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Replace a frame match with a distribution comparison
The key change is the unit of evaluation. PSNR and SSIM compare a predicted frame with a particular reference, then summarize those comparisons across time. FVD instead sends whole clips through an action-recognition network. Each clip contributes one feature vector to either the real or generated collection. The Gaussian fit compresses each collection into its mean and covariance, and Equation (2) compares those summaries. The network supplies the temporal representation; the final distance is computed from statistics, without training a new predictor during evaluation. This also explains unconditional use: real samples are required, but a one-to-one reference future for every generated clip is unnecessary. For conditional videos, including the context lets the embedding see transitions into the generated continuation. e03e04e06
5.2 Use two kinds of validation before trusting the score
The paper first asks whether the metric notices controlled damage, then whether its ranking matches human preferences. These questions are complementary. Figure 3 tests reactions to spatial noise and altered temporal order; Figure 6 asks which feature-and-distance combination tracks the chosen intensity sequences. Table 1 then tests generated BAIR videos using models selected to be equal or spread under one metric. A strong score in an equal-PSNR column means FVD can detect quality differences that the selection metric missed. Figure 4 adds a separate statistical requirement: compare equal sample counts. Reader interpretation: corruption sensitivity, human agreement and estimator repeatability each support a different part of the evaluation argument; none alone certifies general semantic correctness. e08e09e10e11e17e18
5.3 Turn video quality into explicit temporal questions
SCV complements the scalar metric by making intended events easier to describe. A collecting unit should cause the nearby shard to disappear, a battle should preserve distinct interacting entities, and a medivac should unload the same cargo it picked up. These requirements arise from the scenario rules, while the videos themselves come from deterministic agents acting in randomized environments. The baseline models are trained to predict frames; they are not being evaluated as agents executing those actions. Reader interpretation: a useful evaluation pairs FVD with checks of the particular temporal rule at stake. The RTwM results illustrate the distinction: Table 2 ranks the models numerically, but the authors still describe the scenario as unsolved and leave the cause of failure open. e07e13e14e15e19
5.4 Training and inference
During training
FVD uses already pretrained action-recognition features; this paper does not train a new FVD-specific generator or control policy. Separately, CDNA, SV2P, SVP-FP and SAVP are trained as evaluation subjects using Tensor2Tensor implementations. e04e10e21
Baselines receive 300,000 updates. The learning-rate grid is 10^{-3}, 10^{-4}, 10^{-5}; VAE models vary the reconstruction/KL tradeoff, SV2P uses the cited annealing schedule, and SAVP additionally tunes GAN and GAN-VAE loss weights. The printed KL-weight grid is ambiguous, as recorded below. e21
During inference
Evaluation embeds completed videos and compares aggregate statistics; it does not run feedback control. BAIR, MUtB, CMS and Brawl use two context plus fourteen predicted frames; KTH uses ten plus ten. FVD evaluation uses 256 validation samples for BAIR and 1,024 for KTH/SCV. SCV clips begin at frame zero; BAIR/KTH use consecutive subsequences. RTwM's horizon is inconsistent between body and appendix. e06e13e19
5.5 Implementation flow
- Embed entire clips
Apply a pretrained Inflated 3D Convnet (I3D) to each video. Candidate representations are final logits and last-pooling outputs from RGB Kinetics-400 or Kinetics-600 models. Subsequent experiments select Kinetics-400 logits following the diagnostic comparison. e04e18
- Estimate two feature distributions
Fit a multivariate Gaussian to each set of embeddings, estimating its mean and covariance. Compute the Gaussian Fréchet expression below. The Gaussian approximation occurs in learned feature space; it is not a claim that raw videos are Gaussian. e03
- Separate alternative metrics
Kernel Video Distance (KVD) instead uses a polynomial-kernel MMD estimator on I3D features. The image baseline averages Inception frame features, or frame-feature differences, before applying the same Gaussian distance. These alternatives test the distance and representation choices. e05e08
- Construct controlled video challenges
Deterministic SC2LE agents create replays with scenario-controlled randomness, then render the same states at different resolutions. Right-click cues are visible in the videos. Each scenario supplies 10,000 training, 2,000 validation and 2,000 test videos. e19
6. Experiments & results
Fréchet Video Distance (FVD) compares distributions of real and generated videos using pretrained action-recognition features and Gaussian statistics. The paper validates sensitivity to temporal corruption and agreement with human preferences, then introduces StarCraft 2 Videos (SCV) to expose failures in motion, interaction and memory. Its central tradeoff is a useful distribution-level score whose interpretation depends on the embedding, sample count and evaluation protocol.
This metric-and-benchmark paper contains no FVD network-architecture schematic or newly proposed generator architecture: Section 2 defines the method in prose and equations. Figure 2 therefore supplies the original benchmark-method visual, with the FVD information flow explained in the walkthrough. Figures 3, 4 and 6 supply genuine metric diagnostics rather than generator-component ablations. The paper reports FVD and subjective video-quality evaluations, but no executed-action success table. e03e04e07e08e09e11e14e18
6.1 Read the original evidence
Table 1. FVD often agrees better with humans, with visible exceptions across selection protocols. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Columns describe how ten models were selected; rows identify the metric subsequently compared with human judgments. The 'eq.' columns hold one selection metric nearly constant near its best quartile. The 'spr.' columns spread models across that metric's range. Thus the 81.0% entry in the FVD row and eq. PSNR column asks whether FVD separates models that PSNR treats as tied. It does not report accuracy over an unrestricted model population. The bottom row measures agreement between the first two human raters. N/A entries mark evaluation by the same metric that was deliberately held equal. e10e11
What it supports. Within the equal-PSNR selection, FVD agrees with humans 81.0% of the time, versus KVD's 73.8% and averaged FID's 52.0%. In the spread-FVD selection, FVD reaches 71.9%. These support the metric's usefulness under the study's BAIR protocols, rather than an unconditional ordering of evaluation methods.
Where the evidence stops. The table qualifies its caption's broad superiority claim: averaged FID wins spread-SSIM, and KVD narrowly wins spread-PSNR. Human agreement concerns subjective video quality. The study does not directly measure action success or establish a universal perceptual threshold.
Table 2. Model rankings vary by task and resolution, while low relative FVD does not certify scenario success. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read down a fixed dataset-and-resolution column; lower FVD is better. The retained caption is essential because the paired SCV columns have no individual resolution labels: left is 64 × 64 and right is 128 × 128. SAVP leads BAIR at 116.4, whereas CDNA leads MUtB at 128 × 128 with 51.4. Keep these comparisons within their protocols: BAIR uses 256 evaluation samples, and KTH/SCV use 1,024. Appendix D's generated examples provide a separate check on what the scores mean visually, including disappearing units and blurry multi-unit scenes. e13e14e15e19
What it supports. The table identifies strong baselines but does not imply that the underlying scenarios are solved. For RTwM at 128 × 128, SV2P has the lowest score, 1026.1, compared with SAVP's 1055.4. The accompanying discussion nevertheless reports failure across the models on the transport-and-memory scenario.
Where the evidence stops. Cross-dataset scores differ in content, clip length and sometimes sample count; treat them as separate evaluations. Table 2 supplies no uncertainty. RTwM's two-context-plus-32-output wording conflicts with Appendix B's 32-frame total and remains unresolved.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Human agreement when PSNR cannot distinguish models BAIR; ten models selected to have equal PSNR near the best quartile. Three generated videos per selected model; pairwise ratings by two raters, with a third resolving disagreement. | FVD: 81.0% Agreement with human judgment (%) | KVD: 73.8%; averaged FID: 52.0%; SSIM: 44.6%; agreement among raters: 84.4%. Supports discrimination among models tied by PSNR under this selection protocol. PSNR/SSIM selection uses the best frame-averaged score among 100 generations per context; FVD uses distribution statistics. e10e11 |
| BAIR conditional video prediction Two context frames plus fourteen outputs; 256 validation samples for FVD. | SAVP: 116.4 FVD (lower is better) | SV2P: 262.5; CDNA: 296.5; SVP-FP: 315.5. SAVP leads this benchmark column. These are reported point estimates without table-level uncertainty or action-execution evaluation. e13e14 |
| MUtB conditional video prediction at 128 × 128 SCV-MUtB; two context plus fourteen output frames; 1,024 evaluation samples. | CDNA: 51.4 FVD (lower is better) | SVP-FP: 121.3; SAVP: 204.4; SV2P: 710.5. The ranking differs from BAIR, illustrating why a single benchmark winner cannot be assumed across scenarios. e13e14 |
| RTwM conditional video prediction at 128 × 128 SCV-RTwM; 1,024 evaluation samples; two context frames, with unresolved total/output-length wording. | SV2P: 1026.1 FVD (lower is better) | SAVP: 1055.4; CDNA: 1295.4; SVP-FP: 2031.4. The numerical winner is not a successful memory model: the authors report that all evaluated models fail RTwM. No scenario-success rate is supplied. e13e14e15e19 |
| Perceptual resolution of FVD BAIR comparisons around base scores 200 and 400; 1,800 video comparisons. | Authors interpret differences of about 50 points or more as typically perceptible. Human agreement as FVD difference increases | Smaller differences are described as near chance; Figure 5 plots standard errors and counts similar-quality judgments as disagreement. A study-specific calibration, not a universal significance threshold. The plotted base-200 curve includes a point below 0.5 despite the prose's 'never worse' qualification. e12 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 3. Temporal corruptions can change FVD even when the original frames remain recognizable. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the top row's black rectangle, blur, Gaussian noise and salt-and-pepper tests, then compare the bottom row's local swaps, global swaps, interleaving and switching. Blue, orange and green denote the three datasets listed in the legend. Each panel has its own vertical scale: local-swap values cannot be compared visually with Gaussian-noise bar heights without reading the axes. The representation is I3D Kinetics-400 logits. Appendix A defines the intensity settings separately for each corruption; for example, swap intensity increases the number of swaps, whereas switching intensity changes the frame at which the video changes. e04e08e17
What it supports. FVD responds to temporal alterations as well as static image damage, supporting the rationale for a video embedding. The plots also show that response is not uniformly monotonic: interleaving has a pronounced peak at its second intensity, and some static-noise curves flatten or decline.
Where the evidence stops. Intensity is not a shared perceptual unit across these panels. Switching uses the number of frames until the switch, not the number of switches. This diagnostic measures response to constructed corruption, not human preference or guaranteed physical consistency.
Figure 4. Even real-versus-real FVD depends strongly on the number of sampled videos. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow the horizontal axis as sample count doubles at each tick, from 2^5 to 2^12. Both sides of the comparison are non-overlapping subsets drawn from the same BAIR distribution, so the changing score is not evidence that a generator improved. It reflects the estimation of feature means and covariances from finite samples. The small error bars are standard errors across fifty trials, as specified in the caption. Read the curve's downward trend and its error bars separately: the first exposes systematic dependence on sample count; the second describes repeatability of the estimate at a fixed count. e03e09
What it supports. The curve demonstrates why a lower reported FVD is insufficient without its sample protocol. Two estimates of the same underlying distribution can be well above zero with small subsets. Matching sample counts is therefore an explicit requirement for comparisons in this paper.
Where the evidence stops. Small standard errors do not remove finite-sample bias. These are BAIR subset comparisons, not uncertainty estimates for every generated-model score in Table 2. The figure does not establish a universal sample count that is sufficient across datasets.
Figure 6. The metric's embedding and distance choices matter differently for different corruptions. Original paper, p. 11 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Each horizontal group is a corruption type, and each colored bar is one metric configuration from the legend. The blue shades are FVD with Kinetics-400 or Kinetics-600 logits or pooling features. Warm shades are the corresponding KVD choices; gray bars are the averaged-frame and averaged-frame-difference FID baselines. The vertical axis is Spearman rank correlation with the intensity sequence, so higher bars indicate better tracking of its ordering rather than lower video distance. Compare configurations within each group. The paper selects Kinetics-400 logits for subsequent FVD experiments, describing it as among the strongest configurations across these tests. e04e08e17e18
What it supports. The diagnostic supports selecting a temporal representation, while showing that no single configuration tops every corruption group. Interleaving is especially instructive: its correlations are weaker for the selected FVD than for several static corruptions, consistent with the nonmonotonic interleaving response visible in Figure 3.
Where the evidence stops. These correlations measure ordering of synthetic noise intensities, not agreement with human raters. Appendix A uses 90% of available train and test samples for this diagnostic. That pooled corruption study should not be confused with held-out generator evaluation.
7. Analysis & limitations
7.1 What the evidence leaves open
The human study is on BAIR and selected model families. Table 1 does not show universal dominance: averaged FID exceeds FVD for spread-SSIM selections (62.7% versus 58.4%), and KVD narrowly exceeds it for spread-PSNR (63.8% versus 63.5%). Generalization to other video domains remains unestablished here. e10e11
FVD depends on the pretrained representation and a Gaussian approximation. It measures video distributions, not whether each conditional prediction follows its own context. Concatenating context can expose switching, but does not establish a guarantee of conditional correctness or control competence. e03e04e06e08
Finite-sample FVD is positively biased even for two subsets of the same BAIR distribution. Figure 4 reports small standard errors across 50 trials at fixed sample size; that repeatability does not remove bias. Compare scores using matched sample counts. e09
SCV sacrifices real-world visual fidelity for controllable interactions. The reported CMS disappearance errors, Brawl blurring and RTwM failures motivate semantic diagnostics, but the paper leaves RTwM's failure cause—longer horizon versus scenario complexity—unresolved. e07e15
7.2 Questions for discussion
- How much can FVD improve while SCV entity identity, collection order or cargo consistency remains wrong?
- Would the approximately 50-point perceptual separation persist after changing the dataset, sample count or I3D representation?
8. Reproducibility audit
8.1 Requirements and known gaps
Use the specified I3D feature choice, matched clip construction and sample counts, and the SCV split/rendering protocol. Appendix B skips the first two rendered frames and records every sixth frame for MUtB, fourth for CMS/Brawl and eighth for RTwM. The introduction reports over 100 GPU-years overall. Code and pretrained-weight links are claims of availability, not verification. Exact software/checkpoint versions, batch size and GPU model/count are not specified in the supplied implementation description. e04e13e19e21e22e02
Preserve unresolved source details: Appendix C prints β = (10^{-6}, 10^{-5}, 10^{-6}, 10^{-3}), repeating 10^{-6}; do not silently replace it. Section 4.4 says two context frames plus up to 32 additional RTwM frames, whereas Appendix B says exactly 32 frames total. The KVD formula also prints j ≠ j under its final sum. These require clarification before an exact implementation claim. e05e13e19e21
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Test temporal sensitivity while controlling sample count
Reader-proposed experiment, not performed: use fixed-length held-out BAIR clips and the specified Kinetics-400-logit embedding. Compare intact clips with versions subjected to 4, 8, 12, 16, 20 and 24 global frame swaps, alongside the averaged-frame Inception baseline. Keep the clip collection identical within each comparison; repeat at matched sample counts of 256 and 1,024, with independent real-versus-real draws as an estimation control. Swapping should alter temporal order without changing the multiset of frames. Test whether FVD consistently separates these clips beyond real-versus-real variation while averaged-frame FID remains insensitive. Failure of that separation would challenge temporal sensitivity under the tested protocol. e04e08e09e17
Check 2: Separate medivac memory errors from long-horizon degradation
Reader-proposed experiment, not performed: after resolving the conflicting RTwM length descriptions, train one fixed baseline under equal compute on two conditions. Preserve the standard pickup–transport–unload task in one; in the other, hold the cargo identity and count constant across examples while preserving comparable transport durations and rendering settings. Use the same splits, horizons and FVD sample count. Annotate whether unloaded unit types/counts match the pickup phase, alongside FVD and visible-object persistence. If the controlled condition improves cargo consistency substantially at comparable horizons, memory demand is implicated; equally severe degradation in both conditions would keep horizon or general visual modeling as plausible explanations. e07e13e15e19e20e21
8.3 Reading coverage
Visual audit: All 17 supplied PDF pages were rendered and visually inspected, including the title/version/affiliations, Equations (1)–(2), the printed KVD formula, Figures 1–10, Tables 1–4, the complete evaluation protocol, Appendix B data generation and Appendix C hyperparameters. All six final original crops were separately viewed; axes, legends, table rows and the resolution-defining Table 2 caption remain readable. Figure 5's below-chance point, the RTwM length inconsistency, duplicated β entry and KVD summation typo are preserved as source qualifications. The inspected static appendix examples do not substitute for viewing full moving clips. No separate supplements or linked implementation artifacts were inspected.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract
- 1. Introduction
- 2. Fréchet Video Distance
- 3. Starcraft 2 Videos
- 4. Experiments: 4.1 Noise Study; 4.2 Effect of Sample Size on FVD; 4.3 Human Evaluation; 4.3.1 Resolution of FVD; 4.4 Baseline Results on SC2 Benchmark Data Sets; 4.5 Correlation of FVD with SSIM and PSNR
- 5. Conclusion
- References
- A. Noise Study
- B. SCV Data Generation; B.1. Scenario Parameters
- C. Benchmark Hyperparameters
- D. Examples of Video Models on SCV
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- The extraction limitation was addressed by visually inspecting every page of the supplied 17-page PDF, including all figures and tables. Separate supplements, linked code, pretrained weights and dataset files were not inspected; no experiments were reproduced.
- Version scope: the supplied title page identifies arXiv:1812.01717v2 [cs.CV], 27 March 2019. The catalog records the original 3 December 2018 submission and 2018 publication year. Title and all six authors agree, with Raphaël rendered without its accent in the catalog. The original v1 was not supplied, so revision contents were not compared.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title/author block, equal-contribution footnote and arXiv margin
The title matches the manifest. The six authors are Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphaël Marinier, Marcin Michalski and Sylvain Gelly. Affiliations are Johannes Kepler University; IDSIA, SUPSI, USI; and Google Brain. The margin identifies v2, 27 March 2019.
Go to primary source ↓e02PDF pp. 1–2, Abstract and Section 1 contributions
Motivates temporal coherence, appearance and diversity evaluation and introduces FVD plus SCV. The final contribution bullet reports over 100 GPU-years of experiments.
Go to primary source ↓e03PDF p. 2, Section 2, Equations (1)–(2) and following paragraph
Defines the Gaussian Fréchet expression using means and covariances and motivates fitting distributions in learned feature space.
Go to primary source ↓e04PDF p. 3, Section 2, I3D paragraphs; p. 6, Section 4.1, embedding choice
Compares RGB Kinetics-400/600 I3D logits and pooling features; selects Kinetics-400 logits for subsequent FVD experiments.
Go to primary source ↓e05PDF p. 3, Section 2, MMD formula and polynomial-kernel paragraph
KVD uses MMD with kernel k(a,b) = (a^T b + 1)^3. The printed last summation condition is j ≠ j, an unresolved typographical defect.
Go to primary source ↓e06PDF p. 3, Section 2, comparison with PSNR/SSIM
FVD evaluates distributions and can include context plus generated frames to detect switching; it supports unconditional evaluation without paired target sequences.
Go to primary source ↓e07PDF pp. 4–5, Figure 2 and Section 3 scenario descriptions
Defines MUtB motion, CMS collection, Brawl interactions and RTwM pickup/transport/unloading consistency; discusses the visual-fidelity tradeoff.
Go to primary source ↓e08PDF p. 5, Figure 3 and Section 4.1; p. 6, Section 4.1
Tests four static and four temporal corruptions on BAIR, Kinetics and HMDB51, comparing I3D variants and averaged image-feature baselines.
Go to primary source ↓e09PDF p. 6, Section 4.2 and Figure 4
Non-overlapping BAIR subsets produce positive FVD at finite sample sizes; the curve spans 2^5–2^12 samples and error bars are standard errors over 50 trials.
Go to primary source ↓e10PDF pp. 6–7, Section 4.3, model selection and Set-up
BAIR study uses four model families, two context plus fourteen output frames, 256 unseen reference videos, best-of-100 PSNR/SSIM, ten-model equal/spread selections and three displayed videos per selected model.
Go to primary source ↓e11PDF p. 7, Table 1, FVD/KVD/Avg. FID/SSIM and Among raters rows; eq. PSNR, spr. FVD, spr. SSIM and spr. PSNR columns
Equal-PSNR agreement is 81.0% FVD, 73.8% KVD, 52.0% averaged FID, 44.6% SSIM and 84.4% among raters. Spread-FVD agreement is 71.9% for FVD. Spread-SSIM favors averaged FID 62.7% over FVD 58.4%; spread-PSNR favors KVD 63.8% over FVD 63.5%.
Go to primary source ↓e12PDF p. 7, Section 4.3.1; p. 8, Figure 5 and caption
Uses base FVD 200/400 and 1,800 comparisons to interpret perceptual separations near 50 points. Ties count against agreement; error bars are standard errors. The base-200 curve has a sub-0.5 point despite the textual 'never worse' statement.
Go to primary source ↓e13PDF p. 8, Section 4.4, evaluation protocol paragraph
Specifies context/output horizons, BAIR's 256 and KTH/SCV's 1,024 evaluation samples, clip selection and the two-context-plus-up-to-32-output RTwM wording.
Go to primary source ↓e14PDF p. 8, Table 2, all model rows; BAIR, KTH and paired SCV-resolution columns
Reports FVD for CDNA, SV2P, SVP-FP and SAVP. SCV left/right entries are 64 × 64/128 × 128. SAVP has BAIR 116.4 and KTH 78.0; CDNA has MUtB-128 51.4; SV2P has RTwM-128 1026.1. The table supplies no uncertainty.
Go to primary source ↓e15PDF p. 8, Section 4.4 qualitative discussion; pp. 14–17, Appendix D, Figures 7–10
Discusses disappearing units, incorrect CMS collection sequences, blurry Brawl groups and RTwM failure. Appendix images order rows as reference, CDNA, SV2P, SVP-FP, SAVP.
Go to primary source ↓e16PDF p. 8, Section 4.5
Reports weaker correlation of FVD with PSNR than with SSIM, over the collection of model configurations used for the final correlation experiment.
Go to primary source ↓e17PDF p. 11, Appendix A and Table 3
Uses 90% of available train/test samples for corruption diagnostics. Maps intensities to parameters, including 4–24 local/global swaps, 2–6 interleaved sequences and switching after 1–5 frames.
Go to primary source ↓e18PDF p. 11, Figure 6, legend and eight noise-type groups
Spearman correlations compare four FVD and four KVD embeddings plus two FID baselines. Kinetics-400 logits is competitive but not highest in every corruption group.
Go to primary source ↓e19PDF p. 12, Appendix B and Table 4
Gives 10,000/2,000/2,000 training/validation/test videos per scenario, deterministic agents, visual click cues, replay rendering, sampling strides, initialization-frame skipping and exactly 32 RTwM frames.
Go to primary source ↓e20PDF pp. 12–13, Appendix B.1, scenario parameter paragraphs
Default CMS uses two marines and 20 shards; Brawl uses nine units per army with randomized sides; RTwM has one to four units and two beacons.
Go to primary source ↓e21PDF p. 13, Appendix C, complete paragraph
States Tensor2Tensor implementations, default parameters, learning-rate grid, the duplicated β-grid entry, SV2P annealing, SAVP loss tuning between 10^{-6} and 10^{-3}, and 300,000 updates.
Go to primary source ↓e22PDF p. 2, contribution footnotes; p. 3, footnote 4
Provides links described as FVD code, SCV data and pretrained I3D weights; these links do not establish inspected or reproduced implementations.
Go to primary source ↓8.5 Primary sources
Towards Accurate Generative Models of Video: A New Metric & Challenges ↗
PDF · 9,063 extracted words
Source fingerprint
6802a958cb841e1de093969b2814fc3e0c79978152d279cd78af5ff0379edadd