The Kinetics Human Action Video Dataset
1. Paper overview
In one sentence: Kinetics combines large-scale clip collection with human verification and model-assisted cleanup, exposing complementary appearance and motion signals while retaining label ambiguity and selection-bias questions. e02e03e04e07e08e09e14e16e18
| At a glance | What to know |
|---|---|
| Research problem | |
| Core mechanism | |
| A key reported result | Kinetics held-out action classification: Two-stream RGB+Flow: 61.0 / 81.3 Top-1 / top-5 accuracy (%). Original 400-class Kinetics; all models trained from scratch, validation used for tuning, held-out test evaluation. Final train/validation-union retraining is not explicitly resolved. RGB: 56.0 / 77.3; Flow: 49.5 / 71.9; ConvNet+LSTM: 57.0 / 79.0; 3D-ConvNet: 56.1 / 79.5. Fusion improves over RGB by 5.0 top-1 and 4.0 top-5 percentage points (reader subtraction). This supports complementary signals under this protocol; architecture, parameters and temporal interfaces differ. No repeated-run uncertainty is reported. e13e14 |
| Reading caution | Source description Multiple valid actions may share a clip but only one label is assigned. The authors favor top-5 accuracy; it mitigates some label ambiguity without making annotation exhaustive. e04 |
Core contributions
Table 1. Scale is measured in source videos as well as labeled clips. Original paper, p. 2 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the last two columns together. Total counts clips, while Videos counts the source videos from which those clips were extracted. Kinetics reports 306,245 in both columns because its design retains one clip per source video. UCF-101 instead lists 13,320 clips from 2,500 videos. Next read Actions as the label-space size, not the number of observed action instances. The Clips column mixes minimum counts with an average for ActivityNet-200, so those entries are not identical statistics. Section 2 separately specifies training, validation and test counts; this table describes the full collection. e03e08e19
What it supports. The table supports the paper’s scale argument: Kinetics broadens the class vocabulary to 400 and substantially increases the number of distinct source videos. Counting videos helps distinguish additional observations from repeated cuts of the same recording. It is a dataset-design argument, not a measurement of model generalization.
Where the evidence stops. The printed minimum of 400 conflicts with Appendix A, including taking a shower at 378. A distinct YouTube URL also does not prove unique underlying footage, which motivates the additional content-based cleanup.
2. Motivation
2.1 The problem and the proposed response
Existing action benchmarks offered too little independent video variation to train large networks reliably. Kinetics targets trimmed action classification: clips contain sound, but neither exhaustive action annotation nor temporal localization is its central task. e02e03e04
2.2 What this reading follows
Recognizing an action requires deciding what evidence matters: the object being handled, the movement, or the scene surrounding it. This original Kinetics paper creates a much larger collection of short human-action videos and compares familiar classifiers to expose that problem. Read the collection process before the accuracy table: candidate selection, worker judgments and duplicate removal determine what the benchmark contains. Then compare aggregate performance with classwise diagnostics. The combined RGB and flow model leads the reported Kinetics baselines, but relative motion gains do not guarantee high absolute accuracy. Non-exhaustive labels and preliminary bias analysis remain central evidence boundaries. e02e03e04e07e08e09e14e16e18
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Datasets |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
3.1 Evidence-based assessment
Supports the recorded classification
The recorded dataset category and action-annotation subcategories match the contribution. Architecture, prediction paradigm and quadrant are appropriately not applicable: the recognition baselines produce observed-action labels, with no joint future/action model, inverse-dynamics policy, or environment interaction loop. e02e04e11
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
5. Method in detail
5.1 Build a labeled sample, then question how it was selected
The collection pipeline begins before any worker sees a video. An action vocabulary drives title matching on YouTube, while image classifiers select likely temporal positions inside retrieved videos. Workers then answer a focused visual question about the proposed action. This separation matters: retrieval finds plausible candidates, and human judgments check whether those candidates contain the target. Three confirmations admit a clip to the next stage, but admission does not guarantee unique content. The authors therefore retain one clip per URL and perform feature-based duplicate removal within each class. Two-stream errors subsequently guide class revision and final manual inspection. Reading the stages together explains why the dataset is more than a list of search results, while also locating where classifier preferences could affect its contents. e06e07e08e10
Figure 2. Human review verifies a candidate action using visual evidence. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Begin with the question below the player: the worker is judging a specified action, here riding mule, rather than describing every activity visible in the clip. Match the response icons to the right-hand legend: acceptance, rejection, uncertainty, replay and invalid content have distinct functions. The muted-audio instruction is part of the collection protocol; the dataset videos themselves can contain sound. Sections 3.2–3.3 place this screen after automated candidate retrieval. A clip needs three positive responses, with up to five annotations and early stopping. Positive answers also prompt a duration question that is described in the text but not shown here. e02e04e06e07e12
What it supports. The interface makes human verification a concrete stage between noisy automated proposals and exported labels. Workers can reject missing actions or unsuitable video content, rather than forcing every candidate into the target class. The resulting label certifies a visually supported action under this review process.
Where the evidence stops. Agreement on one proposed action does not establish exhaustive annotation or temporal boundaries. The source mentions collecting whole-clip-duration judgments but does not specify how those judgments were incorporated into the reported baseline training.
5.2 Keep supervised recognition separate from action execution
The benchmark consumes recorded video and predicts an observed action category. Training and inference differ in how much video each model sees. The LSTM learns from frame-feature sequences and emits its test prediction at the last sampled frame. The two-stream baseline trains separate appearance and optical-flow networks and averages their predictions over test snapshots. The 3D model learns spatiotemporal filters on short clips and averages clip scores at test time. None of these Action outputs is a motor command, and no predicted future is fed into a planner. This distinction also explains the catalog judgment: a dataset for action understanding can support later representation learning without itself introducing a world-action architecture or demonstrating successful interaction with an environment. e02e11e13
Figure 3. Three ways to aggregate video evidence into an action label. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow the arrows upward from observations to Action. In panel a, each frame passes through a ConvNet and the LSTM carries information forward in time; Section 4.1 takes the last output for classification. Panel b routes an RGB image and a stack of optical-flow fields through separate ConvNets. Its plus sign schematically combines predictions; Section 4.2 specifies averaging, not a separately learned fusion module. Panel c processes a short image sequence with spatiotemporal convolutions. The diagram’s K and N mark the ends of image and flow sequences; the actual training and testing frame counts differ and are specified in Table 3. e11e13e14
What it supports. These baselines place temporal processing at different stages: after frame features, in an externally computed motion representation, or within convolutional filters. All three produce labels for observed video. Their differing interfaces help interpret the benchmark, but the diagram alone does not establish which representation is best.
Where the evidence stops. The Action arrows denote classification outputs, not controls sent to an environment. Parameter counts and temporal windows differ across these baselines; architectural comparisons therefore cannot isolate a single temporal mechanism.
5.3 Read relative motion gains alongside absolute difficulty
Reader analysis: the aggregate and classwise results answer different questions. Table 4 shows that averaging RGB and flow predictions improves overall Kinetics recognition, even though flow alone performs worse than RGB overall. Table 6 then identifies classes where the relative ordering reverses strongly. A ratio of 5.3 for rock scissors paper sounds decisive, yet Figure 4 still places that class among the difficult examples for the combined model. The useful deduction is that complementary information need not make a problem easy. A small denominator can also enlarge a ratio. To decide whether the mechanism is reliable, request both absolute stream accuracies and uncertainty, and retain the single-label caveat: some apparent errors may name another valid action present in the same clip. e04e14e16e18
5.4 Training and inference
During training
All Kinetics baselines train from scratch using momentum SGD in TensorFlow with synchronous computation over 64 GPUs, for up to 100k steps. Validation loss triggers tenfold learning-rate reductions; learning rate and weight decay are tuned on validation. No explicit loss equation, exact optimizer settings or frozen-module schedule is supplied. e12
Videos are resampled to 25 fps. Stated larger-side sizes are 340 pixels for ResNet and 128 for 3D; random spatial crops are 299×299 and 112×112. Temporal starts and horizontal flips are randomized consistently within each video; short videos loop. Training windows are 25 RGB frames/5 seconds for LSTM, one RGB plus ten flow frames/0.4 seconds for two-stream, and 16 RGB frames/0.64 seconds for 3D. e12e13
During inference
LSTM classification uses its last output over 50 sampled RGB frames/10 seconds. Two-stream averages predictions over snapshots: 25 RGB and 250 flow frames spanning ten seconds. The 3D model averages scores over 16-frame crops, totaling 240 frames/9.6 seconds. These outputs recognize recorded actions; they neither predict future observations nor execute controls. e11e13
5.5 Implementation flow
- Propose action-centered clips
Combine action names from datasets, motion-capture titles and crowd suggestions. Match YouTube titles, then use image classifiers trained from search relevance feedback to find the top two frame responses. Extract five seconds on either side; boundary clips can be shorter. e06
- Verify without audio
Workers judge whether a named human action occurs. Three positive responses are required, with at most five annotations and early stopping. Each 20-video task includes two ground-truth checks; low-accuracy workers receive warnings. Positive judgments also prompt a whole-clip-duration question. e07
- Remove repeated content
Randomly retain one approved clip per URL, removing around 20%. Within each class, average Inception-V1 features from 25 uniformly sampled 224×224 center crops, threshold pairwise cosine similarity, and retain one random example per connected component. The usual threshold is 0.97; exceptions are unspecified. A further 15% reduction is reported. e08
- Revise classes using model errors
Repeatedly train two-stream classifiers, inspect confusions, and merge, split or remove problematic classes. Rank clips by confidence to focus final manual filtering on likely noise and remaining duplicates. This is dataset curation, not an inference-time feedback controller. e08e11
- Compare three temporal interfaces
ResNet-50 features feed a batch-normalized 512-unit LSTM; separate ResNet-50 streams process RGB and ten-frame optical-flow stacks; a C3D variant processes 16 RGB frames. The latter has eight convolutional, five pooling and two fully connected layers, batch normalization, and first-pool temporal stride two. e11
6. Experiments & results
Kinetics turns short, visually verified YouTube actions into a large classification benchmark. Its contribution is the collection and cleanup pipeline plus baseline diagnostics: appearance and motion contribute differently across classes, and combined streams lead the reported Kinetics comparison. This is the original 400-class paper, with non-exhaustive labels and unresolved count inconsistencies (e01, e02, e03, e04, e08, e14, e18, e19).
6.1 Read the original evidence
Table 4. Combined RGB and flow lead the reported Kinetics comparison. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the rightmost dataset block. Kinetics cells contain top-1/top-5 percentages; the other dataset blocks show classification accuracy on split 1. Within the Two-Stream row, compare RGB, Flow and RGB+Flow before comparing architectures. Dashes indicate entries not reported, not zero performance. All Kinetics models train from scratch. For UCF-101 and HMDB-51, the ResNet-based models use ImageNet pretraining, while the 3D model does not. These caption details are essential to reading the table. The Kinetics test set is held out, with validation tuning described separately; final retraining on a train/validation union is not explicitly resolved. e12e13e14e15
What it supports. On Kinetics, fusion reaches 61.0% top-1 and 81.3% top-5, compared with RGB alone at 56.0% and 77.3%. The differences are 5.0 and 4.0 percentage points, respectively, calculated from the table. Flow alone scores lower overall, yet contributes useful complementary predictions when combined with RGB.
Where the evidence stops. No uncertainty or repeated-run statistics accompany these values. Cross-dataset comparisons mix different label spaces, dataset sizes and initialization. Table 3 also shows different model sizes and temporal inputs, so this is not a matched-capacity ablation.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Kinetics held-out action classification Original 400-class Kinetics; all models trained from scratch, validation used for tuning, held-out test evaluation. Final train/validation-union retraining is not explicitly resolved. | Two-stream RGB+Flow: 61.0 / 81.3 Top-1 / top-5 accuracy (%) | RGB: 56.0 / 77.3; Flow: 49.5 / 71.9; ConvNet+LSTM: 57.0 / 79.0; 3D-ConvNet: 56.1 / 79.5. Fusion improves over RGB by 5.0 top-1 and 4.0 top-5 percentage points (reader subtraction). This supports complementary signals under this protocol; architecture, parameters and temporal interfaces differ. No repeated-run uncertainty is reported. e13e14 |
| UCF-101 and HMDB-51 split-1 classification Separate training/testing on split 1 of each dataset; ResNet-based models use ImageNet pretraining, while 3D trains without it. | Fused two-stream: UCF-101 92.5; HMDB-51 63.7. Classification accuracy (%) | 3D-ConvNet: UCF-101 51.6; HMDB-51 24.3. These are not Kinetics-transfer results. Dataset difficulty cannot be isolated from training-set size, label space and differing initialization. e15 |
| Classwise dependence on optical flow Kinetics two-stream diagnostic comparing each independently trained stream in isolation. | Rock scissors paper: 5.3; robot dancing: 3.1; making a cake: 0.1; eating cake: printed 0.0. Flow accuracy / RGB accuracy (dimensionless) | Ratios above one favor flow; below one favor RGB. Absolute classwise accuracies are not provided in this table. Large relative motion benefits can coexist with poor absolute recognition. Printed zero is not evidence that an action contains no motion. e18e16 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 4. The average score hides large differences between action classes. Original paper, p. 10 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read bar lengths against the horizontal Accuracy axis, then examine the class names. The upper group contains the twenty easiest classes and the lower group the twenty hardest under the two-stream model; the sharp transition is produced by this selected display. Riding mechanical bull and presenting weather forecast sit near the easy end, while faceplanting and slapping appear near the difficult end. Several eating and subtle body-motion classes are also difficult. Treat the graph as a ranked diagnostic, not a complete distribution. It contains no exact numeric labels on bars, so the report uses qualitative ordering rather than invented precision. e16e18
What it supports. The two-stream model does not solve every class equally well. Fine visual distinctions and brief or subtle movements remain challenging, motivating checks beyond aggregate accuracy. Rock scissors paper appears among the difficult classes even though Table 6 gives it a large relative flow advantage; relative improvement and absolute competence are different quantities.
Where the evidence stops. Section 4.5 calls this a full list, but the caption and graphic contain only twenty easiest and twenty hardest classes. The omitted middle and absent uncertainty prevent conclusions about the full distribution or ranking stability.
Table 6. Motion’s relative benefit depends strongly on the action class. Original paper, p. 11 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the right column as flow-stream accuracy divided by RGB-stream accuracy for the named class. Values above one favor flow; values below one favor RGB. These are dimensionless ratios, not percentages or percentage-point improvements. The table selects the five largest and five smallest ratios: rock scissors paper leads at 5.3, while cooking and eating examples occupy much of the lower group. The accompanying caption says the ordering was similar when comparing fused RGB+flow with RGB alone, but supplies no corresponding fused ratios. Use Figure 4 and absolute accuracy measurements to assess whether a large ratio also means reliable recognition. e11e14e16e18
What it supports. Flow is especially beneficial relative to RGB for rock scissors paper and robot dancing, with ratios of 5.3 and 3.1. Appearance performs relatively better for making a cake, whose ratio is 0.1. These class-dependent outcomes explain how an overall weaker flow stream can still improve fused predictions.
Where the evidence stops. The table omits the numerator and denominator accuracies and their uncertainty. A large ratio can reflect a weak RGB denominator. Printed 0.0 values have an unspecified rounding rule and do not prove the absence of useful motion.
7. Analysis & limitations
7.1 What the evidence leaves open
Multiple valid actions may share a clip but only one label is assigned. The authors favor top-5 accuracy; it mitigates some label ambiguity without making annotation exhaustive. e04
Model-guided selection and filtering can favor visually prototypical clips. Classwise deduplication does not establish absence of cross-class or cross-split shared content. Neither alternative collection sampling nor leakage rates are measured. e08e10
The preliminary finding of little gender bias is not a fairness guarantee: 340 classes combine apparent balance with cases where gender cannot be assessed. Subgroup counts and uncertainty are absent; crying and wrestling reveal age/context concerns. e09
Source inconsistencies remain unresolved: Appendix A lists sniffing 399, taking a shower 378, tying bow tie 387, using segway 387 and yawning 398 despite the stated 400 minimum. Figure 4 shows 20 easiest and 20 hardest classes, not the prose’s full list; Table 5 has twelve confusions, not ten. Several Appendix B group counts also disagree with their lists. e03e16e17e19e21
7.2 Questions for discussion
- How much of the apparent motion advantage remains after reporting absolute classwise accuracy and uncertainty (e16, e18)?
- Would classifier-independent sampling retain valid atypical actions removed during confidence-guided cleanup (e08, e10)?
- How should non-exhaustive labels affect interpretation of confusion pairs and top-1 errors (e04, e17)?
8. Reproducibility audit
8.1 Requirements and known gaps
A faithful replication needs the original video IDs, intervals, labels and split membership. The paper links these resources and promises trained models, but supplies no dataset license or current availability verification. Preserve edition-specific labels and counts rather than substituting a later release. e01e03e20
Implementation gaps include learning-rate/weight-decay values, momentum, random seeds, flow algorithm, TensorFlow version and class-specific deduplication thresholds. The 3D model reports 15 videos per GPU on K40 hardware; other model batch sizes are unspecified. Larger-side resizing followed by square cropping also leaves handling of incompatible aspect ratios unclear. e08e11e12
Proposed checks: repeat matched RGB/flow/fusion evaluation with absolute classwise accuracies, then audit feature-connected duplicates across classes and splits. Report uncertainty and retain the original single-label protocol alongside any richer annotation audit. e04e08e14e18
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Does stream fusion help the same classes consistently?
Reader-proposed experiment, not performed: preserve the original Kinetics splits and train matched RGB and flow ResNet-50 baselines from scratch, then compare each stream with their prediction average on exactly the same test clips. Hold preprocessing and snapshot positions fixed; record every unresolved implementation choice and use multiple seeds. Report top-1/top-5 accuracy, per-class absolute accuracies, ratios and paired uncertainty estimates. Inspect rock scissors paper, robot dancing and making a cake explicitly. Stable positive fusion gains and consistent classwise stream ordering would support complementarity. Ratios driven by near-zero RGB accuracy, unstable ordering or gains within uncertainty would weaken a strong motion-benefit interpretation. e11e12e13e14e16e18
Check 2: Does classwise cleanup leave shared footage across labels or splits?
Reader-proposed audit, not performed: given the original release manifest and accessible clips, reproduce averaged Inception-V1 features from 25 uniformly sampled 224×224 center crops and the reported 0.97 cosine threshold. Compare the published within-class procedure with a global graph spanning classes and splits. Manually inspect flagged pairs, using random unflagged pairs and similar-scene clips as controls for false matches. Keep the original evaluation intact, then report a separate subset excluding verified train-test duplicates and its accuracy change. Cross-split shared footage plus a meaningful accuracy decrease would support leakage concerns; low verified overlap with stable scores would weaken them. Also log appendix count discrepancies rather than silently repairing the release. e03e08e14e19
8.3 Reading coverage
Visual audit: All 22 supplied PDF pages were rendered and actually inspected, covering the title/version/authors, Figures 1–4, Tables 1–6, every method/training/evaluation detail cited here, the references and both appendices. Six final original crops were separately viewed; the architecture crop was widened and re-viewed to retain its rightmost time label. Arrows and the schematic fusion plus sign were checked against Sections 4.1–4.3; prediction averaging follows the text. Figure 4/Table 5 prose-caption count inconsistencies and Appendix A/B count discrepancies are retained in the report. Full pages supply uncropped captions and table protocol notes. No external videos, code, checkpoints, separate supplements or later editions were inspected.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract
- 1. Introduction
- 2. An Overview of the Kinetics Dataset
- 3. How the Dataset was Built; Sections 3.1–3.6
- 4. Benchmark Performance; Sections 4.1–4.5
- 5. Conclusion and Acknowledgements
- References
- Appendix A. List of Kinetics Human Action Classes (pp. 12–17)
- Appendix B. List of Parent-Child Groupings (pp. 17–22)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- The extraction limitation was addressed by inspecting all 22 PDF pages, all four figures and six tables. All six supplied text chunks were read in full.
- Identity: the title and twelve-author list match the catalog, with João printed with its accent. The artifact is arXiv:1705.06950v1, 19 May 2017; no later revision or dataset edition was supplied or compared (e01).
- The title page gives email addresses but no explicit affiliations; affiliations are omitted from metadata.
- No linked dataset, project site, code, checkpoint or separate supplement was inspected, and no experiments were reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title block and arXiv margin stamp
The title is The Kinetics Human Action Video Dataset; twelve authors include João Carreira. The artifact is arXiv:1705.06950v1 [cs.CV], 19 May 2017. No institutional affiliation block is printed.
Go to primary source ↓e02PDF p. 1, Abstract and Section 1
Kinetics targets classification of approximately ten-second trimmed human-action YouTube clips, with sound, rather than temporal localization. Its motivation is sufficient scale and variation for training deep networks and benchmarking architectures.
Go to primary source ↓e03PDF p. 2, Tables 1–2 and Section 2, Statistics
The paper reports 400 classes and 306,245 clips from as many distinct videos; 400–1150 clips per class, with 250–1000 training, 50 validation and 100 test clips per class. Table 1 lists UCF-101 with 101 classes, 13,320 clips and 2,500 videos. Appendix A contradicts the headline per-class lower bound.
Go to primary source ↓e04PDF p. 2, Section 2, Content and Non-exhaustive annotation; PDF p. 17, Appendix B opening
A clip can contain multiple Kinetics actions while appearing under one label. The authors prefer top-5 evaluation for this reason. Parent-child groups are overlapping, non-comprehensive guides rather than a deep hierarchy.
Go to primary source ↓e05PDF p. 3, Figure 1 and caption
Sequences illustrate individual, person-person and person-object actions: headbanging, shaking hands, playing violin and trumpet, and dribbling versus dunking basketball. The caption stresses that a single image need not distinguish an action.
Go to primary source ↓e06PDF p. 4, Sections 3.1–3.2
Action names come from existing datasets, motion-capture titles and crowd suggestions. YouTube titles retrieve videos; image classifiers derived from Google Image Search relevance feedback identify the top k=2 frame responses. Five seconds on each side are extracted, shortened near video boundaries.
Go to primary source ↓e07PDF pp. 4–5, Section 3.3 and Figure 2
Workers assess whether a named human action occurs, with audio disabled. Tasks contain 20 videos including two ground-truth slots, with a warning below 50% worker accuracy. Acceptance requires three positive responses, with up to five annotations and early stopping. Positive responses trigger a whole-clip-duration question; video IDs, times and labels are exported.
Go to primary source ↓e08PDF p. 6, Section 3.4, De-duplicating videos, Detecting noisy classes and Final filtering
One clip is randomly retained per YouTube link, removing around 20%. Within-class cross-link deduplication averages Inception-V1 features from 25 uniformly sampled 224×224 center crops, thresholds cosine similarity (usually 0.97), and retains one random example per connected component. A further 15% reduction and unspecified threshold adjustments are reported. Two-stream confusions and confidence rankings guide class revision and final manual filtering.
Go to primary source ↓e09PDF pp. 6–7, Section 3.5; PDF p. 10, Section 5
The preliminary bias analysis groups 340 of 400 classes as either not dominated by one gender or not visually assessable for gender. Authors report little classifier-bias evidence in examined cases but note crying errors for non-babies and an age/scene confound in wrestling. A thorough analysis is deferred; subgroup counts and uncertainty are not supplied.
Go to primary source ↓e10PDF p. 7, Section 3.6
The authors discuss RGB-classifier selection bias and argue that long clips retain visual variety. They report no controlled comparison against classifier-independent candidate selection.
Go to primary source ↓e11PDF pp. 7–8, Sections 4.1–4.3 and Figure 3
Baselines comprise ResNet-50 plus a batch-normalized 512-unit LSTM and classification layer; independently trained RGB and optical-flow ResNet-50 streams with prediction averaging; and C3D with eight convolutional, five pooling and two fully connected layers. Ten-frame flow stacks have two channels per frame. C3D uses 16-frame 112×112 inputs, batch normalization and first-pool temporal stride two; 15 videos per GPU batch on K40 GPUs are reported. LSTM uses its last output; 3D averages clip scores. Figure 3 depicts observed-action classification.
Go to primary source ↓e12PDF pp. 8–9, Section 4.4
All Kinetics models train from scratch with momentum SGD in TensorFlow, synchronously over 64 GPUs for up to 100k steps. Learning rate falls tenfold on validation-loss saturation; learning rate and weight decay are tuned on validation. Videos become 25 fps; larger sides are 340 pixels for ResNet and 128 for 3D, with stated random spatial crops of 299×299 and 112×112, temporal crops and consistent horizontal flips. Short videos loop. Exact optimizer values, flow implementation, software version and handling of aspect ratios incompatible with these crops are unspecified.
Go to primary source ↓e13PDF p. 9, Table 3 and caption
Training/testing: LSTM has 25/50 RGB frames over 5/10 seconds and 29M parameters; two-stream has 1 RGB+10 flow over 0.4 seconds versus 25 RGB+250 flow over 10 seconds and 48M parameters; 3D has 16/240 RGB frames over 0.64/9.6 seconds and 79M parameters.
Go to primary source ↓e14PDF p. 9, Table 4, Kinetics columns and caption; Section 4.5 opening; Section 4.4
Held-out Kinetics top-1/top-5 percentages: LSTM 57.0/79.0; RGB 56.0/77.3; flow 49.5/71.9; RGB+flow 61.0/81.3; 3D 56.1/79.5. Section 4.5 describes train/validation use and held-out testing; Section 4.4 specifies validation tuning, without explicitly resolving final train/validation-union retraining. No repeated-run statistics or error bars are supplied.
Go to primary source ↓e15PDF p. 9, Table 4, UCF-101 and HMDB-51 columns and caption; Sections 4.4–4.5
Split-1 accuracies: fused two-stream 92.5% on UCF-101 and 63.7% on HMDB-51; 3D 51.6% and 24.3%. ResNet-based baselines use ImageNet pretraining on these datasets; 3D does not. All Kinetics models train from scratch.
Go to primary source ↓e16PDF p. 9, Section 4.5, Class difficulty; PDF p. 10, Figure 4 and caption
Figure 4 shows the 20 easiest and 20 hardest classes, although p. 9 calls it a full list. Riding mechanical bull and presenting weather forecast appear at the easy end; slapping and faceplanting at the difficult end. Eating and subtle body motions are discussed as difficult. Bars have an accuracy axis but no exact printed values or uncertainty.
Go to primary source ↓e17PDF p. 9, Section 4.5, Class confusion; PDF p. 11, Table 5 and caption
The prose refers to ten top confusions; the table and caption contain twelve. Pairs include riding mule / riding or walking with horse at 40%, swing dancing / salsa dancing at 36%, and triple jump / long jump at 26%. The table does not explicitly define the percentage denominator or direction beyond Class 1 and Class 2 headings.
Go to primary source ↓e18PDF pp. 9–10, Section 4.5, Classes where motion matters most; PDF p. 11, Table 6 and caption
Flow/RGB class-accuracy ratios: rock scissors paper 5.3; sword fighting and robot dancing 3.1; air drumming 2.8; exercising arm 2.5; making a cake, cooking sausages and sniffing 0.1; eating cake and making a sandwich 0.0. Underlying classwise accuracies, confidence intervals and rounding rules are absent.
Go to primary source ↓e19PDF p. 12, Appendix A introduction; PDF p. 16, entries 321 and 348; PDF p. 17, entries 369, 376 and 398; compare PDF pp. 1–2
Appendix A says bracketed numbers count clips. Below the claimed 400 minimum it lists sniffing 399, taking a shower 378, tying bow tie 387, using segway 387 and yawning 398. The source does not reconcile these exceptions with the overview or split bounds.
Go to primary source ↓e20PDF p. 2, end of Section 1; PDF p. 10, Section 5
A project page is identified for video URLs and temporal intervals; trained TensorFlow baselines are promised for future release. The supplied paper does not specify a dataset license or establish current resource availability.
Go to primary source ↓e21PDF p. 17, Appendix B introduction; PDF p. 18, body motions and cloths; PDF p. 20, mobility–land; PDF p. 22, touching person
Appendix B is non-exclusive and non-comprehensive. Printed group sizes can disagree with lists: body motions says 16 but lists 15; cloths says 8 but lists 9; mobility–land says 20 but lists 21; touching person says 11 but lists 10.
Go to primary source ↓8.5 Primary sources
The Kinetics Human Action Video Dataset ↗
PDF · 9,934 extracted words
Source fingerprint
af6f03ec60374ba2974e0a8a52b080834b3da49172eb5f2490c6b9c189071541