The "Something Something" Video Database for Learning and Evaluating Visual Common Sense
1. Paper overview
In one sentence: Recording fine-grained actions from structured language prompts creates a demanding video-recognition dataset, but recognition errors alone do not establish physical common sense. identityaction-examplescontrastive-groupsdataset-scaledistributionssubset-protocolbaseline-table
| At a glance | What to know |
|---|---|
| Research problem | Author claim The authors argue that high-level activity labels and image-based recognition can reward object or scene shortcuts without representing how an interaction unfolds. Even an action such as opening changes appearance across doors, bags and zippers. Their proposed route toward visual common sense is therefore detailed supervision on observable interactions, with language expressing distinctions that a fixed vocabulary of broad activities would miss. motivationstructured-labels |
| Core mechanism | Source description A crowd-recorded collection of 108,499 videos, 174 template classes and 2–6-second clips, averaging 4.03 seconds. The reported train/validation/test ratio is 8:1:1, with each worker confined to one split. dataset-scale |
| A key reported result | 40-class selected action-template classification: 2D+3D-CNN: 63.8 / 50.7 Top-1 / top-2 error (%); lower is better. The ten groups plus 30 additional common classes, 53,267 videos total. Table 4's validation-versus-test reporting split is unspecified. Pre-3D CNN + Avg: 66.2 / 52.7; Pre-2D CNN + LSTM: 77.8 / 68.0; 2D CNN + Avg: 88.0 / 78.5. Combining encodings improves the reported comparison. Changing class definitions, sample count, pretraining and temporal encoders prevents treating all differences as isolated effects of motion modeling. subset-protocolframe-encodersclip-encodersbaseline-table |
| Reading caution | Reader analysis Label ambiguity complicates training and interpretation. The authors use top-k reporting and simplified groups, which reduce difficulty but also change the semantic task. Neither establishes that a model learned physical common sense. baseline-scopesubset-protocoldiscussion |
Core contributions
- Source description
A crowd-recorded collection of 108,499 videos, 174 template classes and 2–6-second clips, averaging 4.03 seconds. The reported train/validation/test ratio is 8:1:1, with each worker confined to one split. dataset-scale
- Source description
Structured captions combine prescribed actions with worker-selected object descriptions. Groups of closely related actions, including pretending, aim to make object outcomes matter. A collection platform supports resumable submissions, worker-dependent task choices, automated checks and subsequent human verification. structured-labelssamplingcontrastive-groupscollection-platform
Figure 4. The supervision describes what happens to an object, including when an apparent placement is only pretended. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each row from left to right, then connect its outcome to the words below it. In the first row, the remote remains inside the box after the hand withdraws. In the second, the hand brings candy toward the chair but takes it away again. The last two rows illustrate an induced fall and a changing spatial relationship. Italic phrases identify object descriptions supplied by workers; the surrounding action wording supplies the template. Sections 3.3–3.4 explain how workers choose objects and receive related action groups. The illustrated mechanism is the production of grounded training examples; it is not a neural-network flowchart. action-examplessamplingcontrastive-groupsstructured-labelsbaseline-scope
What it supports. These examples make the intended distinction concrete: similar-looking hand movements can correspond to different object outcomes. The paper's design asks recognition systems to attend to those outcomes. Structured captions also connect a reusable action pattern to particular objects without requiring a separate template for every noun phrase.
Where the evidence stops. These are illustrative frame samples, not model predictions or matched counterfactual pairs. Section 4 evaluates template classification, with expanded captions deferred. The figure therefore supports the collection design, not a demonstrated ability to reason about unseen physical situations.
2. Motivation
2.1 The problem and the proposed response
The authors argue that high-level activity labels and image-based recognition can reward object or scene shortcuts without representing how an interaction unfolds. Even an action such as opening changes appearance across doors, bags and zippers. Their proposed route toward visual common sense is therefore detailed supervision on observable interactions, with language expressing distinctions that a fixed vocabulary of broad activities would miss. motivationstructured-labels
2.2 What this reading follows
A hand moving toward a chair does not tell you whether it leaves an object there. Something-Something makes such distinctions part of the supervision: workers enact prescribed actions, choose everyday objects and complete short descriptions. This reading covers the 2017 paper's 108,499-video collection and its template-classification baselines. Follow the examples first, then inspect the dataset's uneven coverage before comparing errors. The best combined encoder is reported on simplified 10- and 40-class problems; the full 174-class result comes from a different experiment. The figures clarify what was collected, while the table shows how much remains unresolved about learning the intended concepts. identityaction-examplescontrastive-groupsdataset-scaledistributionssubset-protocolbaseline-table
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Datasets |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
3.1 Evidence-based assessment
Supports the recorded classification
The dataset category and video-action/language-annotation scope are supported by the collection protocol. Architecture, prediction paradigm and quadrant are appropriately not applicable to this resource: its classification baselines do not constitute joint future/action prediction, inverse dynamics or a world-action control model. No separate reannotation procedure is demonstrated in this paper. structured-labelscollection-platformbaseline-scopediscussion
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
5. Method in detail
5.1 Begin with a distinction that the camera can reveal
The collection reverses the usual labeling workflow. Instead of finding a video and attaching a broad category, the platform gives a worker an action template to enact. The worker supplies the object choice and noun phrase, making an instance such as placing a remote into a box. Figure 4 shows why the label must describe more than a recognizable object: approaching a chair with candy can end without leaving it there. Section 3.4 groups related actions and encourages the same objects across them, aiming to remove easy object-based shortcuts. Automated checks and subsequent human review check submissions. The causal ambition belongs to the dataset design: making outcomes necessary for discrimination should encourage useful representations. That ambition is not itself evidence that the trained models achieved physical understanding. samplingaction-examplescontrastive-groupscollection-platformbaseline-scope
5.2 Follow the observed video into a class prediction
The baselines consume an already recorded interaction. VGG-16 encodes individual frames; averaging yields one video descriptor, while the recurrent alternative uses the last state of a 256-unit LSTM. The scratch 3D alternative encodes nine-frame clips, masks padding and stacks four descriptors. The Sports-1M-initialized 3D model instead averages five overlapping 16-frame clips sampled at 8 fps. Concatenating the best 2D and 3D encodings produces the combined representation. Training targets action-template classes, not the completed object descriptions. Temporal offsets vary during training and are fixed at zero during validation/testing. This distinction matters for a world-model reading: the output labels an observed action. It neither predicts a future image nor issues a motor command, even though the discussion connects visual understanding to possible future visuomotor systems. frame-encodersclip-encoderspreprocessingbaseline-scopediscussion
5.3 Interpret task difficulty without conflating protocols
Before interpreting Table 4, reconstruct what a class means. The easy task combines 41 selected labels into ten groups, covering 28,198 videos; the next task retains those groups and adds 30 classes, covering 53,267 videos. Higher error on the second task is therefore not a controlled experiment that changes only the number of labels. The full-class experiment introduces another training stage through initialization from the 40-class model, and only one row reports it. Human performance comes from a separate informal study of approximately 700 test samples and should not be treated as a matched ceiling. Reader analysis: these results establish that the tested baselines struggle under the reported configurations. To attribute that struggle to physical reasoning rather than label ambiguity, sampling imbalance or shortcuts requires additional controlled checks. subset-protocolbaseline-tablefull-class-and-humanbaseline-scopedistributions
5.4 Training and inference
During training
The task is supervised template classification, with full captioning and expanded-label evaluation deferred. Scratch and pretrained baselines are distinguished; the pretrained 3D model starts from Sports-1M and is fine-tuned. The full-class experiment additionally starts from a model pretrained on the 40 selected classes. Dropout on the first fully connected layer and batch normalization on the last are reported as helpful, without quantitative isolation. baseline-scopeframe-encodersclip-encodersfull-class-and-human
Training uses a random temporal offset between zero and the downsampling factor of four. Flipping invariant classes and small rotations reportedly produced no significant gain. Progressive caption complexity is a proposed data curriculum; the paper does not evaluate a full language curriculum. preprocessingstructured-labels
During inference
Validation and testing fix the temporal offset at zero. An observed video's frame/clip encoding supports template prediction, reported with top-k errors. There is no action extraction, rollout planner, execution interface or feedback-control loop in these baselines; references to robotics motivate future applications. preprocessingframe-encodersclip-encodersbaseline-tablediscussion
5.5 Implementation flow
- Specify the action before recording
Workers receive templates, choose suitable objects, enact the interaction and enter noun phrases during upload. This samples plausible action–object combinations through human choices rather than attempting uniform coverage of their enormous Cartesian product. samplingcollection-platform
- Make nearby meanings distinguishable
Action groups encourage recording similar actions with the same object, such as putting something behind an object versus only pretending. Figure 4 pairs temporal samples with completed descriptions; its italic noun phrases expose the variable slots. This is dataset design, not an explicit contrastive training loss. contrastive-groupsaction-examplesbaseline-scope
- Prepare temporal observations
Baseline preprocessing samples at 24 fps and normally resizes to 84 × 84 pixels; pretrained models determine their own input resolution. Temporal Gaussian filtering precedes downsampling toward 6 fps. The paper calls the filter variance 48 pixels, an unusual unit specification that remains unresolved. preprocessing
- Encode frames or clips
VGG-16 frame features are averaged, or ImageNet-pretrained features enter an LSTM whose final 256-unit hidden state represents the video. The scratch 3D model stacks four masked nine-frame clip encodings after padding to 36 frames, yielding 4096 dimensions. The pretrained 3D variant averages five overlapping 16-frame clips at 8 fps. The combined model concatenates the best 2D and 3D video encodings. frame-encodersclip-encoders
6. Experiments & results
Something-Something turns everyday human–object interactions into supervised video-understanding tasks: workers enact action templates, select objects and fill the templates' noun slots. The 2017 paper describes 108,499 clips and 174 template classes, with real-versus-pretended actions intended to discourage superficial recognition. Its experiments predict template labels; they do not demonstrate future-video generation, physical-property estimation or robot control. Standard baselines remain weak, and the strongest combined encoder is evaluated only on simplified label subsets.
This dataset paper contains example video frames and collection-distribution plots, but no neural architecture diagram. Figure 4 therefore illustrates the data-creation mechanism, with encoders explained from Section 4.2. The supplied paper has no controlled ablation table removing contrastive grouping or holding pretraining and sampling fixed; Figures 2–3 serve as dataset diagnostics, and Table 4 provides baseline comparisons. Supplementary action lists, class mappings, platform screenshots and prediction examples are referenced but absent. No caption-generation, future-prediction or robot-execution results are supplied. action-examplesdistributionsframe-encodersclip-encodersbaseline-scopebaseline-tablerelease-and-supplement
6.1 Read the original evidence
Table 2. The collection's scale is tied to this 2017 source, not to an inferred later release. Original paper, p. 3 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Separate the four quantities before using them. The first row counts videos; the second counts template labels, not all possible completed sentences or objects. The duration row reports a mean in seconds, while the last row gives a rounded class average. Section 3 adds an essential evaluation constraint absent from the crop: the train, validation and test splits follow an 8:1:1 ratio and keep every worker in a single split. Use that paragraph with the table when defining a reproduction. Neither the mean class size nor the total video count tells you how many examples each evaluation class contains. identitydataset-scaledataset-diversitydistributions
What it supports. The paper contributes a sizable collection of short, labeled interactions: 108,499 videos across 174 templates with a 4.03-second average duration. Its worker-based partition is a concrete safeguard against sharing the same contributor across train and evaluation splits, and belongs in any faithful description of the dataset.
Where the evidence stops. This is a dataset inventory, not an accuracy result. Exact split counts are not given here, and worker separation does not establish that objects or action–object combinations are disjoint. The rounded average also hides the imbalance visible in Figure 2.
Table 4. Lower errors favor combined encodings on simplified subsets, while full-class evidence is sparse. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. First read the header: every value is an error percentage, so smaller is better. Compare methods vertically within one class block and one top-k column. The 10-class block merges 41 original classes; the 40-class block adds 30 common classes to those groups. Thus moving horizontally changes both semantic granularity and available data. Section 4.2 distinguishes scratch, pretrained, averaged, recurrent and stacked encoders; it also gives different clip sampling for the pretrained 3D model. Finally inspect the dashes: they mark unreported combinations, including the combined model on 174 classes. The table supports baseline comparisons, not a complete factorial test of architecture or pretraining. baseline-tablesubset-protocolframe-encodersclip-encodersfull-class-and-human
What it supports. The combined model reports 44.9% top-1 error on 10 classes and 63.8% on 40, compared with 47.5% and 66.2% for pretrained 3D averaging. Only the pretrained 3D row covers all 174 labels, reporting 88.5% top-1 error. Stronger subset performance does not imply that the combined model's full-class performance was measured.
Where the evidence stops. Section 4.3 reports 70.3% full-class top-5 error, whereas this table shows 70.0%; both are preserved. The table does not identify validation versus test or report uncertainty. Models also differ in pretraining and temporal processing, limiting causal interpretation.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| 10-class grouped action-template classification 41 selected original classes merged into 10 groups, 28,198 videos total. The dataset has worker-disjoint 8:1:1 splits; Table 4 does not explicitly identify its reporting split as validation or test. | 2D+3D-CNN: 44.9 / 27.1 Top-1 / top-2 error (%); lower is better | Pre-3D CNN + Avg: 47.5 / 29.2; Pre-2D CNN + LSTM: 52.3 / 34.1. Concatenation has the lowest reported errors on this simplified problem. It is not a result on ten unchanged original labels or on all 174 classes. dataset-scalesubset-protocolbaseline-table |
| 40-class selected action-template classification The ten groups plus 30 additional common classes, 53,267 videos total. Table 4's validation-versus-test reporting split is unspecified. | 2D+3D-CNN: 63.8 / 50.7 Top-1 / top-2 error (%); lower is better | Pre-3D CNN + Avg: 66.2 / 52.7; Pre-2D CNN + LSTM: 77.8 / 68.0; 2D CNN + Avg: 88.0 / 78.5. Combining encodings improves the reported comparison. Changing class definitions, sample count, pretraining and temporal encoders prevents treating all differences as isolated effects of motion modeling. subset-protocolframe-encodersclip-encodersbaseline-table |
| 174-class action-template classification Full label set; prose describes a 3D CNN pretrained on the 40-class task. Table 4 labels the row Pre-3D CNN + Avg; its reporting split is unspecified. | Table 4: 88.5 / 81.5 / 70.0; Section 4.3 instead gives top-5 error 70.3%. Top-1 / top-2 / top-5 error (%); lower is better | Other Table 4 methods have no reported full-class entries. Informal human evaluation is a separate protocol. The full task remains difficult. The top-5 discrepancy is unresolved; no number is silently corrected and missing cells are not estimated. baseline-tablefull-class-and-human |
| Informal human classification on 174 classes Ten individuals classified approximately 700 test samples in total. | Approximately 60 Accuracy (%) | The full-class machine result is reported as error on an unspecified reporting split; the human study is small and informal. This indicates human headroom while also exposing label difficulty; it is not a calibrated ceiling or a matched statistical comparison. full-class-and-humanbaseline-table |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 2 — dataset coverage diagnostic. The truncated class histogram exposes uneven coverage of simple and more specialized interactions. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the vertical axis: bar heights count videos, not classification performance. Next read the slanted action labels and notice the ellipsis between the two groups of bars. The paper explicitly truncates this distribution for display, so the gap does not mean intermediate classes are empty. The left side includes frequent actions such as holding; the right side contains much rarer interactions. Section 3 supplies the exact endpoints: 986 videos for holding something and 77 for poking a hole into a substance. Treat this as a diagnostic of sampling coverage when reading the later error table, rather than as an ablation of a model component. dataset-scaledataset-diversitydistributionscontrastive-groupsbaseline-table
What it supports. The nominal average of roughly 620 videos per class masks substantial variation. Some of the fine distinctions motivating the collection have far less support than frequent actions. Reader analysis: overall classification error can therefore mix differences in semantic difficulty with differences in available examples; this figure alone cannot separate them.
Where the evidence stops. The figure is truncated and has no per-class model errors or split-specific counts. It cannot establish that rarer classes caused particular failures. No causal effect of contrastive grouping is measured by these bars.
Figure 3 — duration and object diagnostics. Short clips and familiar object labels characterize the observed collection, with uneven lexical frequencies. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the panels separately because their horizontal axes have different meanings. The upper histogram bins duration in seconds, as established by the caption; its vertical axis counts videos. The lower panel lists only the 15 most frequent object labels. Box is tallest, followed by bottle and water, while bottle and water bottle remain separate entries. Section 3 cautions that distinct object names include differences in case, stemming and determiners. Connect this with Section 3.3: workers select the objects they consider suitable for a template. These plots describe the resulting observations and language; they do not map the entire space of possible physical interactions. dataset-scaledataset-diversitydistributionssampling
What it supports. The collection favors short demonstrations and common manipulable objects. Reader analysis: human object selection supplies plausible combinations efficiently, but frequency diversity is not equivalent to controlled coverage of every action–object pairing. A reproduction should preserve raw descriptions and document any normalization used to study object generalization.
Where the evidence stops. The text states a 2–6-second range, while the histogram includes a small 6–7-second bin; endpoint/binning details are unstated. The lower panel is not a count of unique physical objects, and exact bar counts should not be inferred from the raster.
7. Analysis & limitations
7.1 What the evidence leaves open
Label ambiguity complicates training and interpretation. The authors use top-k reporting and simplified groups, which reduce difficulty but also change the semantic task. Neither establishes that a model learned physical common sense. baseline-scopesubset-protocoldiscussion
Collection is uneven: class counts range from 77 to 986, and the 23,137 distinct object strings include spelling, case and determiner variants. Worker separation addresses contributor overlap, but object-disjoint or action–object-combination generalization is not demonstrated. Contrastive grouping is motivated as bias reduction without a controlled removal experiment. dataset-scaledataset-diversitydistributionscontrastive-groupsbaseline-table
7.2 Questions for discussion
- Would matched real-versus-pretended action pairs still defeat an encoder when object and background cues are controlled?
- How much apparent generalization survives withholding action–object combinations in addition to workers?
8. Reproducibility audit
8.1 Requirements and known gaps
Reproduction needs the exact 2017 release and worker-based partitions, class mappings, pretrained checkpoints and preprocessing choices. The paper links a release but supplies no license terms here. Supplementary class-selection details and example predictions are referenced rather than included. release-and-supplementdataset-scalesubset-protocolclip-encoders
The supplied experiment description leaves optimizer, learning-rate schedule, batch size, training duration, seeds, hardware/software environment, precise loss and frozen-layer policy unspecified. Resolve the Gaussian variance units, the Table 4 reporting split and top-5 discrepancy before claiming numerical replication. Repeated-run uncertainty is not reported. preprocessingframe-encodersclip-encodersbaseline-tablefull-class-and-human
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Does temporal order change the decision?
Reader-proposed; not run. Recover the original worker partitions and label mapping, then evaluate each reproduced encoder on identical held-out videos with either original or randomly permuted frame order. Keep frame content, temporal sample count, weights and preprocessing fixed; use deterministic evaluation. For the averaging baseline, permute the same extracted frame-feature set as an implementation control: its average should remain unchanged. Report per-class error changes for directional movement and real-versus-pretended actions, alongside aggregate errors. If recurrent and 3D decisions are insensitive to order on these classes, the claim that their advantage depends on sequence information would be weakened. This within-model comparison avoids attributing pretraining differences to temporal order. dataset-scalesubset-protocolcontrastive-groupspreprocessingframe-encodersclip-encodersbaseline-table
Check 2: Does recognition transfer to withheld action–object combinations?
Reader-proposed; not run. Start from the 2017 worker-disjoint split and document a noun-normalization mapping while retaining raw descriptions. Construct an additional evaluation set of action–object combinations absent from training, requiring each component action and object to remain represented separately. Compare it with a worker-disjoint control matched for class support, training-set size and object frequency; keep encoder and training choices fixed. Report per-class errors and real-versus-pretended confusions with repeated-run uncertainty. A substantial additional error increase would expose a gap between worker generalization and compositional generalization. If the required matching is infeasible for rare classes, report those exclusions and counts rather than silently changing the task. dataset-scaledataset-diversitysamplingcontrastive-groupsdistributionsbaseline-scope
8.3 Reading coverage
Visual audit: All nine pages were rendered and visually inspected, including the title/author block, Figures 1–4, Tables 1–4, collection protocol, preprocessing, encoder specifications, results, discussion and references. All five final original crops were separately viewed. Page 6 supplies grouping and sampling evidence for the examples and proposed checks; pages 7–8 supply training, inference and numerical interpretation details. Table 4's 70.0 versus prose 70.3 top-5 discrepancy and Figure 3's duration-bin ambiguity are retained. No architecture arrows, equations or mask symbols occur in these crops; padding masks are described only in Section 4.2. The referenced supplement/appendix, dataset clips, code and later editions remain outside this reading. No experiments were run.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9. Appendix coverage: not present.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract
- 1. Introduction
- 2. Related work
- 2.1. Learning intuitive physics
- 3. The something-something dataset
- 3.1. Crowdsourced video recording
- 3.2. Natural language and curriculum learning
- 3.3. Sampling action-object combinations
- 3.4. Grouping and contrastive examples
- 3.5. Data collection platform
- 4. Baseline experiments
- 4.1. Pre-processing
- 4.2. Model specifications
- 4.3. Results
- 5. Discussion
- References
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- Identity/version note: the title and all 14 authors match the catalog, allowing title capitalization/quotation typography and the title-page spelling Joanna Materzyńska versus catalog Joanna Materzynska. This is the ICCV 2017 CVF open-access paper, printed pages 5842–5850. Its title-page notice states that it is identical to the IEEE Xplore version apart from the watermark; that second artifact was not independently inspected. No numbered revision is established. The catalog BibTeX key ssv2 is not evidence of a later dataset edition; none was supplied or substituted.
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout. This acquisition limitation was addressed by visually inspecting all nine PDF pages and all five final crops.
- Separate supplemental material availability has not been fully verified. The referenced action-group lists, platform screenshots, 41-to-10 class mapping and prediction examples are not present in the supplied nine-page PDF; no supplement was supplied.
- Dataset files, external release pages, code and cited works were not inspected. No experiments were reproduced. Institutional affiliations are not explicitly listed on the inspected title page; email domains were not treated as affiliations.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
identityPDF p. 1 (printed p. 5842), title, author block and CVF notice; PDF p. 9 (printed p. 5850)
Title matches apart from casing/quotation typography; all 14 authors are credited, including Joanna Materzyńska. The CVF notice identifies the ICCV open-access version and states equivalence to IEEE Xplore except for the watermark. The supplied paper ends with references on p. 9.
Go to primary source ↓motivationPDF pp. 1–3, Abstract, Sections 1, 2 and 2.1
The authors motivate video supervision for physical aspects and fine-grained actions, discuss object/action interdependence and shortcuts, and distinguish this approach from pixel prediction and learning through physical interaction.
Go to primary source ↓dataset-scalePDF p. 3 (5844), Section 3 and Table 2
Reports 108,499 videos, 174 labels, 2–6-second duration, 4.03-second mean and a rounded 620-video class average. Splits follow an 8:1:1 train/validation/test ratio, keeping each worker in only one split.
Go to primary source ↓dataset-diversityPDF p. 3, Section 3, paragraphs below Table 2
Reports 23,137 distinct object names including lexical variants, an estimate of at least a few thousand distinct objects, 1,133 workers and 127.32 workers per class on average. Counts range from 77 for poking a hole into a substance to 986 for holding something.
Go to primary source ↓structured-labelsPDF pp. 4–6, Section 3.2
Templates combine predefined language with worker-provided slots; current captions emphasize short descriptions. Increasing language complexity in response to attainable validation accuracy is proposed as a curriculum.
Go to primary source ↓samplingPDF p. 6, Section 3.3
Workers receive action templates, choose objects, record clips and enter object descriptions, exploiting the nonuniform plausibility of action–object combinations.
Go to primary source ↓contrastive-groupsPDF p. 6, Section 3.4, including example groups
Similar actions and pretending examples are intended to reduce reliance on hand position, velocity or camera cues. Groups encourage reuse of objects and include actual versus pretended placement and poking with different outcomes.
Go to primary source ↓collection-platformPDF pp. 6–7, Section 3.5
The platform supports batch submission, worker-conditional task options and feedback. Workers upload videos and fill template roles; automated checks include length and uniqueness, and a human operator subsequently checks each submission.
Go to primary source ↓action-examplesPDF p. 5, Figure 4 and caption; PDF p. 1, Figure 1 and caption
Figure 4 shows placing a remote in a box, pretending to place candy on a chair, pushing a chilli off a table and moving a puncher closer to scissors, with object phrases in italics. Figure 1 illustrates picking up a shoe and filling the template slot.
Go to primary source ↓distributionsPDF p. 4, Figures 2–3 and captions; PDF p. 3, Section 3
Figure 2 explicitly truncates the class-frequency display. Figure 3 shows video-duration frequencies in seconds and frequencies of the 15 most common object labels; box has the tallest object bar. The duration graphic includes a small bar in the 6–7-second bin despite the text's stated 2–6-second range.
Go to primary source ↓baseline-scopePDF p. 7, Section 4 before Section 4.1
Experiments classify label templates; full captions and expanded labels are deferred. Label ambiguities and difficulty motivate top-k reporting and simplified class groups. Grouping is also described as useful initialization.
Go to primary source ↓preprocessingPDF p. 7, Section 4.1
Specifies 24-fps sampling, normally 84 × 84 resolution, temporal Gaussian filtering with stated variance 48 pixels and target 6 fps, random training offset relative to factor four and fixed zero evaluation offset. Flips and small rotations reportedly gave no significant gain.
Go to primary source ↓frame-encodersPDF pp. 7–8, Section 4.2, introductory paragraph and 2D variants
VGG-16 frame features are averaged for scratch or ImageNet-pretrained variants; pretrained features also feed a 256-unit LSTM whose last state is used. Dropout on the first fully connected layer and batch normalization on the last are described as beneficial.
Go to primary source ↓clip-encodersPDF p. 8, Section 4.2, 3D-CNN + Stack, Pre-3D-CNN + Avg and 2D+3D-CNN paragraphs
Scratch 3D features use 1024-unit fully connected layers, nonoverlapping nine-frame clips, padding to 36 frames and masked four-column stacking into 4096 dimensions. Sports-1M initialization is fine-tuned with 8-fps input and five 16-frame clips overlapping by eight frames; their features are averaged. Combined encodings are concatenated.
Go to primary source ↓subset-protocolPDF p. 7, Table 3; PDF p. 8, Section 4.3, first paragraph
The 10-class task merges 41 selected original classes and contains 28,198 videos. The 40-class task retains those groups and adds 30 common classes, totaling 53,267 samples. Exact grouping and prediction examples are referred to an appendix absent from this PDF.
Go to primary source ↓baseline-tablePDF p. 8, Table 4, all method rows and 10/40/174-class error columns
Combined 2D+3D errors are 44.9/27.1 for 10 classes and 63.8/50.7 for 40. Pretrained 3D averages give 47.5/29.2 and 66.2/52.7; pretrained 2D+LSTM gives 52.3/34.1 and 77.8/68.0. Scratch 2D averages give 76.5/58.9 and 88.0/78.5. Only pretrained 3D averages have full-class entries: 88.5/81.5/70.0 for top-1/2/5. No uncertainty or validation/test designation appears in the table.
Go to primary source ↓full-class-and-humanPDF p. 8, Section 4.3, final two paragraphs
Prose describes full-class training from a 40-class-pretrained 3D CNN and reports 88.5% top-1 and 70.3% top-5 error, conflicting with Table 4's 70.0. An informal study has ten people classify approximately 700 test samples in total at approximately 60% accuracy.
Go to primary source ↓discussionPDF p. 8, Section 5
The work is positioned as learning through observation with supervised fine-grained labels; robotics and video-based visuomotor policies motivate the broader problem. Dataset expansion is an ongoing plan, not an evaluated control system.
Go to primary source ↓release-and-supplementPDF p. 2, footnote 1; PDF p. 6, end of Section 3.4; PDF p. 7, end of Section 3.5; PDF p. 8, Section 4.3
A footnote announces an available dataset version. The body refers action-group lists and platform details to supplementary material, and class mappings and predictions to an appendix. These materials and license terms are not included in the supplied paper.
Go to primary source ↓8.5 Primary sources
The "Something Something" Video Database for Learning and Evaluating Visual Common Sense ↗
PDF · 6,116 extracted words
Source fingerprint
593431b2c05ac8da3e41a7fdafac1c8e99e464d660d6013787726a4571d3089b