HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
1. Paper overview
In one sentence: Weak subtitle alignment at web-video scale can train a useful text–video embedding, but its transfer depends on negative sampling, domain and target adaptation. e-identitye-noisee-embeddinge-negativese-youcooke-lsmdc
| At a glance | What to know |
|---|---|
| Research problem | Source description Manual video descriptions are expensive to collect at the scale needed for broad language–vision correspondence. The paper asks whether existing narration can supply useful supervision despite inaccurate transcripts, irrelevant speech and imperfect temporal alignment. e-identitye-noise |
| Core mechanism | Source description The dataset spans 23,611 visual tasks and about 1.22 million videos. The abstract reports 136 million clips; Table 2 gives 136.6 million. Subtitle timestamps replace newly commissioned clip descriptions. e-scalee-collectione-noise |
| A key reported result | YouCook2 text-to-video clip retrieval: HowTo100M only: 6.1 / 17.3 / 24.8; 46. Pretrain then fine-tune: 8.2 / 24.5 / 35.3; 24. R@1 / R@5 / R@10 (%); median rank (lower is better). Approximately 3.5k validation clips; test descriptions unavailable. Overlapping validation videos removed from HowTo100M pretraining. Same model trained on YouCook2: 4.2 / 13.7 / 21.5; 65. HGLMM FV CCA: 4.6 / 14.3 / 21.6; 75. Fine-tuned R@10 exceeds HGLMM by 13.7 percentage points, not 13.7% relative. This is the authors' constructed retrieval protocol, not an official YouCook2 benchmark. e-evaluatione-youcook |
| Reading caution | Reader analysis Only 51% of 400 inspected pairs contain at least one visually present object or action mention. This is a permissive correspondence test, not sentence-level accuracy. ID deduplication leaves reuploads possible, and Table 2 shows strongly uneven domain coverage. e-noisee-collectione-scale |
Core contributions
- Source description
The dataset spans 23,611 visual tasks and about 1.22 million videos. The abstract reports 136 million clips; Table 2 gives 136.6 million. Subtitle timestamps replace newly commissioned clip descriptions. e-scalee-collectione-noise
Figure 1. Video and narration provide two views of a weakly paired training example. Original paper, p. 1 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow the two input arrows separately. The green path receives visual content; the blue path receives the subtitle text. Their black output arrows both enter the circle marked Joint Embedding, consistent with the two mappings in Section 4. The circle represents comparable vectors, not a generated video or an action command. The diagram is deliberately coarse: the actual implementation uses pre-extracted visual features, a shallow text CNN and gated mappings. Equations (1)–(3) on PDF page 4 define the cosine comparison and context gates that the schematic does not show. Those gates modulate feature dimensions before the two modalities are compared. e-identitye-embeddinge-featurese-noisee-evaluation
What it supports. The learning signal comes from treating narration and its video interval as a positive pair. A shared representation can then support a new text query without requiring that query to have been a training caption. This is a correspondence mechanism; temporal action localization is a downstream use of its similarity scores.
Where the evidence stops. Do not read the nearby dots as proof that any particular subtitle fully describes its clip. The paper's manual audit finds only partial object/action correspondence in many pairs, and the figure does not expose alignment errors or training losses.
2. Motivation
2.1 The problem and the proposed response
Manual video descriptions are expensive to collect at the scale needed for broad language–vision correspondence. The paper asks whether existing narration can supply useful supervision despite inaccurate transcripts, irrelevant speech and imperfect temporal alignment. e-identitye-noise
2.2 What this reading follows
A narrated tutorial already contains a rough description of what is happening on screen. HowTo100M turns that observation into a large pretraining dataset: each subtitle line becomes the description of its corresponding video interval. The central difficulty is that narration is only weakly aligned with visible content. This paper tests whether scale and a simple matching objective can make that supervision useful. Read the visuals as a chain from data composition to representation, then from training choices to evaluation. The strongest evidence concerns retrieval and localization in recorded videos. The movie results provide a useful counterweight to the broad transfer story. e-identitye-noisee-embeddinge-negativese-youcooke-lsmdc
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Datasets |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
3.1 Evidence-based assessment
Supports the recorded classification
The dataset classification is supported: narrated videos supply language supervision for pretraining and action understanding. 'Language annotation & reannotation' applies to automatic subtitle-based annotation; a separate relabeling method is not established. The WAM architecture, prediction-paradigm and quadrant axes are not applicable. Two modality encoders sharing an embedding do not establish a One Model world–action architecture or inverse-dynamics control. e-collectione-noisee-embeddinge-evaluation
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Turn narration into a training signal without assuming clean alignment
The acquisition pipeline begins with a task vocabulary, not a fixed list of action-segment labels. Semi-automatic filtering keeps visual activities, and task queries collect videos whose English subtitle lines carry timestamps. Each line defines both a text input and a candidate matching video interval. This makes the pipeline scalable, but inherits the speaker's timing and vocabulary. Someone may name an object before it appears, discuss an off-screen step or ask viewers to subscribe. The manual audit's 51% figure only requires one mentioned object or action to be visible. Read that statistic as a warning against treating every pair as a precise annotation. The resulting supervision still contains enough shared information for the model to learn useful matching functions, as tested by downstream retrieval and localization. e-collectione-noisee-youcooke-cross
Table 2. Large scale coexists with a highly uneven distribution across instructional domains. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each row across three distinct units: task names, source videos and extracted subtitle-aligned clips. Here k denotes thousands and M denotes millions. A task count measures the search vocabulary, while a clip count also depends on how much narration each source video contributes. Start with Food and Entertaining, then compare Home and Garden and Hobbies and Crafts with the much smaller categories below. The bottom row reports 136.6M clips; the abstract and Table 1 use 136M. Preserve these source presentations instead of treating the rounded category values as an independently exact census. e-scalee-collectione-noisee-youcook
What it supports. Food and Entertaining contributes 54.4M clips, compared with 29.5M for Home and Garden and 29.8M for Hobbies and Crafts. Computers and Electronics contributes 0.6M. The dataset covers many task names, but the evidence does not support an evenly balanced training distribution. This composition is relevant when interpreting strong transfer to cooking videos.
Where the evidence stops. The rows describe collection categories, not verified action labels or clean semantic coverage. Subtitle quality varies, and YouTube-ID deduplication can leave reuploads. This table alone cannot establish the number of distinct visual situations.
5.2 Use the negative pair to decide what matching must mean
A positive pair alone does not specify which aspect of a video should align with language. The two-direction ranking loss adds a comparison: the true pair should outrank a clip with the wrong caption and a caption with the wrong clip. Gated mappings learn this ordering over precomputed representations. If negatives always come from different videos, a shared kitchen background may help separate cooking from unrelated topics without distinguishing steps inside that kitchen. Mixing in negatives from the same original video removes some of that easy context. This is the authors' rationale, and the stronger gains on YouCook2 and CrossTask are consistent with it. As reader analysis, the ablation supports this explanation without proving it: it does not directly inspect attention or isolate background dependence, and MSR-VTT supplies a small counterexample. e-embeddinge-traininge-negatives
5.3 Separate pretraining benefit, target adaptation and action understanding
The evaluation story has three layers. First, an off-the-shelf embedding trained on HowTo100M can retrieve instructional clips and assign action-step descriptions to temporal positions. Second, fine-tuning on a target dataset tests whether the initialization remains useful when annotated examples are available. Third, comparison with other model families asks whether that particular system is competitive under the selected protocol. These layers need separate conclusions: YouCook2 improves even without adaptation, MSR-VTT benefits most after adaptation, and LSMDC retains a substantial gap to JSFusion. The paper's additional cross-dataset pretraining experiment favors HowTo100M on R@10 for all three targets, but does not make the architectures or training budgets identical across external baselines. As a reading judgment, these are representation-transfer results for observed videos; neither retrieval nor temporal localization constitutes learning to execute the depicted actions. e-evaluatione-youcooke-msre-lsmdce-transfere-embedding
5.4 Training and inference
During training
Use a bidirectional max-margin ranking objective with margin 0.1 and Adam at learning rate 10^{-4}. Half the negative pairs come from the same original video and half from other videos, intended to discourage reliance on shared background. The source reports under three days on one Tesla P100 after feature extraction; this excludes acquisition and feature-extraction cost. e-traininge-features
During inference
Rank candidate clips by similarity to a text query. For CrossTask, score frames against action-label descriptions and use an inference procedure similar to [68]. The supplied text does not fully specify that temporal assignment procedure. These outputs concern recorded-video understanding; no future-state prediction, policy rollout or executed robot control is evaluated. e-embeddinge-evaluation
5.5 Implementation flow
- Collect visual tasks and videos
Select physical activities from 12 WikiHow categories using semi-automatic verb filtering. Query YouTube with task names prefixed by 'how to'; retain the top 200 results with English subtitles, including uploaded, ASR-generated or translated text. Exclude videos with fewer than 100 views, fewer than 100 words, or duration above 2,000 seconds. Deduplicate by video ID. e-collection
- Build weak pairs
Pair each subtitle line with its timestamped interval. Lines need not be complete sentences, and speech can precede or follow the visible action. The collection therefore supplies correspondence hypotheses rather than clean action-segment annotations. e-noise
- Represent each modality
Temporally max-pool pre-extracted ImageNet ResNet-152 features sampled once per second and Kinetics ResNeXt-101 16-frame features sampled 1.5 times per second. Concatenate them into a 4,096-dimensional video vector. Remove English stop-words and apply a shallow 1D CNN over pretrained GoogleNews word2vec representations for text. e-featurese-embedding
- Learn gated mappings
Separate video and caption mappings project into a common 4,096-dimensional space. Each multiplies a linear projection by a sigmoid context gate. The paper reports 67 million parameters for these embedding mappings; the visual feature extractors are pretrained and pre-extracted, rather than jointly updated from pixels in this training procedure. e-embeddinge-features
6. Experiments & results
HowTo100M uses subtitle timing to turn narrated web videos into weakly paired clips and text. A shallow joint embedding learns correspondence at scale, supporting retrieval and action-step localization. Its strongest transfer is instructional; movie retrieval exposes the limits of off-the-shelf transfer.
6.1 Read the original evidence
Table 5. HowTo100M pretraining improves cooking-video retrieval, with further gains after target fine-tuning. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Begin with Trainset, because the three Ours rows answer different questions. YouCook2 means target-only training; HowTo100M means transfer without target training; the two-line PT/FT entry means pretraining followed by target fine-tuning. Higher R@1, R@5 and R@10 are better, while lower Median R is better. The surrounding evaluation section places these numbers on approximately 3.5k validation clips because test descriptions were unavailable, and removes overlapping validation videos from HowTo100M. Read the matched-model comparison before comparing against HGLMM, which the authors run using their features. The caption's PT and FT definitions are retained to keep the training conditions visible. e-youcooke-evaluatione-embedding
What it supports. R@10 increases from 21.5 with target-only training to 24.8 off the shelf, then to 35.3 after fine-tuning. The final model also lowers median rank from 65 to 24 compared with target-only training. Its R@10 advantage over HGLMM's 21.6 is 13.7 percentage points, a difference rather than a relative percentage gain.
Where the evidence stops. These are results for the paper's constructed YouCook2 retrieval protocol, not an official test benchmark. Better retrieval supports visual–language matching in cooking videos; it does not demonstrate execution of the retrieved cooking actions.
Table 7. Movie adaptation improves several matched-model metrics while leaving clear limits. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start at the bottom three Ours rows to isolate training-data effects within this model family. HowTo100M alone transfers poorly to movies; LSMDC-only training supplies domain-specific supervision; PT/FT combines them. Check every metric rather than inferring all-metric improvement from the bold R@10 value. The pretraining-plus-fine-tuning row improves R@5, R@10 and median rank relative to LSMDC-only training, while R@1 slips slightly. Then compare with JSFusion above the horizontal divider: it remains stronger across the displayed metrics. Section 5.2 specifies the official 1,000-pair movie test set, so these scores should not be combined numerically with YouCook2's different retrieval pool. e-lsmdce-evaluation
What it supports. Fine-tuning the pretrained model yields 27.9 R@10 versus 25.0 for LSMDC-only training, while median rank improves from 44 to 40. However, R@1 changes from 7.2 to 7.1, and JSFusion reaches 34.1 R@10. The supported conclusion is useful but incomplete transfer after adaptation, not state-of-the-art movie retrieval.
Where the evidence stops. The source attributes the weaker off-the-shelf result to the domain gap. These comparisons do not separately measure the contributions of narration style, movie appearance and dataset size, and no uncertainty estimates establish the reliability of the small R@1 difference.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| YouCook2 text-to-video clip retrieval Approximately 3.5k validation clips; test descriptions unavailable. Overlapping validation videos removed from HowTo100M pretraining. | HowTo100M only: 6.1 / 17.3 / 24.8; 46. Pretrain then fine-tune: 8.2 / 24.5 / 35.3; 24. R@1 / R@5 / R@10 (%); median rank (lower is better) | Same model trained on YouCook2: 4.2 / 13.7 / 21.5; 65. HGLMM FV CCA: 4.6 / 14.3 / 21.6; 75. Fine-tuned R@10 exceeds HGLMM by 13.7 percentage points, not 13.7% relative. This is the authors' constructed retrieval protocol, not an official YouCook2 benchmark. e-evaluatione-youcook |
| CrossTask action-step localization 18 tasks and 2.7k instructional videos; train on HowTo100M only, excluding CrossTask test videos. | 33.6 Average recall (%): correct step assignments divided by the number of steps, then averaged across tasks | Zhukov et al.: 22.4; Alayrac et al.: 13.3; the reported fully supervised reference: 31.6. The gain over Zhukov is 11.2 percentage points. The 'upper-bound' row is a particular supervised reference, not a mathematical ceiling; different training resources prevent a universal conclusion about supervision. e-evaluatione-cross |
| MSR-VTT text-to-video clip retrieval The clip-retrieval test set used by [63]; compare target-only training, HowTo100M-only training and pretraining followed by fine-tuning. | Pretrain then fine-tune: 14.9 / 40.2 / 52.8; 9. R@1 / R@5 / R@10 (%); median rank | Target-only: 12.1 / 35.0 / 48.0; 12. HowTo100M-only R@10: 29.6; JSFusion R@10: 43.2. Pretraining improves the matched model after adaptation; off-the-shelf transfer underperforms target-only training. e-evaluatione-msr |
| LSMDC text-to-video clip retrieval Official 1,000-pair movie retrieval test set; pretrain on HowTo100M and fine-tune on LSMDC. | 7.1 / 19.6 / 27.9; 40. R@1 / R@5 / R@10 (%); median rank | Target-only: 7.2 / 18.3 / 25.0; 44. HowTo100M-only R@10: 14.0. JSFusion: 9.1 / 21.2 / 34.1; 36. Adaptation helps R@5, R@10 and median rank versus target-only training, but slightly lowers R@1 and does not beat JSFusion. e-evaluatione-lsmdc |
| Negative-pair sampling ablation HowTo100M training; compare cross-video negatives alone with the half-intra-video mixture. | YouCook2 24.8; CrossTask 33.6; LSMDC 14.0; MSR-VTT 29.6. Retrieval R@10 (%) and CrossTask average recall (%) | Without intra-video negatives: 18.1; 25.7; 12.3; 30.1, respectively. The largest gains occur on instructional tasks; MSR-VTT falls by 0.5 percentage points. No uncertainty estimates are supplied. e-negativese-training |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 3. The same-video negative mixture helps instructional evaluations most, with an MSR-VTT exception. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use the retained caption as the column key: M is MSR-VTT, L is LSMDC, Y is YouCook2 and C is CrossTask. The first three columns report retrieval R@10; the last reports average localization recall, so compare rows within a column rather than averaging all four numbers. 'With intra-negative' refers to the mixture described on this page: half the negatives are drawn from the same original video and half from other videos. Same-video clips often share the scene, making background similarity less useful for distinguishing them. This is the authors' mechanism hypothesis; the table measures downstream performance, not what pixels the model attends to. e-negativese-traininge-evaluation
What it supports. YouCook2 rises from 18.1 to 24.8 R@10 and CrossTask from 25.7 to 33.6 average recall: gains of 6.7 and 7.9 percentage points. LSMDC also improves, from 12.3 to 14.0. MSR-VTT decreases from 30.1 to 29.6, so the evidence supports a task-dependent benefit rather than a universal improvement.
Where the evidence stops. No seed variation or confidence intervals accompany these rows. Better retrieval is consistent with reduced background shortcuts, but does not directly identify the learned mechanism; repeated actions within one video can also create ambiguous negatives.
Figure 3. The scale study increases both the number of videos and the allowed search-rank range. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow each colored curve from left to right, using the legend to keep CrossTask's average recall separate from retrieval R@10. The horizontal axis counts source videos, not clip-caption pairs; its spacing is nonuniform in raw video count. Section 5.4 defines the eight training sets by progressively admitting lower-ranked YouTube search results, from the top two results to the top 200. The smallest set has 15k videos and the largest is the full HowTo100M collection. The broad improvement is visible, but some neighboring points are flat or lower. Thus the figure should not be paraphrased as strict monotonic improvement at every increment. e-scalinge-collectione-evaluation
What it supports. All four evaluations finish above their small-data starting points. The result supports the usefulness of substantially increasing the available narrated-video training data within this collection procedure. The authors interpret the final trend as motivation to collect still more data, although the figure itself reports only the tested training sizes.
Where the evidence stops. Search rank and dataset size change together; relevance, domain composition and training exposure can therefore change too. This is not a compute-matched estimate of a pure scaling law, and the plot provides no uncertainty bars or tested predictions beyond its endpoint.
7. Analysis & limitations
7.1 What the evidence leaves open
Only 51% of 400 inspected pairs contain at least one visually present object or action mention. This is a permissive correspondence test, not sentence-level accuracy. ID deduplication leaves reuploads possible, and Table 2 shows strongly uneven domain coverage. e-noisee-collectione-scale
The scale study changes search-rank thresholds along with dataset size, confounding quantity with relevance and composition. The plotted trend is broadly upward with local dips; extrapolated gains beyond the observed range remain the authors' expectation. e-scaling
Tables 3–7 provide no seed variation or confidence intervals. Strong averages and curated retrieval examples do not establish uniform domain generalization or causal action understanding. e-negativese-crosse-youcooke-msre-lsmdce-transfer
7.2 Questions for discussion
- Would intra-video negatives still help after controlling for repeated actions and near-identical captions?
- How much of the scale gain survives fixed search-rank composition and fixed optimization compute?
8. Reproducibility audit
8.1 Requirements and known gaps
Reproduction requires the collected video IDs and timed text, pretrained visual and word features, the gated embedding, the negative-pair mixture and the exact evaluation lists. Preserve the stated CrossTask/YouCook2 overlap exclusions. Dataset, model and code release is claimed, but this PDF does not provide license terms. e-collectione-featurese-traininge-evaluatione-release
The supplied body omits batch size, total optimization steps, detailed text-CNN configuration and the full temporal inference algorithm; it refers loss implementation to an absent appendix. These are unresolved reproduction inputs, not settings to infer from the reported runtime. e-embeddinge-traininge-evaluatione-supplements
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Test whether same-video negatives reduce reliance on scene context
Reader-proposed, not executed: train matched embeddings using cross-video negatives alone versus the paper's half-intra-video mixture. Hold feature caches, training subset, batch construction, negative count, optimization updates and seeds fixed. Evaluate YouCook2 retrieval and CrossTask localization, then use an independently labeled diagnostic set of clips from the same scene that differ in the depicted step. Record repeated-action or equivalent-caption pairs separately to identify false negatives. The background-shortcut account predicts a larger benefit on within-scene step discrimination than on unrelated-scene retrieval. If only the latter improves, the proposed mechanism is weakened even if aggregate recall increases. e-traininge-negativese-featurese-evaluation
Check 2: Separate added data from search-rank composition in the scale study
Reader-proposed, not executed: reconstruct several of Section 5.4's rank-threshold subsets and compare each with an equal-sized sample from the full pool, stratified by task category. Run both fixed-update and fixed-exposure schedules with identical features and model settings; retain all documented evaluation-overlap exclusions. Measure the four reported evaluations and variation across seeds. If rank-selected and broadly sampled sets differ at the same size, Figure 3 combines quantity with selection effects. If improvement disappears only under fixed updates, extra training exposure is a competing explanation. Either observation would limit a pure data-scale interpretation without disputing the original plotted measurements. e-scalinge-collectione-scalee-traininge-featurese-evaluation
8.3 Reading coverage
Visual audit: All 11 supplied PDF pages were visually inspected, including the title/version/author block; Figures 1–6; Tables 1–7; the equations, training and evaluation details on pages 4–6; the uncropped CrossTask and MSR-VTT tables on page 7; and references on pages 9–11. All six final original crops were inspected. Figure 1's arrow directions agree with the two mappings in Equations (1)–(3); the schematic omits the gates and loss, which are explained from text. Short table captions were retained where they define abbreviations or training conditions. The separate appendices referenced on pages 4–5 were not supplied; their contents and other editions remain outside this pass.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11. Appendix coverage: not present.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract (PDF p. 1)
- 1. Introduction (PDF pp. 1–2)
- 2. Related work (PDF pp. 2–3)
- 3. The HowTo100M dataset; 3.1 Data collection; 3.2 Paired video clips and captions (PDF pp. 3–4)
- 4. Text-video joint embedding model (PDF pp. 4–5)
- 5. Experiments; 5.1 Implementation details; 5.2 Datasets and evaluation setups (PDF pp. 5–6)
- 5.3 Study of negative pair sampling strategy; 5.4 Scale matters (PDF pp. 5–6)
- 5.5 Comparison with state-of-the-art (PDF pp. 6–8)
- 5.6 Cross-dataset fine-tuning evaluation; 5.7 Qualitative results; 6. Conclusion; Acknowledgements (PDF p. 8)
- References (PDF pp. 9–11)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- The extraction limitation was resolved by inspecting all 11 PDF pages and all six final original crops.
- Separate supplemental material availability has not been fully verified.
- Appendices A–C are referenced through [33] but are absent from this supplied 11-page PDF. Their additional dataset statistics, loss implementation and positive-pair selection experiments were not supplied or read.
- Identity and edition: the title and all six authors match the catalog. PDF p. 1 identifies the CVF open-access ICCV version as identical to the accepted version except for its watermark, and distinguishes the final proceedings version. No separate revision number or revision date is established; no other edition was inspected.
- Code, the project site and dataset files were not inspected; no experiments were reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e-identityPDF p. 1 (printed p. 2630), title block, CVF watermark, Abstract and Figure 1
The observed title and author order match the catalog: Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev and Josef Sivic. Miech and Zhukov share equal-contribution credit; Alayrac has a 'Now at DeepMind' note. The watermark identifies the open-access accepted version. Figure 1 directs video and text branches into a joint embedding.
Go to primary source ↓e-scalePDF p. 1, Abstract; PDF p. 2, Table 1, HowTo100M row; PDF p. 4, Table 2, all categories and total row
The abstract reports 136M clips from 1.22M videos; Table 1 reports 136M clips, 1.221M videos and 134,472 hours. Table 2 totals 23.6k tasks, 1.22M videos and 136.6M clips. Food and Entertaining has 54.4M clips, Home and Garden 29.5M, Hobbies and Crafts 29.8M, and Computers and Electronics 0.6M.
Go to primary source ↓e-collectionPDF pp. 3–4 (printed pp. 2632–2633), Section 3.1
Semi-automatic physical-verb filtering within 12 WikiHow categories yields 23,611 tasks. Queries use 'how to', top-200 results and English subtitles. Minimums are 100 views and 100 words; maximum duration is 2,000 seconds. Deduplication uses YouTube IDs and may miss reuploads.
Go to primary source ↓e-noisePDF p. 4, Section 3.2, pairing and Statistics paragraphs
Subtitle lines define clip intervals. Noise includes unrelated speech, temporal mismatch and incomplete ASR text. A 400-pair sample has a visible object/action mention in 51%; a separate 100-video sample is 71% instructional, 12% vlogs and 7% reviews/advertisements. Mean clips last four seconds and captions contain four words after stop-word removal.
Go to primary source ↓e-embeddingPDF p. 4, Section 4, Equations (1)–(3); PDF p. 5, context gating and Clip and caption representation paragraphs
Separate gated mappings f and g learn cosine correspondence. Video, caption and common embedding dimensions are 4,096; the stated mapping size is 67M parameters. Visual features are pre-extracted; text features use a shallow 1D CNN over precomputed word embeddings.
Go to primary source ↓e-trainingPDF p. 5 (printed p. 2634), Section 4, Loss and Sampling strategy paragraphs
The displayed two-direction hinge ranking loss uses margin 0.1 and Adam with learning rate 10^{-4}. Half the negative pairs are intra-video. Authors motivate this by reducing background shortcuts, and report that attempted automatic positive-pair selection did not improve results, referring details to Appendix C.
Go to primary source ↓e-featuresPDF p. 5, Section 5.1, Video features, Text pre-processing and Training time
ResNet-152/ImageNet features are extracted at 1 Hz; ResNeXt-101/Kinetics 16-frame features at 1.5 Hz, max-pooled and concatenated. Text uses GoogleNews word2vec and stop-word removal. Embedding training is reported below three days on one Tesla P100 after feature extraction.
Go to primary source ↓e-evaluationPDF p. 5, Section 5.2 opening; PDF p. 6, Section 5.2
CrossTask uses 18 tasks and 2.7k videos, frame–label similarity and inference similar to [68]; recall measures correctly placed steps. CrossTask test and YouCook2 validation overlaps are removed from pretraining. YouCook2 retrieval uses approximately 3.5k validation clips; MSR-VTT uses [63]'s test set; LSMDC uses 1,000 official test pairs.
Go to primary source ↓e-negativesPDF p. 5, Table 3, both rows and M/L/Y/C columns; PDF p. 6, Section 5.3
Without versus with intra-video negatives: MSR-VTT R@10 30.1/29.6, LSMDC 12.3/14.0, YouCook2 18.1/24.8, and CrossTask average recall 25.7/33.6. No uncertainty is shown.
Go to primary source ↓e-scalingPDF p. 6, Figure 3 and Section 5.4
Search-rank thresholds 2, 3, 5, 10, 20, 40, 80 and 200 produce 15k, 28k, 52k, 104k, 197k, 364k, 648k and full-dataset training sets. Recall generally improves with size; the figure has local dips. The authors' prediction of further gains is extrapolative.
Go to primary source ↓e-crossPDF p. 7, Table 4, all rows, Average column and Make Meringue column; Section 5.5 CrossTask paragraph; PDF p. 6, Section 5.5
Average recalls are 31.6 for the supervised reference, 13.3 for Alayrac, 22.4 for Zhukov and 33.6 for HowTo100M. On Make Meringue, HowTo100M is 41.9 versus Zhukov's 43.0. The surrounding prose makes a stronger claim about absence of domain bias than these averages establish.
Go to primary source ↓e-youcookPDF p. 7, Table 5, all method rows and metric columns; Section 5.5 YouCook2 paragraph
HowTo100M-only R@10 is 24.8; target-only 21.5; pretraining plus fine-tuning 35.3; HGLMM 21.6. The final row is 8.2/24.5/35.3 with median rank 24. The authors construct this retrieval benchmark using their features.
Go to primary source ↓e-msrPDF p. 7, Table 6, JSFusion and all Ours rows; Section 5.5 MSR-VTT paragraph
Pretraining plus fine-tuning scores 14.9/40.2/52.8 and median rank 9; target-only 12.1/35.0/48.0 and 12; HowTo100M-only 7.5/21.2/29.6 and 38; JSFusion R@10 is 43.2.
Go to primary source ↓e-lsmdcPDF p. 8, Table 7, JSFusion and all Ours rows; Section 5.5 continuation
Fine-tuned scores are 7.1/19.6/27.9 and median rank 40; target-only 7.2/18.3/25.0 and 44; off-the-shelf 4.0/9.8/14.0 and 137; JSFusion 9.1/21.2/34.1 and 36.
Go to primary source ↓e-transferPDF p. 3, Figure 2; PDF p. 7, Section 5.5 MSR-VTT paragraph; PDF p. 8, Figures 4–6 and Sections 5.6–5.7
Figure 2 uses selected matched pairs arranged by linguistic similarity. Figure 4 compares MSR-VTT supervision fractions; the text claims comparable prior-state-of-the-art performance with 20% supervision. Figure 5 shows HowTo100M as the best tested pretraining source for target R@10. Figure 6 gives curated retrieval examples including cut paper versus cut wood.
Go to primary source ↓e-supplementsPDF p. 4, Section 3.2 appendix references; PDF p. 5, Section 4 appendix references; PDF p. 9, reference [33]
Additional dataset statistics are assigned to Appendix A, loss implementation to B, and positive-pair sampling to C of [33], the same-title 2019 preprint. These appendix contents are not in the supplied main-paper PDF.
Go to primary source ↓e-releasePDF p. 8, Section 6 Conclusion; PDF p. 9, reference [1]
The authors claim public dataset, pretrained-model and code availability at the project page. The supplied PDF does not specify a dataset license.
Go to primary source ↓8.5 Primary sources
HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips ↗
PDF · 7,961 extracted words
Source fingerprint
075a8a436b3c1a2aa795fc9e2b5c59b356fdbca58538f0f97c94b126ae17a54a