Neural Discrete Representation Learning
1. Paper overview
In one sentence: VQ-VAE makes discrete representations trainable by separating nearest-neighbor encoding, gradient routing and prior learning, trading exact reconstruction for a compact space that can support structured generation. e02e03e04e05e09e10e13
| At a glance | What to know |
|---|---|
| Research problem | Source description Powerful autoregressive decoders can ignore a VAE's latent variables, undermining representation learning even when likelihood is good. The paper seeks discrete codes that retain useful structure while remaining trainable without high-variance discrete sampling estimators. Its objective is representation and generation, with predictive video as one application. e02e03e13 |
| Core mechanism | |
| A key reported result | CIFAR10 likelihood comparison: VQ-VAE: 4.67 bits/dimension Negative log-likelihood bound in bits/dimension; lower is better. Shared standard VAE architecture with latent capacity varied; independent Gaussian or categorical priors; evaluated after 250,000 steps. The evaluation split and winning capacity configurations are not stated. Continuous VAE: 4.51; VIMCO: 5.14. All reported likelihoods are lower bounds. VQ-VAE approaches but does not equal the continuous baseline, and improves on VIMCO in this experiment. No uncertainty estimates or capacity-by-capacity results are provided. e06 |
| Reading caution |
Core contributions
- Source description
A separately trained PixelCNN or WaveNet prior models dependencies among compressed codes; generation then decodes sampled codes into observations. e05
Figure 1. Discrete lookup in the forward pass, copied reconstruction gradients in the backward pass. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the blue path from the input dog image to the encoder output z_e(x), then through the index grid z and the selected embeddings z_q(x) to the decoder. The upper table contains the shared dictionary entries. Although the middle label is q(z|x), Equations (1)–(2) define a deterministic nearest-neighbor choice, not a soft mixture. The red curved arrow points backward from the decoder-input side to the encoder-output side: it transports the reconstruction gradient across lookup. The right panel places the continuous encoder output near e2 and illustrates how an encoder update can change a later assignment. e03e04
What it supports. The mechanism separates the representation used for decoding from the path used to train the encoder. The forward pass always uses a dictionary entry, while the copied gradient supplies an approximate learning signal. Dictionary and commitment losses then align the two sides of that bottleneck.
Where the evidence stops. Figure 1 omits the stop-gradient loss terms; check Equation (3) for their destinations. That equation prints positive log likelihood alongside positive penalties, leaving a minimization-sign ambiguity. The report preserves this rather than silently rewriting the source.
2. Motivation
2.1 The problem and the proposed response
Powerful autoregressive decoders can ignore a VAE's latent variables, undermining representation learning even when likelihood is good. The paper seeks discrete codes that retain useful structure while remaining trainable without high-variance discrete sampling estimators. Its objective is representation and generation, with predictive video as one application. e02e03e13
2.2 What this reading follows
The central question is what a compressed code must retain when a decoder can generate much of the detail itself. VQ-VAE answers with a learned dictionary: an encoder chooses entries, a decoder reconstructs observations from those entries, and a later autoregressive model learns how code combinations occur. Follow the architecture first, then distinguish reconstruction pairs from newly sampled images. The tiny global scene code and speech waveforms probe what the bottleneck preserves; the final video strip shows prediction under supplied actions. These examples motivate discrete representations, while the paper's sparse quantitative evaluation leaves latent dependence, speech invariance and control usefulness only partly tested. e02e03e04e05e09e10e13
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Foundational work |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
This table preserves the labels recorded at reading time. The current major category is Components of WAMs. View the current classification.
3.1 Evidence-based assessment
Supports the recorded classification
The recorded Foundational work / Visual encoders & representations placement is supported. The encoder–codebook–decoder and separately fitted prior are representation/generation components. The video experiment consumes actions and predicts observations; it neither predicts actions jointly nor implements inverse dynamics or action execution. Architecture, prediction paradigm and quadrant remain Not applicable for this catalog's world-action taxonomy. e03e05e13
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Why a hard lookup needs three learning signals
Start with an encoder output that lies between dictionary entries. Forward computation chooses only its nearest entry, and the decoder must reconstruct from that quantized vector. Since the index changes discontinuously, the paper copies the decoder-input gradient directly to the continuous encoder output. That signal alone does not update the dictionary under the stated gradient routing. The dictionary term therefore pulls the chosen entry toward the stopped encoder output, while the commitment term pulls the encoder toward the stopped entry. Stop-gradient is what distinguishes these apparently similar distances: it selects which side moves. The decoder receives reconstruction learning only. Equation (3) prints the reconstruction term as positive log likelihood; preserve that notation issue when translating the explanation into a minimized implementation loss. e03e04
5.2 Why reconstruction is only the first training stage
A collection of good reconstructions does not yet specify a generative distribution over entire code sequences or grids. During autoencoder training, VQ-VAE uses a uniform prior; for its deterministic posterior, the KL contribution is constant with respect to encoder parameters. The paper then fits a separate autoregressive prior to the learned discrete representations. A PixelCNN can model image indices, and a WaveNet can model speech codes. At generation time, that prior proposes the codes and the decoder maps their embeddings into observations. This explains why Figure 2 and Figure 3 answer different questions: one tests retaining information from an input, the other tests sampling structured combinations. Appendix A.1 offers EMA dictionary updates, but Section 3.2 says the reported experiments used the dictionary-loss route. e03e04e05e07e08e14
5.3 How far the video example reaches toward a world model
The video demonstration provides a useful connection to predictive world modeling: context frames become a latent representation, given actions condition a future sequence, and decoding produces visible predictions. The source specifically says pixel generation can wait until the latent sequence has been generated. A reader can infer that the compressed space is useful as a substrate for imagination, but useful control requires further evidence. There is no reported mechanism here for choosing actions, optimizing rewards, or correcting a rollout with newly observed environmental feedback. Even action dependence is not isolated by the displayed comparison, because the two rows begin with different scenes and action-free results are omitted. This is why the catalog's foundational representation classification remains appropriate without assigning an action-model quadrant. e03e05e13
5.4 Training and inference
During training
Experiments use commitment weight beta = 0.25. The authors report little change across 0.1–2.0, without a plotted sweep. Dictionary and commitment terms are averaged over latent positions. Appendix A.1 gives an alternative EMA dictionary update, with decay 0.99; Section 3.2 explicitly says EMA was not used for the reported experiments. e04e14
The CIFAR10 encoder has two stride-2, 4-by-4 convolutions and two residual blocks, with 256 hidden units. The decoder reverses this pattern using residual blocks and transposed convolutions. Adam uses learning rate 0.0002, batch size 128 and 250,000 steps; VIMCO uses 50 training samples. e06
VCTK speech training uses 109 speakers, six stride-2 encoder convolutions, 64-fold temporal downsampling and 512 discrete values; the decoder also receives speaker identity. A separate 460-speaker speech-generation experiment uses 128-fold downsampling and trains the prior on 40,960 waveform timesteps, or 320 latent timesteps. e10e11
During inference
Reconstruction encodes a supplied observation, selects embeddings and decodes them. Unconditional generation instead samples codes ancestrally from the fitted prior. Speaker conversion keeps the extracted codes and changes the decoder's speaker identity; content preservation is an author-reported listening result. e05e10
For video, supplied context and actions condition generation in latent space. Frames are decoded deterministically after the latent sequence has been generated, so image synthesis need not recur inside the prior rollout. No action optimizer, reward objective or environmental feedback loop is described. e13
5.5 Implementation flow
- Encode and quantize
The encoder maps an observation x to z_e(x). At each latent position, nearest-neighbor lookup selects one of K embeddings of dimension D. The posterior is one-hot, and the decoder receives the selected vector z_q(x), not a continuous mixture. Speech, images and video use one-, two- and three-dimensional latent feature spaces. e03
- Separate the gradient destinations
Copy the decoder-input reconstruction gradient to the encoder output across the nondifferentiable lookup. A dictionary loss moves the selected embedding toward the stopped encoder output; a commitment loss moves the encoder toward the stopped embedding. Reconstruction updates encoder and decoder, while the middle term updates embeddings. e04
- Learn a distribution over codes
The first stage uses a uniform prior, making the deterministic posterior's KL term constant. After representation training, fit a categorical autoregressive prior over codes. Image priors use PixelCNN; audio priors use WaveNet. Joint prior/autoencoder training is left to future work. e03e05
6. Experiments & results
VQ-VAE learns a discrete codebook by quantizing encoder outputs, training reconstruction with a straight-through gradient, and fitting an autoregressive prior afterward. It supports compact image and speech representations and action-conditioned latent video generation. Its evidence combines likelihood bounds, a phoneme probe and qualitative examples; it does not evaluate action selection or executed control.
The complete supplied PDF has seven figures but no quantitative results tables. CIFAR10 likelihood and phoneme accuracy are reported in prose on pages 5 and 8, which were visually inspected. No controlled ablation table or plot is supplied: commitment-weight robustness is a prose statement, and action-free video results are explicitly omitted for space. Figure 5 is included as a qualitative latent-use diagnostic, with that boundary stated. Six original crops cover the mechanism, reconstruction, generation, scene diagnostic, speech and video; no substitute table or chart has been created. e15e06e12e04e13
6.1 Read the original evidence
Figure 2. A discrete image grid retains recognizable content while losing fine detail. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Compare corresponding positions across the two four-column blocks, rather than treating neighboring images as a time sequence. The left block contains originals; the right block decodes their inferred codes. The paper's setup maps each 128-by-128 RGB image to a 32-by-32 grid of indices, each choosing among 512 dictionary entries. Examine small contours and textures, such as the binocular edges and the dog's fur, alongside the preserved object arrangement. These are reconstruction pairs, so the decoder is receiving information extracted from each particular input. They test the bottleneck's fidelity, not the learned prior's ability to invent an image. e07e06
What it supports. The authors calculate about 42.6-fold nominal bit reduction by comparing 8-bit RGB pixels with 9-bit indices. The examples show recognizable structures despite blur. This supports the feasibility of a compact representation, while the bit calculation describes code capacity rather than measured compressed file size.
Where the evidence stops. No reconstruction score, codec comparison or uncertainty accompanies these pairs. The separate CIFAR10 likelihood result on this page is 4.67 bits/dimension for VQ-VAE versus 4.51 for the continuous VAE and 5.14 for VIMCO; it is a different dataset and evaluation.
Figure 3. A learned prior generates code grids that the image decoder turns into samples. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read down each column to compare generated examples associated with a category named in the original caption: kit fox, gray whale, brown bear, admiral butterfly, coral reef, alp, microwave and pickup. There is no original-image panel here. Section 3.3 explains the sequence: first train the VQ-VAE representation, then fit a PixelCNN distribution over the discrete grid, sample indices autoregressively, and decode the selected embeddings. Section 4.2 notes that the one-channel code grid requires spatial masking in this prior. Compare broad layout and object-like structure, but also inspect distorted contours instead of equating category resemblance with complete realism. e05e07e08
What it supports. These samples provide qualitative evidence that the fitted prior can organize discrete codes into recognizable visual structures. The representation bottleneck alone does not specify which complete grids are likely; prior learning supplies that distribution. The division between reconstruction and sampling is therefore central to interpreting Figures 2 and 3.
Where the evidence stops. The figure provides neither a numerical generation-quality metric nor a sample-selection protocol. The neighboring DeepMind Lab Figure 4 also has an unresolved size discrepancy: its caption says 128-by-128, while the surrounding experimental text specifies 84-by-84 inputs.
Figure 6. Speech reconstruction can change the waveform; linguistic invariance requires separate evidence. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the panels left to right using Figure 6's caption: original waveform, reconstruction with the same speaker identity, and reconstruction with a different identity. The axes have numerical ticks but no printed unit labels, and the vertical ranges differ, so avoid inferring a calibrated loudness comparison. The described VCTK encoder downsamples the waveform by 64 and uses 512 discrete values; the decoder receives both codes and speaker identity. For conversion, the encoded sequence is retained while the identity input changes. Visible waveform differences show that the reconstructions do not simply copy the original sample sequence. e10e12
What it supports. The authors report preserved speech content and changed voice characteristics, but the plots alone establish only waveform differences. The separate phoneme experiment on page 8 provides a numerical association: a simple mapping from 128 code values to 41 phonemes attains 49.3% accuracy against a 7.2% most-likely-phoneme baseline.
Where the evidence stops. The linked audio was not supplied for listening, so perceptual conversion quality remains unverified in this reading. Section 4.3's reconstruction paragraph mistakenly cites Figure 7; the waveform comparison is Figure 6. The phoneme probe uses a different downsampling configuration.
Figure 7. Action-conditioned video is generated in latent space before frames are decoded. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Locate the vertical divider and the Provided/Generated labels. The six frames to its left are the conditioning prefix; the ten frames to its right are predictions. The upper row uses repeated move-forward actions and the lower row repeated move-right actions, as specified by the caption. Their initial scenes differ, so these rows are not a same-context intervention. Section 4.4 says the sequence is generated in latent space first, with each frame decoded afterward using a deterministic decoder. The arrows above the strip mark the provided and generated regions; they do not depict an environmental feedback or action-selection loop. e13
What it supports. The source demonstrates a way to roll out predicted observations under externally supplied actions without decoding pixels at every latent-generation step. In the shown examples, the authors report continued visual quality and local geometry. This supports a predictive application of the representation, with no measured planning or control success.
Where the evidence stops. The paper gives no quantitative action-sensitivity or geometry metric. It reports similar results from a model without actions but omits them for space. These strips therefore cannot establish how much the action input improves predictions or whether a controller could safely rely on them.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| CIFAR10 likelihood comparison Shared standard VAE architecture with latent capacity varied; independent Gaussian or categorical priors; evaluated after 250,000 steps. The evaluation split and winning capacity configurations are not stated. | VQ-VAE: 4.67 bits/dimension Negative log-likelihood bound in bits/dimension; lower is better | Continuous VAE: 4.51; VIMCO: 5.14. All reported likelihoods are lower bounds. VQ-VAE approaches but does not equal the continuous baseline, and improves on VIMCO in this experiment. No uncertainty estimates or capacity-by-capacity results are provided. e06 |
| ImageNet discrete reconstruction capacity 128-by-128 RGB images encoded into a 32-by-32 grid with K = 512; qualitative originals/reconstructions in Figure 2. Split unspecified. | Approximately 42.6-fold, calculated by the authors from 8-bit pixels and 9-bit indices Nominal representation bit reduction, with qualitative reconstruction | Original images versus their decoded representations; no codec baseline or reconstruction-quality score. This is a capacity calculation, not measured file compression. The displayed reconstructions retain recognizable content with blur; Figure 3 separately shows samples from the fitted prior. e07e08 |
| Phoneme correspondence of unsupervised speech codes 128 discrete values at 25 Hz, downsampling factor 640; map each value to its conditionally most likely one of 41 phonemes. Labels were not used for VQ-VAE training; mapping/evaluation split unspecified. | 49.3% 41-way phoneme classification accuracy | 7.2% for the most-likely-phoneme baseline, described as the random-latent comparison. Supports phoneme association under a simple post-hoc mapping. It is not end-to-end speech recognition, and the evaluation description does not establish held-out mapping generalization. e12 |
| Action-conditioned DeepMind Lab video generation Six observed frames followed by ten generated frames; repeated move-forward or move-right actions in Figure 7. | Authors report preserved local geometry and visual quality over the illustrated continuation. Qualitative rollout appearance; no numerical dynamics or control metric | An action-free model reportedly gave similar results, but those samples and quantitative comparisons are omitted. Demonstrates latent video generation under supplied actions. Different contexts in the two rows and the absent action-free comparison prevent a controlled assessment of action sensitivity. e13 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 5. A 27-bit global code provides a qualitative test of scene information retention. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Match each top-row scene to the image immediately below it. Section 4.2 describes a second VQ-VAE operating on the first model's 21-by-21 latent grid. Its global representation contains only three categorical variables, each with 512 choices and its own embedding dictionary. The second-stage PixelCNN expands that global code into a sampled lower-level grid, and the first-stage decoder converts the grid into an 84-by-84 image. The caption calls the intermediate generator a second PixelCNN prior, while the body describes its role as the second VQ-VAE's decoder. Both place it between the global code and the first-stage latent grid. e09
What it supports. The three indices have a nominal capacity of 27 bits. The displayed pairs retain broad scene organization while regenerating textures and other details, which the authors interpret as meaningful use of the global latents despite a powerful PixelCNN. This is evidence about the illustrated reconstructions, not an exact scene encoding.
Where the evidence stops. This is a qualitative diagnostic, grouped here with ablation material, rather than a controlled ablation. There are no shuffled-code, zero-code or decoder-only rows. Such controls would help separate conditioning information from structure the decoder can generate on its own.
7. Analysis & limitations
7.1 What the evidence leaves open
The 27-bit scene reconstruction is a useful latent-use diagnostic, but no matched latent-shuffling or decoder-only control establishes how much information the global code contributes. The commitment-weight robustness statement also lacks numerical ablation results. e04e09
Image and video quality is mainly demonstrated through selected samples. Speech-content and speaker-conversion claims require listening evidence beyond waveform shape. The supplied paper reports neither a systematic control benchmark nor uncertainty for its numerical comparisons. e06e08e10e12e13
The source has presentation inconsistencies: Figure 4's caption says 128-by-128 although surrounding DeepMind Lab text specifies 84-by-84; the audio paragraph points to Figure 7 although the waveform is Figure 6. These are preserved as unresolved editorial discrepancies. e08e10
7.2 Questions for discussion
- How much of the global-code reconstruction survives when codes are shuffled across scenes?
- Would phoneme correspondence persist with the code-to-phoneme mapping fitted and tested on disjoint speakers?
- Does action conditioning improve held-out rollout fidelity when context, architecture and training data are controlled?
8. Reproducibility audit
8.1 Requirements and known gaps
Reproduction needs separate representation and prior training, the stated code grids, and exact reconstruction-loss scaling. Appendix EMA maintains moving counts and sums of assigned encoder outputs and divides sum by count; it is an alternative to, not evidence about, the reported dictionary-loss runs. e04e05e07e14
Missing details include exact evaluated CIFAR10 capacities and split, phoneme mapping/evaluation separation, full video-prior/action-conditioning configuration, random seeds, and hardware, runtime and software versions. Appendix A.1 supplies EMA equations, not these experimental specifications. e06e12e13e14
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Does the 27-bit code actually determine scene structure?
Reader-proposed check, not performed: reproduce the two-stage DeepMind Lab setup with three K=512 global indices and the same trained decoder. On held-out scenes, compare correct codes with codes shuffled across scenes and a fixed code, using matched sampling seeds and multiple decoder samples per condition. Predefine wall-layout agreement and reconstruction metrics, and report paired differences with uncertainty. If correct codes fail to outperform shuffled codes on scene-specific layout, that would weaken the claimed evidence of informative global latents. Keeping the decoder fixed isolates conditioning dependence more directly than comparing separately trained models. e09e04
Check 2: Does action conditioning improve a matched latent rollout?
Reader-proposed check, not performed: reserve DeepMind Lab trajectories with six-frame contexts and ten-frame futures. Match training data, latent codec, prior capacity and training budget between action-conditioned and action-free priors. For the conditioned prior, also compare true, shuffled and counterfactual action sequences from the same initial context. Evaluate generated frames against the held-out futures and, where the simulator setup permits, compare wall or camera geometry. True actions should improve prediction over shuffled actions and the action-free model; failure to do so would undermine an action-dependent dynamics interpretation. This tests prediction, not a reproduced control policy. e13e05
8.3 Reading coverage
Visual audit: All 11 PDF pages were rendered and visually inspected, covering title/authors/version, complete method and loss equations, all seven figures and captions, the prose numerical results, references and Appendix A.1. Figure 1's forward/backward arrows were cross-checked against Equations (1)–(3); Figure 7's supplied/generated divider was checked against Section 4.4. All six final original crops were separately viewed with labels and waveform ticks retained. The inspection preserves the loss-sign ambiguity, the Figure 4 size discrepancy and the audio paragraph's incorrect figure reference. Numerical evidence on uncropped pages 5 and 8 and EMA details on page 11 are included in the declared scope. No separate supplements or linked audio were inspected.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract
- 1 Introduction
- 2 Related Work
- 3 VQ-VAE
- 3.1 Discrete Latent variables
- 3.2 Learning
- 3.3 Prior
- 4 Experiments
- 4.1 Comparison with continuous variables
- 4.2 Images
- 4.3 Audio
- 4.4 Video
- 5 Conclusion
- References
- A Appendix
- A.1 VQ-VAE dictionary updates with Exponential Moving Averages
Outside the original text pass
- Identity/version note: the inspected title page reads Neural Discrete Representation Learning, by Aaron van den Oord, Oriol Vinyals and Koray Kavukcuoglu, all at DeepMind. It identifies arXiv:1711.00937v2 [cs.LG], 30 May 2018, and carries the NIPS 2017 conference footer. Title and authors match the catalog. This report reviews the supplied v2 artifact; v1 and a separate proceedings edition were not supplied, so their textual differences are unverified.
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- The extraction limitation was addressed by visually inspecting all 11 supplied PDF pages and all seven figures, including equations, captions and the appendix.
- Separate supplemental material availability has not been fully verified.
- Linked audio samples and external materials were not opened or listened to. Code was not inspected and no experiments were reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title/author block, arXiv margin and conference footer
Title and authors match the supplied identity: Aaron van den Oord, Oriol Vinyals and Koray Kavukcuoglu, each at DeepMind. The artifact is marked arXiv:1711.00937v2, 30 May 2018, with a NIPS 2017 footer.
Go to primary source ↓e02PDF pp. 1–2, Abstract and Section 1, contribution bullets
The authors motivate useful discrete representations and claim to address posterior collapse with powerful decoders, demonstrating images, speech and predictive video.
Go to primary source ↓e03PDF p. 3, Sections 3–3.1, Equations (1)–(2)
The encoder output selects the nearest of K D-dimensional embeddings. The categorical posterior is deterministic one-hot; a uniform prior gives constant log-K KL divergence. Modalities use 1D, 2D and 3D latent feature spaces.
Go to primary source ↓e04PDF pp. 3–4, Section 3.2, Figure 1 and Equation (3)
Straight-through gradients connect decoder input to encoder output. Stop-gradient separates dictionary and commitment updates. The printed loss uses positive log likelihood; beta is 0.25, with claimed robustness from 0.1 to 2.0. Terms average over latent positions. EMA is explicitly not used in the experiments.
Go to primary source ↓e05PDF pp. 4–5, likelihood discussion and Section 3.3
The first-stage prior is fixed uniform; an autoregressive categorical prior is fitted afterward, using PixelCNN for images and WaveNet for audio. Ancestral sampling enables generation. Joint training is future work; likelihood evaluation uses the encoded latent configuration as an approximation/bound.
Go to primary source ↓e06PDF p. 5, Section 4.1, architecture/training paragraph and numerical comparison
CIFAR10 uses a shared architecture while varying latent capacity, 256 hidden units, Adam 2e-4, 250,000 steps and batch 128; VIMCO uses 50 samples. VAE/VQ-VAE/VIMCO report 4.51/4.67/5.14 bits/dim; likelihoods are lower bounds. No evaluation split, per-configuration results or uncertainties are given.
Go to primary source ↓e07PDF pp. 5–6, Section 4.2, compression calculation, Figure 2 and reconstruction discussion
128-by-128 RGB ImageNet inputs map to 32-by-32 discrete indices with K=512, giving a nominal 42.6-fold bit reduction. The original/reconstruction pairs show blur. The section describes pixel MSE and leaves perceptual losses to future work.
Go to primary source ↓e08PDF p. 6, Section 4.2, Figures 3–4 and captions
PixelCNN uses spatial masking on a single-channel latent grid. Figure 3 shows ImageNet prior samples across eight named categories. DeepMind Lab prose uses 84-by-84 inputs and 21-by-21 codes, while Figure 4's caption labels samples 128-by-128.
Go to primary source ↓e09PDF pp. 6–7, Section 4.2 final paragraph and Figure 5/caption
A second VQ-VAE models first-stage 21-by-21 codes through a PixelCNN decoder, using three separate K=512 latents, or 27 bits. Sampled reconstructions retain scene properties but change details. The authors interpret this as meaningful latent use; no matched ablation is shown.
Go to primary source ↓e10PDF pp. 7–8, Section 4.3, VCTK setup, reconstruction/conversion paragraphs and Figure 6
VCTK has 109 speakers; six stride-2, size-4 encoder convolutions yield 64-fold downsampling with 512 discrete values. Decoder conditioning includes speaker identity. Figure 6 compares original, same-speaker and changed-speaker waveforms. Content preservation is an author listening claim; the body mistakenly refers to Figure 7.
Go to primary source ↓e11PDF p. 7, Section 4.3 final paragraph; p. 10, reference [30]
The separate unconditional speech experiment uses 460 speakers, 128-fold downsampling, and 40,960 waveform timesteps (2.56 seconds) yielding 320 latent timesteps. Authors report words and partial sentences in linked samples. Reference [30] identifies LibriSpeech.
Go to primary source ↓e12PDF p. 8, Section 4.3 final paragraph and footnote 1
A 128-value code at 25 Hz and downsampling factor 640 is mapped to 41 phonemes by conditional mode. Accuracy is 49.3% versus 7.2% for the prior most likely phoneme. Labels are excluded from VQ-VAE training; mapping/evaluation split is unspecified. The footnote notes contextual latent meanings.
Go to primary source ↓e13PDF p. 8, Section 4.4 and Figure 7/caption
Six context frames precede ten generated frames for repeated forward/right actions. Generation proceeds in latent space before deterministic image decoding. Quality/local geometry claims are qualitative. An action-free model is reported but omitted for space; no action selection or control evaluation is presented.
Go to primary source ↓e14PDF p. 11, Appendix A.1, Equations (4)–(8); p. 4, Section 3.2
EMA is derived from averaging encoder outputs assigned to each dictionary entry. Decayed counts and sums define updated embeddings; gamma=0.99 is said to work well. Section 3.2 says this alternative was not used in the reported experiments.
Go to primary source ↓e15PDF pp. 4–8, Figures 1–7 and Sections 3.2–4.4; p. 11, Appendix A.1
The supplied paper contains seven figures and no quantitative tables. CIFAR10 and phoneme numbers appear in prose. Figure 5 is a qualitative diagnostic; the beta sweep has no plotted values and action-free video results are explicitly not shown.
Go to primary source ↓8.5 Primary sources
Neural Discrete Representation Learning ↗
PDF · 5,704 extracted words
Source fingerprint
d62036803c190a82718b718f59a87adb2be8b586e4887a394ad0cef15c56301e