Auto-Encoding Variational Bayes
1. Paper overview
In one sentence: AEVB trains recognition and generation through a differentiable latent-sampling transformation, gaining efficient approximate inference while retaining a variational gap and a restricted experimental posterior family. e-identitye-probleme-bounde-sgvbe-vaee-fig2e-fig3e-samples
| At a glance | What to know |
|---|---|
| Research problem | Source description Nonlinear directed latent-variable models can make both the marginal likelihood and true posterior intractable. Repeating MCMC inference for each observation also becomes costly on large datasets. The target is efficient approximate ML/MAP learning of global parameters, posterior inference for continuous latent codes, and marginal inference for observations, using minibatches rather than full-dataset updates. e-problem |
| Core mechanism | |
| A key reported result | MNIST marginal likelihood with 50,000 training points: At about 60 million training observations processed, AEVB is approximately -138 (visual read-off, not a tabulated exact value). Estimated held-out marginal log likelihood; higher is better.. Figure 3 right: 100 hidden units per network and 3 latent variables. Appendix E evaluates the first 1,000 train and test observations, with 50 posterior samples per observation and 4 HMC leapfrog steps. Wake-sleep is approximately -141 at the same budget; MCEM is visibly lower than both. The larger-data panel favors AEVB under this low-dimensional evaluator. These plot estimates have no reported uncertainty interval and are distinct from Figure 2 lower bounds. e-fig3e-evaluation |
| Reading caution | Reader analysis The reported posterior family is diagonal Gaussian and the experiments concern two image datasets. Time-series models, hierarchical architectures and global-parameter SGVB experiments are future directions. No action-conditioned transition model, planner or executed control policy is evaluated. e-vaee-traininge-futuree-fullvb |
Core contributions
- Source description
AEVB shares a learned recognition model across observations and jointly optimizes it with the generator. The neural example combines a Gaussian latent posterior with a Bernoulli or Gaussian decoder; diagonal covariance is an example choice, not a requirement of the general estimator. e-graphe-vaee-mlp
Figure 1. Generation and recognition traverse the same latent-variable problem in opposite directions. Original paper, p. 2 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Begin at the unshaded z node and follow the solid downward arrow to shaded x: this is the generator's conditional observation model. The dashed curved arrow runs upward from x to z and represents approximate posterior inference, with recognition parameters phi. The plate marked N repeats the observation–latent pair over the dataset; theta and phi are shared outside it. The caption and Section 2.1 agree on these directions. Theta also points into z in this general graphical model, but Section 3's particular VAE fixes the latent prior to a standard Gaussian, so that example has no learned prior parameters. e-graphe-probleme-vaee-labelse-mlp
What it supports. The diagram separates the model of how data arise from the model used to infer their hidden causes. Jointly learning the recognition and generative parameters is central to AEVB's efficient inference. The dashed route approximates a posterior; it does not turn inference into an exact reversal of generation.
Where the evidence stops. Section 5 on p. 6 swaps the parenthetical encoder/decoder names. This guide follows Figure 1, the p. 3 definitions and Appendix C: recognition is the encoder, generation the decoder. The graphic specifies distributions rather than neural-layer details.
2. Motivation
2.1 The problem and the proposed response
Nonlinear directed latent-variable models can make both the marginal likelihood and true posterior intractable. Repeating MCMC inference for each observation also becomes costly on large datasets. The target is efficient approximate ML/MAP learning of global parameters, posterior inference for continuous latent codes, and marginal inference for observations, using minibatches rather than full-dataset updates. e-problem
2.2 What this reading follows
An image generator can readily map a latent code to an image distribution, yet inferring which code explains a particular image may be intractable. Kingma and Welling train a second network to approximate that inference, and connect both networks through one variational objective. The crucial operation is a change in how random codes are produced: parameter-independent noise passes through a differentiable transformation. This illustrated reading follows that mechanism into the learning curves and generated images, then separates improved lower-bound optimization from held-out likelihood and visual quality. It reviews the supplied arXiv v11 revision dated 10 December 2022. e-identitye-probleme-bounde-sgvbe-vaee-fig2e-fig3e-samples
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Foundational work |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
This table preserves the labels recorded at reading time. The current major category is Components of WAMs. View the current classification.
3.1 Evidence-based assessment
Supports the recorded classification
The foundational visual-representation classification is supported by the learned image encoder and generator. Architecture, prediction paradigm and world/action quadrant remain not applicable: the source learns static observation distributions with latent codes, not joint futures and actions or inverse dynamics. Joint encoder/decoder optimization does not establish a One Model world-action architecture. e-graphe-vaee-traininge-future
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Learn a reusable approximation to the hidden cause of an image
Start with the two directions in Figure 1. Generation begins with a continuous latent code and produces an observation distribution. Inference begins with the observed image and asks which latent codes plausibly explain it. Section 2 introduces a recognition distribution because the generator's true posterior cannot generally be evaluated. One shared recognition network predicts a distribution for each image, rather than requiring an independently optimized inference procedure for every image. In the neural example, the network produces Gaussian means and log variances; the decoder maps sampled codes to Bernoulli or Gaussian observation parameters. Both parameter sets are learned together. A useful reader interpretation is that the method invests training effort in making subsequent approximate inference cheap. The source does not claim that the recognition distribution becomes the exact posterior. e-probleme-graphe-vaee-mlp
Figure 4. A Gaussian-quantile grid turns the two-dimensional decoder into a visible image manifold. Original paper, p. 10 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each mosaic as a sweep through a learned two-dimensional latent space. The left model generates Frey Face images; the right generates MNIST digits. According to the caption, the authors start with linearly spaced coordinates on the unit square and transform them with the Gaussian inverse CDF to obtain latent coordinates. Each location is then passed through the learned generative model. This matters because equal steps in the displayed grid correspond to equal steps in Gaussian cumulative probability, not equal distances in latent z. Neighboring cells show how the decoder's image output changes over that constructed grid. e-manifoldse-vaee-graph
What it supports. Both models turn nearby grid locations into visibly related outputs, with face appearance and digit shape changing across the mosaics. This illustrates what can be inspected when the generator has only two latent coordinates. It gives a concrete view of the learned decoder, separate from the recognition network's posterior predictions for observed images.
Where the evidence stops. The panels contain generated outputs, not an encoded-data scatter plot or a temporal rollout. No quantitative smoothness or coverage test is reported. The caption does not specify the precise image statistic displayed for each conditional distribution.
5.2 Keep randomness while exposing a differentiable path
The recognition network must receive a learning signal through a sampled latent code. Section 2.3 achieves this by sampling auxiliary noise from a distribution that does not depend on recognition parameters and transforming that noise differentiably. In the Gaussian example, multiply standard-normal noise elementwise by the encoder's standard deviation and add its mean. This still samples the intended approximate posterior, but a fixed noise draw now leaves a differentiable path through the encoder. Equation (7) combines sampled reconstruction log likelihood with an analytic prior KL whenever possible; Appendix B supplies the Gaussian expression. Algorithm 1 estimates the full-data objective using a scaled minibatch and updates both networks. Its experimental choice of one latent draw per image and 100 images per minibatch is an empirical setting, not a universal guarantee that one draw suffices. e-sgvbe-vaee-kle-training
5.3 Ask which quantity each experiment actually improves
Figure 2 measures the objective being optimized: an estimated lower bound on image log likelihood. Equation (1) says that the remaining gap depends on the approximate posterior, so those curves alone cannot isolate improvements in the generator. Figure 3 instead estimates marginal likelihood with a separate posterior-sampling procedure. It changes the network size and restricts the latent space to three dimensions because the evaluator becomes unreliable at higher dimension. The larger training set favors AEVB, while the smaller set shows MCEM finishing above it on held-out likelihood. Figures 4 and 5 then answer a different question: what images do these learned generators produce? Reader analysis: objective optimization, estimated generalization and visual sample inspection provide complementary evidence, but none is a substitute for the others or evidence of action-conditioned dynamics. e-bounde-fig2e-fig3e-evaluatione-manifoldse-samplese-future
5.4 Training and inference
During training
The reported models have one tanh hidden layer per network, with 500 units for MNIST lower-bound experiments and 200 for Frey Face. The latent prior is standard Gaussian; the image decoder is Bernoulli for MNIST and Gaussian for Frey Face, whose output means use a sigmoid. All parameters are jointly updated with a Gaussian weight prior; no frozen module is described. e-vaee-mlpe-training
Experiments use minibatches M=100 and one latent sample L=1 per observation. Adagrad global stepsizes are selected from {0.01, 0.02, 0.1} using early training performance; initialization is reported as N(0, 0.01). Appendix F additionally derives variational inference over global parameters, but that extension was not experimentally evaluated here. e-traininge-fullvb
During inference
For an observed image, evaluate the recognition network and sample its approximate posterior without a separate iterative inference procedure. To generate an image, sample the latent prior and then the decoder distribution. MCMC is used in the likelihood evaluator and MCEM baseline, not required for AEVB recognition. e-probleme-graphe-vaee-evaluation
5.5 Implementation flow
- Define generation and recognition
The generator samples a latent z from its prior and an observation x conditionally on z. Recognition reverses the inferential direction: q_phi(z|x) approximates the posterior. Figure 1 distinguishes solid generative arrows from dashed recognition arrows; this is a probability model, not a temporal feedback loop. e-graphe-problem
- Optimize a tractable lower bound
The objective combines expected reconstruction log likelihood with a negative KL divergence from the approximate posterior to the prior. Its gap to log marginal likelihood is the KL divergence to the true posterior, so improving the bound can reflect changes in both generation and approximate inference. e-bound
- Differentiate through sampled codes
The encoder computes a mean and diagonal variance. Independent Gaussian noise is transformed into a posterior sample, which the decoder scores against the observed image. Gradients pass through that deterministic transformation to the encoder and decoder. The full-data bound is estimated by scaling the minibatch sum by N/M. e-sgvbe-vaee-mlp
6. Experiments & results
Auto-Encoding Variational Bayes learns a probabilistic image generator together with a recognition network that approximates its otherwise intractable latent posterior. A differentiable transformation of independent noise makes the variational objective trainable by stochastic gradients. The paper separates the general SGVB estimator from its AEVB learning algorithm and neural variational-autoencoder example. MNIST and Frey Face experiments support improved lower-bound optimization over wake-sleep, while estimated marginal likelihood reveals a more qualified small-data comparison with Monte Carlo EM. These are static-image density-modeling results, not action or dynamics experiments.
The supplied PDF contains five figures and no numerical results table. Its quantitative comparisons are learning curves in Figures 2–3, so this edition preserves those plots and identifies numerical read-offs as approximate. There is no isolated component ablation of reparameterization or analytic KL; the latent-dimension sweep in Figure 2 and random-sample comparison in Figure 5 provide diagnostics only. Figure 1 is a probabilistic graphical model, not a neural-layer architecture drawing; layer details come from the inspected method and Appendix C. All five original figures are included. e-visual-scopee-graphe-mlpe-fig2e-fig3e-samples
6.1 Read the original evidence
Figure 2. AEVB improves the variational bound across the displayed latent-dimension settings. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read one panel at a time before comparing latent dimensions. Red denotes AEVB and green wake-sleep; solid lines are training estimates and dashed lines are test estimates. The vertical coordinate is the average variational lower bound, so higher is better even when MNIST values are negative. The horizontal axis is logarithmic and counts training observations processed, including repeated visits. MNIST occupies the upper row with latent dimensions 3, 5, 10, 20 and 200; Frey Face occupies the lower row with 2, 5, 10 and 20. Section 5 specifies 500 hidden units per MNIST network and 200 for Frey Face. e-fig2e-traininge-bound
What it supports. AEVB's final plotted bounds exceed wake-sleep's in every displayed setting. The authors attribute the lack of increased overfitting with extra latent variables to the bound's regularization. That explanation is an author interpretation of the dimension sweep; these curves measure the objective and its train/test behavior, rather than independently measuring the causal effect of the KL term.
Where the evidence stops. The caption's omitted estimator variance below 1 is not uncertainty across training seeds. There is no KL-removal control. Bound improvement can reflect posterior approximation as well as generator quality, and processed observations are not a direct wall-clock comparison.
Figure 3. Estimated marginal likelihood reveals both AEVB's larger-data advantage and a small-data qualification. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. First choose the panel: the left trains on 1,000 observations and the right on 50,000. Both use three latent variables and 100 hidden units per encoder or decoder, unlike Figure 2's wider networks. Red, green and blue identify AEVB, wake-sleep and MCEM; dashed lines are held-out estimates and solid lines training estimates. Higher marginal log likelihood is better, and the two panels use different vertical ranges. Appendix E evaluates the first 1,000 observations from each train/test set with 50 posterior samples and four HMC leapfrog steps. Follow the dashed lines to assess generalization, rather than selecting the highest solid curve. e-fig3e-evaluatione-fig2
What it supports. In the larger-data panel near 60 million processed observations, AEVB's test curve is about -138 versus wake-sleep's about -141, with MCEM lower; these are approximate visual read-offs. In the smaller-data panel, MCEM finishes slightly above AEVB on held-out likelihood despite AEVB's rapid training improvement. The evidence therefore supports a conditional advantage, not an across-the-board ranking.
Where the evidence stops. The evaluator uses MCMC and a fitted auxiliary density and was considered unreliable at higher latent dimensions. No confidence intervals or exact endpoint table are supplied. These estimates must not be relabeled as the variational bounds from Figure 2.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| MNIST and Frey Face variational lower-bound learning Figure 2: MNIST latent dimensions 3, 5, 10, 20, 200 with 500 hidden units; Frey Face dimensions 2, 5, 10, 20 with 200 units. Train/test curves are shown, but exact split construction is not specified. | AEVB reaches higher final plotted bounds than wake-sleep across the nine displayed settings; exact endpoints are not tabulated. Estimated average variational lower bound per observation; higher is better. | Same recognition architecture for AEVB and wake-sleep. The horizontal axis counts training observations processed, not elapsed time. Supports improved bound optimization. The caption reports estimator variance below 1 and omits it; this is not a confidence interval over independent training runs. e-traininge-fig2 |
| MNIST marginal likelihood with 50,000 training points Figure 3 right: 100 hidden units per network and 3 latent variables. Appendix E evaluates the first 1,000 train and test observations, with 50 posterior samples per observation and 4 HMC leapfrog steps. | At about 60 million training observations processed, AEVB is approximately -138 (visual read-off, not a tabulated exact value). Estimated held-out marginal log likelihood; higher is better. | Wake-sleep is approximately -141 at the same budget; MCEM is visibly lower than both. The larger-data panel favors AEVB under this low-dimensional evaluator. These plot estimates have no reported uncertainty interval and are distinct from Figure 2 lower bounds. e-fig3e-evaluation |
| MNIST marginal likelihood with 1,000 training points Figure 3 left uses the same 100-hidden-unit, 3-latent configuration and Appendix E evaluator, with a smaller training set. | AEVB improves training likelihood quickly, but the final MCEM held-out curve lies above AEVB; exact endpoints are not tabulated. Estimated train and held-out marginal log likelihood. | Both AEVB and MCEM finish above wake-sleep on the held-out curve. The small-data panel does not support a universal held-out-likelihood win for AEVB; its large train/test separation also cautions against treating better optimization as better generalization. e-fig3e-evaluation |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 5. Random generations provide a qualitative diagnostic across latent dimensions. Original paper, p. 10 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Move from left to right through the four independently learned generative models: their latent spaces have 2, 5, 10 and 20 dimensions. The caption identifies the images as random samples, so a cell in one panel is not a matched counterpart of the cell in another. Inspect within-panel diversity, recognizability and ambiguous shapes, then compare these impressions with the train/test behavior in Figure 2. Figure 5 does not include scores, error bars or repeated-run summaries. Its role here is a dimension diagnostic; the source does not present it as an isolated test of the reparameterization or analytic KL calculation. e-samplese-fig2e-visual-scope
What it supports. The displayed models produce varied digit-like images at all four latent dimensions, alongside less recognizable outputs. This establishes a qualitative generation demonstration over several code sizes. It does not determine a best dimensionality or establish that a higher variational bound necessarily gives sharper or more diverse random images.
Where the evidence stops. Random sample grids do not provide a controlled image-quality ranking. Seeds, matched draws and sample-selection details are absent from the caption. The ablation section placement denotes a diagnostic comparison, not a component-removal experiment reported by the authors.
7. Analysis & limitations
7.1 What the evidence leaves open
The reported posterior family is diagonal Gaussian and the experiments concern two image datasets. Time-series models, hierarchical architectures and global-parameter SGVB experiments are future directions. No action-conditioned transition model, planner or executed control policy is evaluated. e-vaee-traininge-futuree-fullvb
Marginal-likelihood estimates become unreliable at higher latent dimension; the comparison therefore uses three dimensions. Appendix D requires low-dimensional sampling and sufficient samples, without a quantitative reliability study. Figure 2 cannot by itself separate generator improvement from a tighter variational approximation. e-bounde-fig3e-evaluation
Section 5 on p. 6 reverses the parenthetical encoder/decoder labels. This report follows Section 3 and Appendix C: recognition is the encoder and generation the decoder. Figure 5 is a qualitative dimension diagnostic, not an isolated mechanism ablation or a measured image-quality ranking. e-labelse-vaee-mlpe-samples
7.2 Questions for discussion
- How much of a higher lower bound comes from a better generator versus a tighter recognition approximation? [e-bound]
- Does the Figure 2 latent-dimension trend persist across specified splits and repeated random seeds? [e-fig2]
- How stable is Figure 3 when the posterior sampler and density estimator are made more accurate? [e-evaluation]
8. Reproducibility audit
8.1 Requirements and known gaps
A faithful comparison needs explicit dataset preprocessing and splits, the stated MLP likelihoods, shared architecture, MAP scaling, initialization and Adagrad selection. The PDF does not fully specify preprocessing, Figure 2 split sizes, random seeds or the annealing schedule. Its compute reference is approximately 20–40 minutes per million training observations on an Intel Xeon at an effective 40 GFLOPS, without an exact processor model or software version. e-mlpe-traininge-fig2e-evaluation
Rebuilding MCEM requires 10 HMC leapfrog steps, an automatically tuned stepsize targeting 90% acceptance, and 5 weight-update steps per acquired sample. Rebuilding evaluation additionally requires the separate density fit in Appendix D; its family and detailed MCMC initialization, burn-in and convergence diagnostics are unspecified. Report these choices explicitly before comparing Figure 3. e-evaluation
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Does integrating the Gaussian KL reduce gradient noise?
Reader-proposed check, not performed: use the paper's MNIST model with 100 hidden units and three latents. At identical parameter checkpoints and fixed 100-image minibatches, compare the generic SGVB estimator in Equation (6) against the analytic-KL estimator in Equation (7), keeping the likelihood, prior, parameterization and one latent draw per image fixed. Repeat noise draws to measure coordinatewise gradient variance and agreement of mean gradients; then compare convergence under equal update and time budgets across seeds. Lower variance without a systematic mean shift would support the proposed variance benefit. A mean discrepancy would instead prompt an implementation audit. e-sgvbe-vaee-kle-traininge-fig3
Check 2: Does the likelihood ranking survive a more stable evaluator?
Reader-proposed check, not performed: freeze matched AEVB, wake-sleep and MCEM checkpoints from the three-latent MNIST comparison, separately for 1,000 and 50,000 training observations. Begin with Appendix E's first-1,000-observation evaluation and 50 posterior samples, explicitly document the density family missing from Appendix D, and fit it on samples separate from those used in the reciprocal estimator. Increase chain length and sample count, repeat independent chains and density fits, and report uncertainty in pairwise held-out differences. Stable signs would support the plotted rankings; reversals or intervals spanning zero would show that evaluator uncertainty limits the comparison. e-fig3e-evaluation
8.3 Reading coverage
Visual audit: All four text chunks were read individually. All 14 PDF pages were rendered at 200 DPI and actually viewed, including the title/author stamp, method equations and Algorithms 1–2, all experiment plots and captions, references, and Appendices A–F. Every final crop of Figures 1–5 was separately viewed. Figure 1 arrow directions were checked against its caption and the method definitions; the p. 6 encoder/decoder label reversal is disclosed. Figure 3 dashed train/test identities, axis ranges and approximate values were checked visually. No tables exist in the supplied PDF. Crop captions were excluded while legends, axes and panel labels were retained. Separate supplements, external code and cited works remain outside this reading.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract and 1 Introduction (p. 1)
- 2 Method, including 2.1–2.4 (pp. 1–5)
- 3 Example: Variational Auto-Encoder (p. 5)
- 4 Related work (p. 6)
- 5 Experiments (pp. 6–8)
- 6 Conclusion and 7 Future work (p. 8)
- References (p. 9)
- A Visualisations (pp. 9–10)
- B Gaussian KL derivation (pp. 10–11)
- C MLP encoders and decoders, C.1–C.2 (p. 11)
- D Marginal likelihood estimator (pp. 11–12)
- E Monte Carlo EM (p. 12)
- F Full VB, including F.1 and Algorithm 2 (pp. 12–14)
Outside the original text pass
- Identity/version note: the title and both authors match the catalog. The supplied title page identifies arXiv:1312.6114v11 [stat.ML], 10 Dec 2022. This report reviews that revision; the catalog lists ICLR 2014. No original 2013/2014 artifact or revision history was supplied, so textual differences between editions are not established.
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- The extraction-only figure limitation was addressed by rendering and visually inspecting all 14 supplied PDF pages and all five final figure crops.
- No separate supplement was supplied. Code, linked resources and cited publications were not inspected; experiments were not reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e-identityPDF p. 1, title block and arXiv margin stamp
The title is Auto-Encoding Variational Bayes, by Diederik P. Kingma and Max Welling, both credited to Machine Learning Group, Universiteit van Amsterdam. The stamp reads arXiv:1312.6114v11 [stat.ML], 10 Dec 2022.
Go to primary source ↓e-problemPDF pp. 1–2, Abstract, Section 1, and Section 2.1
The problem is scalable inference and learning for directed models with continuous latent variables, intractable posteriors and large i.i.d. datasets; the text lists parameter, posterior and marginal inference goals.
Go to primary source ↓e-graphPDF p. 2, Figure 1 and caption; p. 3, opening encoder/decoder definitions
Solid arrows describe p_theta(z)p_theta(x|z); dashed arrows describe q_phi(z|x). The N plate repeats observations and latents; recognition and generative parameters are learned jointly.
Go to primary source ↓e-boundPDF p. 3, Section 2.2, Equations (1)–(3)
The log marginal equals the lower bound plus KL to the true posterior. The bound can be written as negative prior KL plus expected conditional log likelihood.
Go to primary source ↓e-sgvbPDF pp. 3–5, Sections 2.3–2.4, Equations (4)–(8), Algorithm 1
Reparameterization moves sampling to auxiliary noise. The generic and analytic-KL estimators support stochastic gradients; Equation (8) scales a minibatch sum by N/M. Algorithm 1 jointly updates theta and phi.
Go to primary source ↓e-vaePDF p. 5, Section 3, Equations (9)–(10) and footnote 2
The example uses a standard Gaussian prior, a diagonal Gaussian recognition distribution, Gaussian location-scale sampling, and a Bernoulli or Gaussian MLP decoder. Diagonal covariance is a simplifying choice.
Go to primary source ↓e-mlpPDF p. 11, Appendix C.1–C.2, Equations (11)–(12)
Single-hidden-layer MLPs use tanh. Bernoulli decoding uses sigmoid probabilities; Gaussian networks output means and log variances. For recognition, inputs and outputs swap and weights belong to phi.
Go to primary source ↓e-klPDF pp. 10–11, Appendix B, Gaussian KL derivation
The analytic negative KL sums one half of 1 + log sigma_j squared - mu_j squared - sigma_j squared over latent coordinates.
Go to primary source ↓e-trainingPDF pp. 6–7, Section 5, experiment setup and Likelihood lower bound paragraph
Training uses MNIST and Frey Face, equal encoder/decoder hidden widths, Gaussian Frey Face outputs with sigmoid means, a Gaussian weight prior, stated initialization and Adagrad choices, M=100 and L=1. Lower-bound experiments use 500/200 hidden units.
Go to primary source ↓e-labelsPDF p. 6, Section 5, second paragraph; contrast p. 3 definitions and p. 11 Appendix C
The experiments paragraph calls the generative model encoder and variational approximation decoder, reversing the roles defined elsewhere in the same PDF.
Go to primary source ↓e-fig2PDF p. 7, Figure 2, all panels, legend and caption; Section 5 lower-bound paragraph
The figure compares AEVB and wake-sleep train/test lower bounds over nine dataset/dimension settings. The caption claims faster convergence and better solutions, notes omitted estimator variance below 1, and reports 20–40 minutes per million observations on an Intel Xeon at effective 40 GFLOPS.
Go to primary source ↓e-fig3PDF p. 7, Section 5 Marginal likelihood paragraph; p. 8, Figure 3, both panels and legend
The likelihood comparison uses MNIST, 100 hidden units and 3 latents. Training sizes are 1,000 and 50,000. In the right panel at about 60 million processed observations, dashed AEVB and wake-sleep curves read roughly -138 and -141; these are visual approximations. MCEM is lower there, but its final test curve exceeds AEVB in the left panel.
Go to primary source ↓e-evaluationPDF pp. 11–12, Appendix D; p. 12, Appendix E
Likelihood evaluation fits a density to posterior samples and uses a fresh posterior sample set in a reciprocal estimator. The text restricts reliable use to low dimension. MCEM uses 10 leapfrog steps, 90% acceptance and 5 updates; evaluation uses the first 1,000 train/test points with 50 samples and 4 leapfrog steps. Density family and several sampler/schedule details are not supplied.
Go to primary source ↓e-manifoldsPDF p. 10, Figure 4, both panels and caption
The two-dimensional Frey Face and MNIST manifolds are generated by transforming a linearly spaced unit-square grid through the Gaussian inverse CDF and evaluating the learned decoder at those latent coordinates.
Go to primary source ↓e-samplesPDF p. 10, Figure 5, panels (a)–(d) and caption
Random samples from learned MNIST generative models are shown for 2-, 5-, 10- and 20-dimensional latent spaces, without an accompanying quantitative quality metric.
Go to primary source ↓e-futurePDF p. 8, Sections 6–7
The conclusion concerns continuous-latent inference and learning; future directions include hierarchical models, time series, global parameters and supervised latent-variable models.
Go to primary source ↓e-fullvbPDF p. 2, first body paragraph; pp. 12–14, Appendix F, Equations (13)–(24), Algorithm 2
The appendix extends variational inference to both global parameters and local latents through separate noise transformations and gives a Gaussian example. The main text explicitly leaves experiments with global-parameter variational inference to future work.
Go to primary source ↓e-visual-scopePDF pp. 7–8, Figures 2–3; pp. 9–14, Appendices A–F, Figures 4–5 and Algorithms 1–2 on pp. 4 and 14
The supplied paper reports quantitative comparisons as curves, plus manifold and random-sample images. It contains no numerical result table and no isolated analytic-KL or reparameterization-removal ablation; dimension sweeps are the available diagnostics.
Go to primary source ↓8.5 Primary sources
Auto-Encoding Variational Bayes ↗
PDF · 6,728 extracted words
Source fingerprint
bdcb63b79ed88f64f8ee2d43a721930d01dde5441a4dba20ecaa2342b34e017c