Notes on the what and why of 'variational' in VAEs

12 Feb, 2026

Most often, I try to get into the etymology of words and terminologies as a first principles way of understanding and remembering concepts. When I first learned about VAEs, I found it difficult to connect the meaning of variational to the how of it in VAEs. In this post, I construct an overly simplified chain-of-thought that I can revisit if I ever find myself lost in the future again.

Autoencoders

We know that an autoencoder (AE) is a neural network that reduces the dimensionality of the input via an encoding function, and reconstructs this reduced-dimensionality latent via a decoding function. AEs are most popularly U-Nets since the architecture's encoding, bottleneck, and decoding branches lend themselves well to the AE paradigm.

Variational Autoencoders

Variational Autoencoders (VAEs) are a class of deep generative models that approximate a posterior distribution as a continuous latent variable. If we were to impose constraints on the latents that an AE learns, we would get a Variational Autoencoder (VAE). The most common constraint is to force the latent onto a multivariate Gaussian distribution. Critically, this enables VAEs to be generative by sampling the multivariate Gaussian to obtain a latent which is then fed to the decoder.

So what is variational in VAEs?

To understand this, we need to walk back a few steps. Let us start with (1) what inferencing means, before taking a look at (2) Bayesian inferencing. This will finally help make the jump to (3) variational Bayesian inferencing.

1. What is inferencing?

Inferential statistics is the analysis of data to infer properties about the underlying data distribution. This is in contrast to descriptive statistics, which is concerned with (descriptive) properties of data (e.g., mean), with the aim to summarize data.

2. What is Bayesian inferencing?

It is a method of statistical inferencing that involves the Bayes' theorem. Recall that the Bayes' theorem allows us to calculate the probability of a posterior given a prior:

P (A | B) = \frac{P (B | A) \times P (A)}{P (B)}

Computing the posterior is essentially inferring about the underlying distribution. However, computing $P (B)$ is sometimes infeasible if intractable integrals arise. Enter, variational Bayesian inferencing.

3. What is variational Bayesian inferencing?

The word variational means "of, or related to optimization", and therefore the intractable integrals are approximated via an optimization task in variational Bayesian inferencing. An example of such an optimization task is by minimizing the Kullback-Leibler (KL) divergence of a family of distributions to the true posterior.

Back to square one, what is variational in VAEs?

And finally: the variational in VAEs means that the true distribution of the posterior (e.g., a multivariate Gaussian) is approximated via an optimization task. As I already mentioned, the generative capability stems from being able to sample a latent variable and decoding it to obtain a new sample.