What is ELBO?

26 Mar, 2026

Just another chain-of-thought on understanding and remembering ELBO.

Let $𝐗$ be data samples $\{{\mathbf{x}}_{\mathbf{1}},{\mathbf{x}}_{\mathbf{2}},...,{\mathbf{x}}_{\mathbf{N}}\}$, and $𝐙$ be the latent variables $\{{\mathbf{z}}_{\mathbf{1}},{\mathbf{z}}_{\mathbf{2}},...,{\mathbf{z}}_{\mathbf{N}}\}$. Consider a scenario wherein we wish to determine the posterior $p(𝐙|𝐗)$. Now, as per Bayes' theorem:

However, $p(𝐗)$ (aka the evidence) is intractable. One alternative is to look at variational inference to compute a proxy distribution $q(𝐙)$ that is close enough to the posterior $p(𝐙|𝐗)$. One appropriate measure of closeness here would be the Kullback-Leibler divergence, $D_{KL}$:

Kullback-Leibler (KL) divergence

Now, minimizing the KL divergence would look like this:

Enter, ELBO

We are back to square one: $p(𝐗)$ is intractable. However, there is one thing we can do: we can compute KL divergence up to a constant. To do this, let us define the evidence lower bound (ELBO) as the remaining terms in $(1)$:

Note the reversal of the terms in ELBO between $(1)$ and $(2)$.

Now, $(1)$ becomes:

In $(3)$, $\text{log}p(𝐗)$ is a constant, and $D_{KL}$ is non-negative. This essentially means:

Now for the grand reveal: the log of the evidence, $p(\mathbf{\text{X}})$ is lower bounded by $\text{ELBO}(q)$. What this means is that maximizing $\text{ELBO}(q)$ minimizes $D_{KL}$!