Variational Inference Basics (II)

Recap of the Problem Setting

Here is a paragraph from blog of Brian Keng describing the setting of Variational Bayesian Inference:

We’re trying to perform Bayesian inference, which basically means given a model, we’re trying to find distributions for the unobserved variables (either parameters or latent variables since they’re treated the same). This problem usually involves hard-to-solve integrals with no analytical solution.

What we need is to find a distribution $Q$ to approximate the true posterior $P$. This problem starts with the goal to find the $Q$ that could minimize $\mathbb{KL}(q(Z), P(Z|X))$. By assuming each dimension of $Z$ to be independent of each other, we could simplify the optimization problem of minimizing $\mathbb{KL}(q(Z), P(Z|X))$ to be minimizing $\mathbb{KL}(\mathbb{E}_{i\neq j}[ln(P(X, Z))]||q_i(z_i))$.