LDA variational inference Note, LDA parametric solution

1.LDA Theme Model

Given a priori probability parameter αβ, the subject mixed parameter θ, set subject z, the joint distribution of the set word w is

(1)

2.variational Inference

1>variational distribution

Variational Inference algorithm Introduction to the variational distribution:

(3)

is substituted as a posteriori probability p (θ, z, w |α,β). The parameters γ and φ of the variational distribution are obtained by solving the optimization process.

2> a log likelihood function of document, using Jensen inequality

(4)

Thus we see this Jensen ' s inequality provides us with a lower bound on the log likelihood for an arbitrary variational dis Tribution Q (θ,z |γ,φ)

The right end of the formula is represented by L (γ,φ;α,β), which introduces the γ,φ parameter, the vairitional distribution and the true posterior distribution deviation,

Is KL divergence:

(5)

Maximizes the lower bound of L (γ,φ;α,β), which is equivalent to the KL divergence that minimizes the posterior probability of a variable and the true posterior probability. Substituting (4) (1) (3) to obtain

(6)

Γ,α,β all parameters are obtained by using the log likelihood function to obtain the partial derivative

Reference: David Blei

Word version download

LDA variational inference Note, LDA parametric solution