Topic Modeling
 tags
 Machine Learning
http://www.cs.columbia.edu/~blei/topicmodeling.html LDA survey  Github
LDA
The Little Book on LDA https://www.youtube.com/watch?v=FkckgwMHP2s http://www.cs.columbia.edu/~blei/papers/Blei2012.pdf
Dirichlet Distribution
https://www2.ee.washington.edu/techsite/papers/documents/UWEETR20100006.pdf
Dirichlet distribution is a family of continuous multivariate probability distributions parameterized by a vector α of positive reals.
\begin{equation} \theta \sim Dir(\alpha) \end{equation}
\begin{equation} p(\theta) = \frac{1}{\beta(\alpha)} \prod_{i=1}^n \theta_i^{\alpha_i1} I(\theta \in S) \end{equation}
Where \(\theta = (\theta_1, \theta_2, \dots, \theta_n), \alpha = (\alpha_1, \alpha_2, \dots, \alpha_n), \alpha_i > 0\) and
\begin{equation} S = \left\{x \in \mathbb{R}^n : x_i \ge 0, \sum_{i=1}^{n} x_i = 1 \right\} \end{equation}
and \(\frac{1}{\beta(\alpha)} = \frac{\Gamma(\alpha_0)}{\Gamma(\alpha_1)\Gamma(\alpha_2)\dots\Gamma(\alpha_n)}\)
The infinitedimensional generalization of the Dirichlet distribution is the Dirichlet process.
The Dirichlet distribution is the conjugate prior distribution of the categorical distribution (a generic discrete probability distribution with a given number of possible outcomes) and multinomial distribution (the distribution over observed counts of each possible category in a set of categorically distributed observations). This means that if a data point has either a categorical or multinomial distribution, and the prior distribution of the distribution’s parameter (the vector of probabilities that generates the data point) is distributed as a Dirichlet, then the posterior distribution of the parameter is also a Dirichlet.
Exploring a Corpus with the posterior distribution
Quantities needed for exploring a corpus are the posterior expectations of hidden variables. Each of these quantities are conditioned on the observed corpus.
Visualizing a topic is done by visualizing the posterior topics through their pertopic probabilities \(\hat{\beta}\).
Visualizing a document uses the posterior topic proportions \(\hat{\theta}_{d,k}\) and the posterior topic assignments \(\hat{z}_{d,k}\).
Finding similar documents can be done through the Hellinger distance:
\begin{align*} D_{d,k} = \sum_{k=1}^K \left( \sqrt{\hat{\theta}_{d,k}}  \sqrt{\hat{\theta}_{f,k}}\right)^2 \end{align*}
Posterior Inference

Mean Field Variational Inference
Approximate intractable posterior distribution with a simpler distribution containing free variational parameters. These parameters are fit to approximate the true posterior.
In contrast to the true posterior, the mean field variational distribution for LDA is one where the variables are independent of each other, with and each governed by a different variational parameter.
We fit the variational parameters to minimise the KLdivergence to the true posterior.
The general approach to meanfield variational methods  update each variational parameter with the parameter given by the expectation of the true posterior under the variational distribution  is applicable when the conditional distribution of each variable is the exponential family.
Markov Chains
http://setosa.io/ev/markovchains/
Shortcomings
 strong, potentially invalid statistical assumptions:
 topics have no correlation to one another (dirichlet assumes
nearly independent)
 solution: CTM: use a logistic normal distribution
 assumes order of documents don’t matter
 solution: DTM: use logistic normal distribution to model topics evolving over time
 topics have no correlation to one another (dirichlet assumes
nearly independent)
TopicRNN
http://www.columbia.edu/~jwp2128/Papers/DiengWangetal2017.pdf
In TopicRNN, latent topic models are used to capture global semantic dependencies so that the RNN can focus its modeling capacity on the local dynamics of the sequences