Published on Fri Feb 02 2018

An Instability in Variational Inference for Topic Models

Behrooz Ghorbani, Hamid Javadi, Andrea Montanari

Topic models are Bayesian models that are frequently used to capture the structure of certain corpora of documents or images. The standard approach towards approximating the true decomposition of topics is to use a mean field approximation. We show that this approach suffers from an instability that can produce misleading conclusions.

0
0
0
Abstract

Topic models are Bayesian models that are frequently used to capture the latent structure of certain corpora of documents or images. Each data element in such a corpus (for instance each item in a collection of scientific articles) is regarded as a convex combination of a small number of vectors corresponding to `topics' or `components'. The weights are assumed to have a Dirichlet prior distribution. The standard approach towards approximating the posterior is to use variational inference algorithms, and in particular a mean field approximation. We show that this approach suffers from an instability that can produce misleading conclusions. Namely, for certain regimes of the model parameters, variational inference outputs a non-trivial decomposition into topics. However --for the same parameter values-- the data contain no actual information about the true decomposition, and hence the output of the algorithm is uncorrelated with the true topic decomposition. Among other consequences, the estimated posterior mean is significantly wrong, and estimated Bayesian credible regions do not achieve the nominal coverage. We discuss how this instability is remedied by more accurate mean field approximations.

Wed Sep 19 2012
Machine Learning
Variational Inference in Nonconjugate Models
Mean-field variational methods are widely used for approximate posterior inference in many probabilistic models. When the model is conditionally conjugate, the coordinate updates are easily derived and in closed form. In these models, mean-field methods cannot be directly applied.
0
0
0
Tue Dec 10 2019
Machine Learning
Frequentist Consistency of Generalized Variational Inference
This paper investigates Frequentist consistency properties of the posterior distribution of Generalized Variational Inference (GVI) A number of generic and novel strategies are given for proving consistency, relying on the theory of -convergence.
0
0
0
Tue Mar 21 2017
Machine Learning
A Deterministic Global Optimization Method for Variational Inference
In practice it is unclear whether the fixed point identified by the variational inference algorithm is a local or a global optimum. We present a minimal data set for empirically testing convergence. We characterize the loss incurred by choosing a non-optimal approximation distribution.
0
0
0
Thu Jan 10 2013
Artificial Intelligence
Incorporating Expressive Graphical Models in Variational Approximations: Chain-Graphs and Hidden Variables
Global variational approximation methods in graphical models allow efficient inference of complex posterior distributions. The choice of the approximating model determines a tradeoff between the complexity of the procedure and the quality of the approximation. We consider two classes of models that are richer than standard Bayesian networks.
0
0
0
Wed Jun 27 2012
Machine Learning
Variational Bayesian Inference with Stochastic Search
Mean-field variational inference is a method for approximate Bayesian posterior inference. It approximates a full posterior distribution with afactorized set of distributions by maximizing a lower bound on the marginal likelihood.
0
0
0
Mon Oct 09 2017
Machine Learning
-Variational Inference with Statistical Guarantees
We propose a family of variational approximations to Bayesian posterior distributions. The standard variational approximation is a special case of -VB with. A novel class of inequalities are developed for linking the Bayes risk under the approximation to the objective function.
0
0
0