Published on Tue Oct 20 2020

Distributed Learning of Finite Gaussian Mixtures

Qiong Zhang, Jiahua Chen

The proposed split-and-conquer approach has comparable statistical performance with the global estimator based on the full dataset, if the latter is feasible. The new estimator is shown to be consistent and retains root-n consistency under some general conditions.

0
0
0
Abstract

Advances in information technology have led to extremely large datasets that are often kept in different storage centers. Existing statistical methods must be adapted to overcome the resulting computational obstacles while retaining statistical validity and efficiency. Split-and-conquer approaches have been applied in many areas, including quantile processes, regression analysis, principal eigenspaces, and exponential families. We study split-and-conquer approaches for the distributed learning of finite Gaussian mixtures. We recommend a reduction strategy and develop an effective MM algorithm. The new estimator is shown to be consistent and retains root-n consistency under some general conditions. Experiments based on simulated and real-world data show that the proposed split-and-conquer approach has comparable statistical performance with the global estimator based on the full dataset, if the latter is feasible. It can even slightly outperform the global estimator if the model assumption does not match the real-world data. It also has better statistical and computational performance than some existing methods.

Thu Sep 01 2016
Machine Learning
Ten Steps of EM Suffice for Mixtures of Two Gaussians
The Expectation-Maximization (EM) algorithm is a widely used method formaximum likelihood estimation in models with latent variables. We show that the population version of EM, where the algorithm is given access to infinitely many samples, converges to the correct mean vectors.
0
0
0
Tue Jun 27 2017
Machine Learning
Fast Algorithms for Learning Latent Variables in Graphical Models
We study the problem of learning latent variables in Gaussian graphical models. Existing methods for this problem assume that the precision matrix of observed variables is the superposition of a sparse and a low-rank component. In contrast with existing approaches, our algorithms are manifestly non-convex.
0
0
0
Mon Sep 29 2014
Machine Learning
Adaptive Low-Complexity Sequential Inference for Dirichlet Process Mixture Models
We develop a sequential low-complexity inference procedure for Dirichlet process mixtures of Gaussians for online clustering and parameter estimation. We demonstrate through experiments on synthetic and real data sets that our approach is superior to other online state-of-the-art methods.
0
0
0
Sat Sep 26 2020
Machine Learning
An Adaptive EM Accelerator for Unsupervised Learning of Gaussian Mixture Models
We propose an Anderson Acceleration (AA) scheme for the adaptive Expectation-Maximization (EM) algorithm. The proposed algorithm is able to determine the optimal number of mixture componentsautonomously. We demonstrate the accuracy and efficiency of the algorithm with synthetic data sets that are mixtures of Gaussians.
0
0
0
Wed Dec 04 2013
Machine Learning
Faster and Sample Near-Optimal Algorithms for Proper Learning Mixtures of Gaussians
We provide an algorithm for properly learning mixtures of two single-dimensional Gaussians without any separability assumptions. Our sample complexity is optimal up to optimal logarithmic factors. Our algorithm significantly improves upon both Kalai et. al., whose algorithm has a prohibitive dependence on $1/\varepsilon.
0
0
0
Tue Dec 09 2014
Artificial Intelligence
Hierarchical Mixture-of-Experts Model for Large-Scale Gaussian Process Regression
We propose a practical and scalable Gaussian process model for large-scale nonlinear probabilistic regression. Our mixture-of-experts model is conceptually simple and hierarchically recombines computations for an overall approximation of a full Gaussian.
0
0
0