Published on Sat Jun 19 2021

Learning and Generalization in Overparameterized Normalizing Flows

Kulin Shah, Amit Deshpande, Navin Goyal

Normalizing flows (NFs) are an important class of models in unsupervised learning for sampling and density estimation. In supervised learning, it is known that overparameterized neural networks with one hidden layer provably and efficiently learn and generalize. In contrast, the benefit of

1
0
0
Abstract

In supervised learning, it is known that overparameterized neural networks with one hidden layer provably and efficiently learn and generalize, when trained using stochastic gradient descent with sufficiently small learning rate and suitable initialization. In contrast, the benefit of overparameterization in unsupervised learning is not well understood. Normalizing flows (NFs) constitute an important class of models in unsupervised learning for sampling and density estimation. In this paper, we theoretically and empirically analyze these models when the underlying neural network is one-hidden-layer overparameterized network. Our main contributions are two-fold: (1) On the one hand, we provide theoretical and empirical evidence that for a class of NFs containing most of the existing NF models, overparametrization hurts training. (2) On the other hand, we prove that unconstrained NFs, a recently introduced model, can efficiently learn any reasonable data distribution under minimal assumptions when the underlying network is overparametrized.

Fri May 29 2020
Machine Learning
OT-Flow: Fast and Accurate Continuous Normalizing Flows via Optimal Transport
A normalizing flow is an invertible mapping between an arbitrary probability distribution and a standard normal distribution. The log-determinant of the flow's Jacobian can be obtained by integrating the trace of the dynamics' Jacobian along the flow. The proposed OT-Flow approach tackles two critical computational challenges.
1
0
0
Tue Oct 09 2018
Machine Learning
Learning One-hidden-layer Neural Networks under General Input Distributions
Stochastic gradient descent theoretically recovers the true parameters with global initializations and empirically outperforms existing approaches. Central to our approach is the task of estimating the score function from samples, which is of basic and independent interest to theoretical statistics.
0
0
0
Mon Mar 06 2017
Machine Learning
Multiplicative Normalizing Flows for Variational Bayesian Neural Networks
We reinterpret multiplicative noise in neural networks as auxiliary random variables that augment the approximate posterior. We show that with this new approximation we can significantly improve upon classical mean field for Bayesian neural networks.
0
0
0
Wed Jan 08 2020
Machine Learning
Learning Generative Models using Denoising Density Estimators
Learn probabilistic models that can estimate the density of a given set ofamples, and generate samples from that density. We introduce a new generative model based on denoising density estimators (DDEs)
0
0
0
Tue Feb 12 2019
Machine Learning
Gaussian Mean Field Regularizes by Limiting Learned Information
We show how mean field inference improves generalization. We quantify a maximum capacity when the posterior variance is either fixed or learned. Our experiments demonstrate that bounding information between parameters and data effectively regularizes neural networks on both supervised and unsupervised tasks.
0
0
0
Tue Feb 12 2019
Machine Learning
Towards moderate overparameterization: global convergence guarantees for training shallow neural networks
Many modern neural network architectures are trained in an overparameterized regime. This is where the parameters of the model exceed the size of the training data. In practice much more moderate levels of over Parameterization seems to be sufficient.
0
0
0