Published on Fri Jan 31 2020

Generative Modeling with Denoising Auto-Encoders and Langevin Sampling

Adam Block, Youssef Mroueh, Alexander Rakhlin

We show that both DAE and DSM provide estimates of the score of the Gaussian smoothed population density. We then apply our results to the homotopy method of arXiv:1907.05600and provide theoretical justification for its empirical success.

1
0
2
Abstract

We study convergence of a generative modeling method that first estimates the score function of the distribution using Denoising Auto-Encoders (DAE) or Denoising Score Matching (DSM) and then employs Langevin diffusion for sampling. We show that both DAE and DSM provide estimates of the score of the Gaussian smoothed population density, allowing us to apply the machinery of Empirical Processes. We overcome the challenge of relying only on bounds on the score estimation error and provide finite-sample bounds in the Wasserstein distance between the law of the population distribution and the law of this sampling scheme. We then apply our results to the homotopy method of arXiv:1907.05600 and provide theoretical justification for its empirical success.

Thu Apr 13 2017
Machine Learning
Stochastic Gradient Descent as Approximate Bayesian Inference
Stochastic Gradient Descent with a constant learning rate (constant SGD)simulates a Markov chain with a stationary distribution. We show how to adjust the tuning parameters of constant SGD to best match the stationary distribution to a posterior.
0
0
0
Tue Feb 27 2018
Machine Learning
Mirrored Langevin Dynamics
We consider the problem of sampling from constrained distributions. We propose a unified framework, inspired by the classical mirror descent. We prove that, for a general target distribution with strongly convex potential, our framework implies the existence of a first-order algorithm achieving convergence.
0
0
0
Thu Feb 13 2020
Machine Learning
Stochastic Approximate Gradient Descent via the Langevin Algorithm
0
0
0
Mon Jan 08 2018
Machine Learning
Log-concave sampling: Metropolis-Hastings algorithms are fast
We prove a non-asymptotic upper bound on the mixing time of the Metropolis-adjusted Langevin algorithm (MALA) The method draws samples by simulating a Markov chain obtained from the discretization of an appropriate Langevin diffusion, combined with an accept-reject step.
0
0
0
Thu May 23 2019
Machine Learning
Accelerating Langevin Sampling with Birth-death
A fundamental problem in Bayesian inference and statistical machine learning is to efficiently sample from multimodal distributions. We propose a new sampling algorithm based on a birth-death mechanism to accelerate the mixing of Langevin diffusion.
0
0
0
Tue Feb 02 2021
Machine Learning
Exact Langevin Dynamics with Stochastic Gradients
Stochastic gradient Markov Chain Monte Carlo algorithms are popular samplers for approximate inference. We show that many versions of these methods cannot be corrected using Metropolis-Hastings rejection sampling. We can fix this by employing a sampler with Gradient-Guided Monte Carlo.
0
0
0
Fri Dec 06 2019
Machine Learning
Your Classifier is Secretly an Energy Based Model and You Should Treat it Like One
We propose to reinterpret a standard discriminative classifier of p(y|x) as an energy based model for the joint distribution p(x,y) In this setting, the class probabilities can be easily computed as well as unnormalized values of p (x|y)
2
41
150
Fri Jul 12 2019
Machine Learning
Generative Modeling by Estimating Gradients of the Data Distribution
We introduce a new generative model where samples are produced via Langevin dynamics. We perturb the data with different levels of Gaussian noise and jointly estimate the corresponding scores. Our models produce samples comparable to GANs on MNIST,CelebA and CIFAR-10 datasets.
2
0
5
Mon Dec 18 2017
Neural Networks
Size-Independent Sample Complexity of Neural Networks
We study the sample complexity of learning neural networks. We provide new bounds on Rademacher complexity assuming norm constraints on the parameter matrix of each layer.
0
0
0
Wed Dec 04 2019
Artificial Intelligence
Plug and Play Language Models: A Simple Approach to Controlled Text Generation
The Plug and Play Language Model (PPLM) combines a pretrained LM with one or more simple attribute classifiers that guide text generation. In the canonical scenario we present, the attribute models are simple classifiers consisting of a user-specified bag of words or a single learned layer.
1
0
0
Mon Jun 26 2017
Machine Learning
Spectrally-normalized margin bounds for neural networks
This paper presents a margin-based multiclass generalization bound for neural networks that scales with their margin-normalized "spectral complexity" The bound is the product of the spectral norms of the weight matrices, times a certain correction factor.
1
0
0
Thu Jul 09 2015
Machine Learning
Sampling from a log-concave distribution with Projected Langevin Monte Carlo
We extend the Langevin Monte Carlo (LMC) algorithm to compactly supported measures. We show that (projected) LMC allows to sample in polynomial time from a log-concave distribution with smooth potential.
0
0
0
Thu Apr 15 2021
Machine Learning
On Energy-Based Models with Overparametrized Shallow Neural Networks
Energy-based models are a simple yet powerful framework for generative modeling. They are based on a trainable energy function which defines an associated Gibbs measure.
1
0
1
Sat Jun 19 2021
Machine Learning
Deep Generative Learning via Schrödinger Bridge
We propose to learn a generative model via entropy interpolation. The generative learning task can be formulated as.interpolating between a reference distribution and a target distribution based. on the Kullback-Leibler divergence.
2
0
0
Fri Aug 14 2020
Computer Vision
Homotopic Gradients of Generative Density Priors for MR Image Reconstruction
Homotopic gradients of generative density priors are proposed for magnetic resonance imaging (MRI) reconstruction. Only 10% of the k-space data can still generate images of high quality as effectively as standard MRI reconstruction.
0
0
0
Sun Sep 27 2020
Computer Vision
Iterative Reconstruction for Low-Dose CT using Deep Gradient Priors of Generative Model
Iterative reconstruction is one of the most promising ways to compensate for the increased noise due to reduction of photon flux. We integrate the data-consistency as a conditional term into the iterative generative model for low-dose CT.
0
0
0
Fri Jun 19 2020
Machine Learning
Fast Mixing of Multi-Scale Langevin Dynamics under the Manifold Hypothesis
Langevin Dynamics is run in very high dimension on a nonconvex landscape. The high dimension of the sampling space significantly hurts the performance of the algorithm. We leverage a multi-scale approach to help ameliorate this issue.
0
0
0
Mon Dec 28 2020
Computer Vision
Joint Intensity-Gradient Guided Generative Modeling for Colorization
This paper proposes an iterative generative model for solving the automatic colorization problem. The starting point of the unsupervised learning is the observation that the gradient map possesses latent information of the image.
0
0
0