Published on Mon Oct 05 2020

Sample weighting as an explanation for mode collapse in generative adversarial networks

Aksel Wilhelm Wold Eide, Eilif Solberg, Ingebjørg Kåsen

Generative adversarial networks were introduced with a logistic MiniMax cost-reformulation, which normally fails to train due to saturation. While addressing the saturation problem, NS-GAN also inverts the generator's sample weighting, shifting emphasis from higher-scoring to lower-scoring samples.

0
0
0
Abstract

Generative adversarial networks were introduced with a logistic MiniMax cost formulation, which normally fails to train due to saturation, and a Non-Saturating reformulation. While addressing the saturation problem, NS-GAN also inverts the generator's sample weighting, implicitly shifting emphasis from higher-scoring to lower-scoring samples when updating parameters. We present both theory and empirical results suggesting that this makes NS-GAN prone to mode dropping. We design MM-nsat, which preserves MM-GAN sample weighting while avoiding saturation by rescaling the MM-GAN minibatch gradient such that its magnitude approximates NS-GAN's gradient magnitude. MM-nsat has qualitatively different training dynamics, and on MNIST and CIFAR-10 it is stronger in terms of mode coverage, stability and FID. While the empirical results for MM-nsat are promising and favorable also in comparison with the LS-GAN and Hinge-GAN formulations, our main contribution is to show how and why NS-GAN's sample weighting causes mode dropping and training collapse.