Published on Mon Jul 05 2021

On The Distribution of Penultimate Activations of Classification Networks

Minkyo Seo, Yoonho Lee, Suha Kwak

This paper studies probability distributions of penultimate activations of classification networks. We show that, when a classification network is trained with the cross-entropy loss, its final classification layer forms a Generative-Discriminative pair with a generative classifier.

2
0
0
Abstract

This paper studies probability distributions of penultimate activations of classification networks. We show that, when a classification network is trained with the cross-entropy loss, its final classification layer forms a Generative-Discriminative pair with a generative classifier based on a specific distribution of penultimate activations. More importantly, the distribution is parameterized by the weights of the final fully-connected layer, and can be considered as a generative model that synthesizes the penultimate activations without feeding input data. We empirically demonstrate that this generative model enables stable knowledge distillation in the presence of domain shift, and can transfer knowledge from a classifier to variational autoencoders and generative adversarial networks for class-conditional image generation.

Fri Jun 10 2016
Neural Networks
Improved Techniques for Training GANs
We present a variety of new architectural features and training procedures that we apply to the generative adversarial networks (GANs) framework. We focus on two applications of GANs: semi-supervised learning, and the generation of images that humans find visually realistic. Using our new techniques,
0
0
0
Sun May 26 2019
Machine Learning
Classification Accuracy Score for Conditional Generative Models
Deep generative models (DGMs) of images are now sufficiently mature that they produce nearly photorealistic samples. These results suggest that DGMs are learning the data distribution in a perceptually meaningful space. To test this latter hypothesis, we use a number of model classes to infer the class labels of real data.
0
0
0
Thu Nov 06 2014
Machine Learning
Conditional Generative Adversarial Nets
Generative Adversarial Nets [8] were recently introduced as a novel way to train generative models. In this work we introduce the conditional version of these nets, which can be constructed by simply feeding the data. We show that this model can generate MNIST digits conditioned on class labels.
2
0
3
Fri Jun 02 2017
Machine Learning
PixelGAN Autoencoders
The "PixelGAN autoencoder" is a generative path that is conditioned on a latent code. Different priors result in different decompositions of information between the latent code and the decoder. We can achieve competitive semi-supervised classification results on the MNIST,
0
0
0
Sun Aug 06 2017
Machine Learning
Probabilistic Generative Adversarial Networks
Probabilistic Generative Adversarial Network (PGAN) is a new GAN variant based on a new kind of objective function. PGAN is better able to cope with instability problems that are usually observed in the GAN training procedure.
0
0
0
Sat Nov 19 2016
Computer Vision
Semi-Supervised Learning with Context-Conditional Generative Adversarial Networks
We introduce a simple semi-supervised learning approach for images based on in-painting using an adversarial loss. Images with random patches removed are presented to a generator whose task is to fill in the hole. The in-painted images are then presented to a discriminator network
0
0
0
Thu Aug 25 2016
Machine Learning
Densely Connected Convolutional Networks
The Dense Convolutional Network (DenseNet) connects each layer to every other layer in afeed-forward fashion. For each layer, the feature-maps of all preceding layers are used as inputs.
1
7
36
Mon Mar 09 2015
Machine Learning
Distilling the Knowledge in a Neural Network
A new type of ensemble composed of one or more full models and many specialist models. Unlike a mixture of experts, these specialist models can be trained rapidly and in parallel.
5
4
21
Mon May 23 2016
Computer Vision
Wide Residual Networks
Deep residual networks were shown to be able to scale up to thousands of layers and still have improving performance. Each fraction of a percent of improved accuracy costs nearly doubling the number of layers. Training very deep residual networks has a problem of diminishing feature reuse.
2
0
8
Thu Sep 04 2014
Computer Vision
Very Deep Convolutional Networks for Large-Scale Image Recognition
Convolutional networks of increasing depth can achieve state-of-the-art results. The research was the basis of the team's ImageNet Challenge 2014.
2
2
7
Mon Jul 30 2018
Computer Vision
ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design
Currently, the neural network architecture design is mostly guided by the indirect metric of computation complexity, i.e., FLOPs. This work proposes to evaluate the direct metric on the target platform, beyond only considering FLops. Based on a series of controlled experiments, this work derives several
1
0
2
Tue Apr 03 2018
Machine Learning
Hyperspherical Variational Auto-Encoders
The Variational Auto-Encoder (VAE) is one of the most used unsupervised machine learning models. We show that this parameterization fails to model data with a latent hyperspherical structure. We propose using a von Mises-Fisher (vMF) distribution
1
0
2