Published on Fri May 18 2018

Reconstruction of training samples from loss functions

Akiyoshi Sannai

This paper presents a new mathematical framework to analyze the loss functions of deep neural networks with ReLU functions. Using virtual polynomials, we find an algebraic structure for the loss surfaces, called semi-algebraic sets. The main theorem in this paper indicates that there are many risks associated with the training ofDeep neural networks.

0
0
0
Abstract

This paper presents a new mathematical framework to analyze the loss functions of deep neural networks with ReLU functions. Furthermore, as as application of this theory, we prove that the loss functions can reconstruct the inputs of the training samples up to scalar multiplication (as vectors) and can provide the number of layers and nodes of the deep neural network. Namely, if we have all input and output of a loss function (or equivalently all possible learning process), for all input of each training sample $x_i \in \mathbb{R}^n$, we can obtain vectors satisfying for some . To prove theorem, we introduce the notion of virtual polynomials, which are polynomials written as the output of a node in a deep neural network. Using virtual polynomials, we find an algebraic structure for the loss surfaces, called semi-algebraic sets. We analyze these loss surfaces from the algebro-geometric point of view. Factorization of polynomials is one of the most standard ideas in algebra. Hence, we express the factorization of the virtual polynomials in terms of their active paths. This framework can be applied to the leakage problem in the training of deep neural networks. The main theorem in this paper indicates that there are many risks associated with the training of deep neural networks. For example, if we have N (the dimension of weight space) + 1 nonsmooth points on the loss surface, which are sufficiently close to each other, we can obtain the input of training sample up to scalar multiplication. We also point out that the structures of the loss surfaces depend on the shape of the deep neural network and not on the training samples.

Thu Aug 01 2019
Machine Learning
A direct approach for function approximation on data defined manifolds
In much of the literature on function approximation by deep networks, the function is assumed to be defined on some known domain. In practice, the data might not be dense on these domains, and the approximation theory results are observed to be too conservative. In this paper, we propose a more direct approach to function approximation on data defined manifolds.
0
0
0
Fri Feb 16 2018
Neural Networks
Gradient descent with identity initialization efficiently learns positive definite linear transformations by deep residual networks
We analyze algorithms for approximating a function using deep linear neural networks. We focus on algorithms that learn through gradient descent on the population quadratic loss in the case that the distribution over the inputs is isotropic. We show that some forms of regularization toward the identity in each layer do not help.
0
0
0
Mon Nov 11 2019
Machine Learning
Stronger Convergence Results for Deep Residual Networks: Network Width Scales Linearly with Training Data Size
0
0
0
Fri Aug 02 2019
Machine Learning
Deep ReLU network approximation of functions on a manifold
We study a regression problem with inputs on a -dimensional manifold. The problem is embedded into a space with potentially much larger ambient dimension. We show that sparsely connected deep ReLU networks can approximate a function with smoothness index up to error.
0
0
0
Thu Jul 30 2020
Machine Learning
Random Vector Functional Link Networks for Function Approximation on Manifolds
The learning speed of feed-forward neural networks is notoriously slow and has presented a bottleneck in deep learning applications for several decades. To counter this, researchers and practitioners have tried introducing randomness to reduce the learning requirement. Based on the original construction of Igelnik and Pao, single layer neural-
0
0
0
Tue Jun 18 2019
Machine Learning
The Barron Space and the Flow-induced Function Spaces for Neural Network Models
One of the key issues in the analysis of machine learning models is to determine the appropriate function space and norm for the model. In this paper, we address this issue for two representative neural network models: the two-layer networks and the residual neural networks.
0
0
0