Published on Tue Jan 26 2016

Unifying Adversarial Training Algorithms with Flexible Deep Data Gradient Regularization

C. Lee Giles, Daniel Kifer

Many previous proposals for adversarial training of deep neural nets have included di- rectly modifying the gradient. In this paper, we show these proposals are all instances of optimizing a general, regularized objective we call DataGrad.

0
0
0
Abstract

Many previous proposals for adversarial training of deep neural nets have included di- rectly modifying the gradient, training on a mix of original and adversarial examples, using contractive penalties, and approximately optimizing constrained adversarial ob- jective functions. In this paper, we show these proposals are actually all instances of optimizing a general, regularized objective we call DataGrad. Our proposed DataGrad framework, which can be viewed as a deep extension of the layerwise contractive au- toencoder penalty, cleanly simplifies prior work and easily allows extensions such as adversarial training with multi-task cues. In our experiments, we find that the deep gra- dient regularization of DataGrad (which also has L1 and L2 flavors of regularization) outperforms alternative forms of regularization, including classical L1, L2, and multi- task, both on the original dataset as well as on adversarial sets. Furthermore, we find that combining multi-task optimization with DataGrad adversarial training results in the most robust performance.

Wed Oct 25 2017
Machine Learning
mixup: Beyond Empirical Risk Minimization
Large deep neural networks are powerful, but exhibit undesirable behaviors such as memorization and sensitivity to adversarial examples. In this work, we propose mixup, a simple learning principle to alleviate these issues. In essence, mixup trains a neural network on convex combinations of pairs of examples and
3
1
4
Fri Jan 25 2019
Machine Learning
Towards a Deeper Understanding of Adversarial Losses under a Discriminative Adversarial Network Setting
Recent work has proposed various adversarial loss functions for training either generative or discriminative models. It remains unclear what types of functions are valid adversarial losses and how these loss functions perform against one another. We aim to gain a deeper understanding by decoupling the effects of their component
0
0
0
Wed Feb 26 2020
Machine Learning
Overfitting in adversarially robust deep learning
It is common practice in deep learning to use overparameterized networks and train for as long as possible. We find that overfitting to the training set does in fact harm performance to a very large degree in adversarially robust training models.
0
0
0
Tue Jun 04 2019
Machine Learning
Adversarial Training is a Form of Data-dependent Operator Norm Regularization
We establish a theoretical link between adversarial training and operator norm regularization for deep neural networks. We provide extensive empirical evidence on state-of-the-art network architectures to support our theoretical results.
0
0
0
Thu Jun 21 2018
Machine Learning
Gradient Adversarial Training of Neural Networks
gradient adversarial training is an auxiliary deep learning framework applicable to different machine learning problems. We leverage a prior belief that in many contexts,ultaneous gradient updates should be statistically indistinguishable from each other. We enforce this consistency using an auxiliary network that classifies the origin of the gradient tensor.
0
0
0
Tue Apr 16 2019
Machine Learning
Reducing Adversarial Example Transferability Using Gradient Regularization
Deep learning algorithms have increasingly been shown to lack robustness to simple adversarial examples (AdvX) An equally troubling observation is that these examples transfer between different architectures trained on different datasets. We investigate the transferability of adversarial. examples between models using the angle between the input-output
0
0
0