Published on Tue Nov 20 2018

Limited Gradient Descent: Learning With Noisy Labels

Yi Sun, Yan Tian, Yiping Xu, Jianxiang Li

Deep neural networks tend to prioritize the learning of simple patterns over the memorization of noise patterns. This suggests a possible method to search for the best generalization that learns the main pattern until the noise begins to be memorized.

0
0
0
Abstract

Label noise may affect the generalization of classifiers, and the effective learning of main patterns from samples with noisy labels is an important challenge. Recent studies have shown that deep neural networks tend to prioritize the learning of simple patterns over the memorization of noise patterns. This suggests a possible method to search for the best generalization that learns the main pattern until the noise begins to be memorized. Traditional approaches often employ a clean validation set to find the best stop timing of learning, i.e., early stopping. However, the generalization performance of such methods relies on the quality of validation sets. Further, in practice, a clean validation set is sometimes difficult to obtain. To solve this problem, we propose a method that can estimate the optimal stopping timing without a clean validation set, called limited gradient descent. We modified the labels of a few samples in a noisy dataset to obtain false labels and to create a reverse pattern. By monitoring the learning progress of the noisy and reverse samples, we can determine the stop timing of learning. In this paper, we also theoretically provide some necessary conditions on learning with noisy labels. Experimental results on CIFAR-10 and CIFAR-100 datasets demonstrate that our approach has a comparable generalization performance to methods relying on a clean validation set. Thus, on the noisy Clothing-1M dataset, our approach surpasses methods that rely on a clean validation set.

Thu Dec 13 2018
Machine Learning
Learning to Learn from Noisy Labeled Data
Deep neural networks (DNNs) can easily overfit to the label noise. To overcome this problem, we propose a noise-tolerant training algorithm. The proposed method simulates actual training by generating synthetic noisy labels.
0
0
0
Thu Apr 01 2021
Machine Learning
Learning from Noisy Labels via Dynamic Loss Thresholding
0
0
0
Tue Nov 19 2019
Machine Learning
How does Early Stopping Help Generalization against Label Noise?
Noisy labels are common in real-world training data. Overfitting can be avoided by "early stopping" a deep neural network before the noisy labels are memorized.
0
0
0
Thu May 27 2021
Machine Learning
Training Classifiers that are Universally Robust to All Label Noise Levels
Deep neural networks are prone to overfitting in the presence of label noise. We propose a framework that incorporates a new subcategory of Positive-Unlabeled learning. The results show that our framework generally outperforms at medium to high noise levels.
1
0
0
Thu Dec 24 2020
Machine Learning
Identifying Training Stop Point with Noisy Labeled Data
Training deep neural networks with noisy labels is a challenging problem due to over-parameterization. To overcome these issues, we provide a novel training solution, free of these conditions. We analyze the rate of change of the training accuracy for different noise ratios under different conditions to identify a training stop region.
0
0
0
Sun Nov 15 2020
Machine Learning
Coresets for Robust Training of Neural Networks against Noisy Labels
Modern neural networks have the capacity to overfit noisy labels frequently found in real-world datasets. We propose a novel approach with strong theoretical guarantees for robust training of deep networks.
0
0
0