Published on Fri Mar 20 2020

Event-Based Control for Online Training of Neural Networks

Zilong Zhao, Sophie Cerf, Bogdan Robu, Nicolas Marchand

Convolutional Neural Network (CNN) has become the most used method for image classification. During its training the learning rate and the gradient are two key factors to tune for influencing the convergence speed of the model. The Event-Based E/PD is better than the original algorithm.

0
0
0
Abstract

Convolutional Neural Network (CNN) has become the most used method for image classification tasks. During its training the learning rate and the gradient are two key factors to tune for influencing the convergence speed of the model. Usual learning rate strategies are time-based i.e. monotonous decay over time. Recent state-of-the-art techniques focus on adaptive gradient algorithms i.e. Adam and its versions. In this paper we consider an online learning scenario and we propose two Event-Based control loops to adjust the learning rate of a classical algorithm E (Exponential)/PD (Proportional Derivative)-Control. The first Event-Based control loop will be implemented to prevent sudden drop of the learning rate when the model is approaching the optimum. The second Event-Based control loop will decide, based on the learning speed, when to switch to the next data batch. Experimental evaluationis provided using two state-of-the-art machine learning image datasets (CIFAR-10 and CIFAR-100). Results show the Event-Based E/PD is better than the original algorithm (higher final accuracy, lower final loss value), and the Double-Event-BasedE/PD can accelerate the training process, save up to 67% training time compared to state-of-the-art algorithms and even result in better performance.

Mon Nov 18 2019
Machine Learning
Feedback Control for Online Training of Neural Networks
We present E(Exponential)/PD (Proportional Derivative)-Control, a conditional learning rate strategy. It combines a feedback PD controller based on the CNN loss function,with an exponential control signal to smartly boost the learning.
0
0
0
Sat Dec 05 2020
Machine Learning
Weight Update Skipping: Reducing Training Time for Artificial Neural Networks
Artificial Neural Networks (ANNs) are known as state-of-the-art techniques in Machine Learning (ML) They have achieved outstanding results in data-intensive applications. We propose a new training methodology for ANNs that exploits the observation of improvement of accuracy.
0
0
0
Fri Sep 15 2017
Artificial Intelligence
ClickBAIT: Click-based Accelerated Incremental Training of Convolutional Neural Networks
Some applications will require training in real-time on live video streams with a human-in-the-loop. We show that by exploiting the time-ordered nature of the video stream through optical flow-based object tracking, we can increase the effectiveness of human actions by about 8 times.
0
0
0
Thu Sep 26 2019
Computer Vision
Convolutional Neural Networks with Dynamic Regularization
Regularization is commonly used for alleviating overfitting in machine learning. For convolutional neural networks, regularization methods, such as DropBlock and Shake-Shake, have illustrated the improvement in the generalization performance. But these methods lack a self-adaptive ability throughout training.
0
0
0
Tue Jun 13 2017
Computer Vision
Deep Control - a simple automatic gain control for memory efficient and high performance training of deep convolutional neural networks
The technique comprises a scaled per sample map mean subtraction from the raw convolutional filter output followed by scaling of the difference. The same model is used both for training and inference.
0
0
0
Sun Aug 18 2019
Machine Learning
Demystifying Learning Rate Policies for High Accuracy Training of Deep Neural Networks
Learning Rate (LR) is an important hyper-parameter to tune for effective training of deep neural networks (DNNs) We present a set of metrics for evaluating and selecting LR policies and implement them in LRBench, an LR benchmarking system.
0
0
0