Published on Fri Sep 03 2021

Impact of GPU uncertainty on the training of predictive deep neural networks

Maciej Pietrowski, Andrzej Gajda, Takuto Yamamoto, Taisuke Kobayashi, Lana Sinapayen, Eiji Watanabe

Deep neural networks often present uncertainties such as hardware- and software-derived noise and randomness. We studied the effects of such uncertainties on learning outcomes, with a particular focus on the function of GPUs. We found that GPU-induced uncertainty increased learning accuracy of a certain deep neural network.

7
28
112
Abstract

Deep neural networks often present uncertainties such as hardware- and software-derived noise and randomness. We studied the effects of such uncertainty on learning outcomes, with a particular focus on the function of graphics processing units (GPUs), and found that GPU-induced uncertainty increased learning accuracy of a certain deep neural network. When training a predictive deep neural network using only the CPU without the GPU, the learning error is higher than when training the same number of epochs using the GPU, suggesting that the GPU plays a different role in the learning process than just increasing the computational speed. Because this effect cannot be observed in learning by a simple autoencoder, it could be a phenomenon specific to certain types of neural networks. GPU-specific computational processing is more indeterminate than that by CPUs, and hardware-derived uncertainties, which are often considered obstacles that need to be eliminated, might, in some cases, be successfully incorporated into the training of deep neural networks. Moreover, such uncertainties might be interesting phenomena to consider in brain-related computational processing, which comprises a large mass of uncertain signals.

Fri Dec 16 2016
Neural Networks
Neuromorphic Deep Learning Machines
The workhorse of deep learning, the gradient descent Back Propagation (BP) rule, often relies on the immediate availability of network-wide information. We demonstrate an event-driven random BP rule that uses an error-modulated synaptic plasticity for learning deep representations in neuromorphic computing hardware.
0
0
0
Wed Jun 09 2021
Machine Learning
StreamBrain: An HPC Framework for Brain-like Neural Networks on CPUs, GPUs and FPGAs
StreamBrain is a framework that allows neural networks based on BCPNN to be practically deployed in High-Performance Computing systems. It is a domain-specific language (DSL) similar in concept to existing machine learning (ML) frameworks.
1
0
0
Thu Aug 22 2019
Machine Learning
Automated Architecture Design for Deep Neural Networks
Machine learning has made tremendous progress in recent years. Deep learning models are responsible for many state-of-the-art applications of machine learning. Despite their success, deep learning models typicallyare hard to train and difficult to understand.
1
1
0
Tue Sep 01 2020
Machine Learning
Training Deep Neural Networks with Constrained Learning Parameters
Today's deep learning models are primarily trained on CPUs and GPUs. Although these models tend to have low error, they consume high power and utilize large amounts of memory. We believe that deep neural networks (DNNs) running on neuromorphic computing systems would be instrumental for intelligent edge computing systems.
0
0
0
Tue Feb 28 2017
Neural Networks
Improving the Neural GPU Architecture for Algorithm Learning
Algorithm learning is a core problem in artificial intelligence with significant implications on automation level that can be achieved by machines. Recent deep learning methods are emerging for synthesizing an algorithm.
0
0
0
Mon Dec 01 2014
Neural Networks
Scalability and Optimization Strategies for GPU Enhanced Neural Networks (GeNN)
GeNN provides a simulation environment that performs on General Purpose NVIDIA GPUs with a code generation based approach. GeNN allows the users to specify the populations of neurons at different stages, their synapse connection densities and the model of individual neurons.
0
0
0
Tue Jan 09 2018
Artificial Intelligence
Moments in Time Dataset: one million videos for event understanding
We present the Moments in Time Dataset, a large-scale human-annotated collection of one million short videos corresponding to dynamic events within three seconds. We describe the annotation process of our dataset (each video is tagged with one action or activity label among 339 different classes), analyze
0
0
0
Mon May 23 2016
Machine Learning
Deep Learning without Poor Local Minima
In this paper, we prove a conjecture published in 1989 and also partially address an open problem announced at the Conference on Learning Theory (COLT) With no unrealistic assumption, we first prove the following statements for the squared loss function of deep linear neural networks with any depth and widths.
2
0
0
Wed May 25 2016
Neural Networks
Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning
Unsupervised learning remains a difficult unsolved challenge. We explore prediction of future frames in a video sequence as an unsupervised learning rule. We describe a predictive neural network architecture that is inspired by the concept of "predictive coding"
0
0
0
Fri Nov 09 2018
Neural Networks
A Convergence Theory for Deep Learning via Over-Parameterization
Deep neural networks (DNNs) have demonstrated dominating performance in many fields. Since AlexNet, networks used in practice are going wider and deeper. In this work, we prove why stochastic gradient descent (SGD) can find global minima on the training objective of DNNs.
0
0
0
Thu Jan 30 2020
Neural Networks
Non-Determinism in TensorFlow ResNets
Stochasticity in training ResNets for image classification is dominated by the non-determinism from GPUs. Standard deviation of test set accuracy is 0.02 with fixed seeds, compared to 0.027 with different seeds. For test set loss the ratio of standard deviations is more
0
0
0
Fri Oct 03 2014
Neural Networks
cuDNN: Efficient Primitives for Deep Learning
We present a library of efficient implementations of deep learning primitives. Deep learning workloads are computationally intensive, and optimizing their kernels is difficult and time-consuming. The library is easy to integrate into existing frameworks, and provides optimized performance and memory usage.
0
0
0