Published on Tue May 22 2018

ARiA: Utilizing Richard's Curve for Controlling the Non-monotonicity of the Activation Function in Deep Neural Nets

Narendra Patwardhan, Madhura Ingalhalikar, Rahee Walambe

The function developed is a two parameter version of the specialized Richard's Curve and we call it Adaptive Richard's Curve weighted Activation (ARiA) This function is non-monotonous and allows a precise control over convexity by varying the hyper-parameters.

0
0
0
Abstract

This work introduces a novel activation unit that can be efficiently employed in deep neural nets (DNNs) and performs significantly better than the traditional Rectified Linear Units (ReLU). The function developed is a two parameter version of the specialized Richard's Curve and we call it Adaptive Richard's Curve weighted Activation (ARiA). This function is non-monotonous, analogous to the newly introduced Swish, however allows a precise control over its non-monotonous convexity by varying the hyper-parameters. We first demonstrate the mathematical significance of the two parameter ARiA followed by its application to benchmark problems such as MNIST, CIFAR-10 and CIFAR-100, where we compare the performance with ReLU and Swish units. Our results illustrate a significantly superior performance on all these datasets, making ARiA a potential replacement for ReLU and other activations in DNNs.

Mon Jan 22 2018
Machine Learning
E-swish: Adjusting Activations to Different Network Depths
The Rectified Linear Unit (ReLU) is currently the most used activation function. E-swish provides 1.5% and 4.6% accuracy improvements on WRN 10-2. The code to reproduce all our experiments can be found at GitHub.
0
0
0
Mon Jan 25 2021
Neural Networks
Activation Functions in Artificial Neural Networks: A Systematic Overview
0
0
0
Mon Aug 02 2021
Machine Learning
Piecewise Linear Units Improve Deep Neural Networks
The activation function is at the heart of a deep neural networks. Many practitioners prefer the Rectified Linear Unit (ReLU) due to its simplicity and reliability. Recent work on learning the function during training has shown promising results.
1
0
0
Mon Oct 16 2017
Neural Networks
Searching for Activation Functions
The choice of activation functions in deep networks has a significant effect on the training dynamics and task performance. Currently, the most successful and widely-used activation function is the Rectified Linear Unit (ReLU) We propose using automatic search techniques to discover new activation functions.
1
2
7
Fri Dec 27 2019
Neural Networks
Learning Neural Activations
An artificial neuron is modelled as a weighted summation followed by an activation function which determines its output. A wide variety of activation functions such as rectified linear units (ReLU), leaky-ReLU, Swish, MISH, etc. have been explored in the literature.
0
0
0
Tue Sep 08 2020
Neural Networks
TanhSoft -- a family of activation functions combining Tanh and Softplus
Deep learning at its core, contains functions that are composition of alinear transformation with a non-linear function known as activation function. In this work, we propose a family of novel activation functions, namely TanhSoft, with four undetermined hyper-parameters.
0
0
0