Published on Mon May 10 2021

Exploiting Elasticity in Tensor Ranks for Compressing Neural Networks

Jie Ran, Rui Lin, Hayden K. H. So, Graziano Chesi, Ngai Wong
0
0
0
Abstract

Elasticities in depth, width, kernel size and resolution have been explored in compressing deep neural networks (DNNs). Recognizing that the kernels in a convolutional neural network (CNN) are 4-way tensors, we further exploit a new elasticity dimension along the input-output channels. Specifically, a novel nuclear-norm rank minimization factorization (NRMF) approach is proposed to dynamically and globally search for the reduced tensor ranks during training. Correlation between tensor ranks across multiple layers is revealed, and a graceful tradeoff between model size and accuracy is obtained. Experiments then show the superiority of NRMF over the previous non-elastic variational Bayesian matrix factorization (VBMF) scheme.

Wed Aug 12 2020
Computer Vision
Stable Low-rank Tensor Decomposition for Compression of Convolutional Neural Network
Most state of the art deep neural networks are overparameterized and exhibit a high computational cost. This paper is the first study on degeneracy in the tensor decomposition of convolutional kernels. The method results in much lower accuracy degradation and provides consistent performance.
0
0
0
Fri May 25 2018
Machine Learning
Tensorial Neural Networks: Generalization of Neural Networks and Application to Model Compression
We propose tensorial neural networks (TNNs) as a generalization of existing neural networks. TNNs extend tensor operations on low order operands to those on high order ones. The problem of parameter learning is challenging, as it corresponds to hierarchical nonlinear tensor decomposition.
0
0
0
Sun Mar 24 2019
Machine Learning
MUSCO: Multi-Stage Compression of neural networks
The low-rank tensor approximation is very promising for the compression of deep neural networks. We propose a new simple and efficient iterative approach. We demonstrate the efficiency of our method comparing to non-iterative ones.
0
0
0
Fri May 24 2019
Machine Learning
Bayesian Tensorized Neural Networks with Automatic Rank Selection
Tensor decomposition is an effective approach to compress over-parameterized neural networks. However, directly applying tensor compression in the training process is a challenging task due to the difficulty of choosing a proper tensor rank. This paper proposes a Bayesian tensorized neurological network.
0
0
0
Thu Jun 01 2017
Machine Learning
Tensor Contraction Layers for Parsimonious Deep Nets
Tensor methods are noted for their ability to discover multi-dimensional dependencies. The TCL is the first attempt to incorporate tensor contractions as end-to-end trainable neural network layers. Applied to existing networks, TCLs reduce the dimensionality of the activation tensors.
0
0
0
Mon Jul 26 2021
Computer Vision
Towards Efficient Tensor Decomposition-Based DNN Model Compression with Optimization Framework
Advanced tensor decomposition has been widely studied for deep neural network (DNN) model compression. Compressing CNNs using TT/TR always suffers significant accuracy loss. We propose a systematic framework for tensor decomposition-based model compression using Alternating Direction Method of HaitianMultipliers (ADMM)
1
0
0