Elasticities in depth, width, kernel size and resolution have been explored
in compressing deep neural networks (DNNs). Recognizing that the kernels in a
convolutional neural network (CNN) are 4-way tensors, we further exploit a new
elasticity dimension along the input-output channels. Specifically, a novel
nuclear-norm rank minimization factorization (NRMF) approach is proposed to
dynamically and globally search for the reduced tensor ranks during training.
Correlation between tensor ranks across multiple layers is revealed, and a
graceful tradeoff between model size and accuracy is obtained. Experiments then
show the superiority of NRMF over the previous non-elastic variational Bayesian
matrix factorization (VBMF) scheme.