Published on Fri Jul 02 2021

Disentangling Transfer and Interference in Multi-Domain Learning

Yipeng Zhang, Tyler L. Hayes, Christopher Kanan

The benefits of transfer in multi-domain learning have not been adequately studied. Learning multiple domains could be beneficial or these domains couldInterfere with each other given limited network capacity. We propose new metrics disentangling interference and Transfer and set up experimental protocols.

1
0
0
Abstract

Humans are incredibly good at transferring knowledge from one domain to another, enabling rapid learning of new tasks. Likewise, transfer learning has enabled enormous success in many computer vision problems using pretraining. However, the benefits of transfer in multi-domain learning, where a network learns multiple tasks defined by different datasets, has not been adequately studied. Learning multiple domains could be beneficial or these domains could interfere with each other given limited network capacity. In this work, we decipher the conditions where interference and knowledge transfer occur in multi-domain learning. We propose new metrics disentangling interference and transfer and set up experimental protocols. We further examine the roles of network capacity, task grouping, and dynamic loss weighting in reducing interference and facilitating transfer. We demonstrate our findings on the CIFAR-100, MiniPlaces, and Tiny-ImageNet datasets.

Mon Dec 03 2018
Neural Networks
Transferring Knowledge across Learning Processes
In complex transfer learning scenarios new tasks might not be tightly linked to previous tasks. Approaches that transfer information contained only in the finalized parameters of a source model will struggle. We propose Leap, aframework that achieves this by transferring knowledge across learning processes.
0
0
0
Mon Mar 15 2021
Machine Learning
How to distribute data across tasks for meta-learning?
Meta-learning models transfer knowledge acquired from previous tasks to learn new ones. They are trained on benchmarks with a fixed number of data points per task. Since labelling of data is expensive, finding the optimal allocation of labels across training tasks may reduce costs.
1
0
1
Tue May 23 2017
Neural Networks
Latent Multi-task Architecture Learning
Multi-task learning (MTL) allows deep neural networks to learn from related tasks by sharing parameters with other networks. MTL involves searching an enormous space of possible parameter sharing to find layers or subspaces that benefit from sharing.
0
0
0
Wed May 15 2019
Machine Learning
Learning What and Where to Transfer
Deep learning has expanded to real-world problems with insufficient volume of training data. We propose a novel transfer learning approach based on meta-learning that can automatically learn what knowledge to transfer from the source network to where in the target network.
0
0
0
Fri May 31 2019
Neural Networks
Modular Universal Reparameterization: Deep Multi-task Learning Across Diverse Domains
Deep learning applications continue to become more diverse. Can general problem solving arise from jointly learning several diverse tasks? To approach this question, deep multi-task learning is extended.
0
0
0
Thu Jul 02 2020
Neural Networks
Learn Faster and Forget Slower via Fast and Stable Task Adaptation
Training Deep Neural Networks (DNNs) is still highly time-consuming and compute-intensive. adapting a pretrained model may significantly accelerate this process. Fast And Stable Task-adaptation is an easy to apply fine-tuning algorithm.
0
0
0
Wed Feb 11 2015
Machine Learning
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Batch Normalization allows us to use much higher learning rates and be less careful about initialization. It also acts as a regularizer, in some cases eliminating the need for Dropout. Applied to a state-of-the-art image classification model, Batch Normalized achieves the same
2
342
1,163
Tue Feb 02 2021
Machine Learning
Scaling Laws for Transfer
When we train increasingly large neural networks from-scratch on a fixed-size dataset, they eventually become data-limited and stop improving in performance. We find that pre-training effectivelymultiplies the fine-tuning dataset size.
5
60
270
Sat May 18 2019
Computer Vision
Which Tasks Should Be Learned Together in Multi-task Learning?
A neural network can be trained to solve multiple tasks simultaneously using multi-task learning. This can save computation at inference time as only a single network needs to be evaluated. Unfortunately, this often leads to inferior overall performance as task objectives can compete.
1
3
13
Wed Mar 28 2018
Computer Vision
End-to-End Multi-Task Learning with Attention
The Multi-Task Attention Network (MTAN) consists of a single shared network containing a global feature pool, together with a soft-attention module for each task. We show that our architecture is state-of-the-art in multi-task learning compared to existing methods.
1
3
13
Wed Oct 10 2018
Machine Learning
Multi-Task Learning as Multi-Objective Optimization
Multi-task learning is inherently a multi-objective problem because different tasks may conflict. A common compromise is to optimize a proxy objective that minimizes a weighted linear combination of per-task losses. This workaround is only valid when the tasks do not compete, which is rarely the case.
2
0
10
Wed May 01 2019
Machine Learning
Similarity of Neural Network Representations Revisited
Recent work has sought to understand the behavior of neural networks by comparing representations between layers and between different trained models. We show that neither CCA nor any other statistic that is invariant to invertible linear transformation can measure meaningful similarities between representations of higher dimension. We introduce a similarity index that
4
1
6