Published on Sun Jun 13 2021

Deep Bayesian Unsupervised Lifelong Learning

Tingting Zhao, Zifeng Wang, Aria Masoomi, Jennifer Dy
0
0
0
Abstract

Lifelong Learning (LL) refers to the ability to continually learn and solve new problems with incremental available information over time while retaining previous knowledge. Much attention has been given lately to Supervised Lifelong Learning (SLL) with a stream of labelled data. In contrast, we focus on resolving challenges in Unsupervised Lifelong Learning (ULL) with streaming unlabelled data when the data distribution and the unknown class labels evolve over time. Bayesian framework is natural to incorporate past knowledge and sequentially update the belief with new data. We develop a fully Bayesian inference framework for ULL with a novel end-to-end Deep Bayesian Unsupervised Lifelong Learning (DBULL) algorithm, which can progressively discover new clusters without forgetting the past with unlabelled data while learning latent representations. To efficiently maintain past knowledge, we develop a novel knowledge preservation mechanism via sufficient statistics of the latent representation for raw data. To detect the potential new clusters on the fly, we develop an automatic cluster discovery and redundancy removal strategy in our inference inspired by Nonparametric Bayesian statistics techniques. We demonstrate the effectiveness of our approach using image and text corpora benchmark datasets in both LL and batch settings.

Fri Oct 26 2018
Machine Learning
Accumulating Knowledge for Lifelong Online Learning
Lifelong learning can be viewed as a continuous transfer learning procedure. The learning procedure for each task is interactive. The predicted result for a given task is made by combining two intermediate predictions.
0
0
0
Tue May 02 2017
Machine Learning
A Strategy for an Uncompromising Incremental Learner
We show that phantom sampling helps avoid catastrophic forgetting during incremental learning. We apply these strategies to competitive multi-class learning of deep neural networks. We further put our strategy to test on challenging cases, including cross-domain increments and incrementing on a novel label space.
0
0
0
Wed Sep 04 2019
Machine Learning
Lifelong Machine Learning with Deep Streaming Linear Discriminant Analysis
Conventional deep neural networks suffer from catastrophic forgetting when they are incrementally updated. An alternative paradigm is online learning in a single pass through the training dataset. In streaming learning, an agent learns instances one-by-one and can be tested at any time.
0
0
0
Wed Mar 06 2019
Artificial Intelligence
Representative Task Self-selection for Flexible Clustered Lifelong Learning
Lifelong machine learning models are with prescribed size and can degenerate the performance for both learned tasks and coming ones when facing a new task environment (cluster) We propose a new incremental clustered lifelong learning framework with two knowledge libraries: feature learning library and model knowledge library, called Flexible Clustered Lifelong Learning (FCL3)
0
0
0
Mon Jan 25 2021
Machine Learning
Online Continual Learning in Image Classification: An Empirical Survey
Online continual learning for image classification studies the problem of forgetting old tasks in the presence of more recent tasks. Many methods and tricks have been introduced to address this problem, but many have not been fairly and systematically compared under a variety of realistic and practical settings. iCaRL remains competitive when the memory buffer is small; GDumb outperforms many recently proposed methods in medium-size datasets and
4
1
0
Thu Nov 21 2019
Machine Learning
A Conceptual Framework for Lifelong Learning
Humans can learn a variety of concepts and skills incrementally over the course of their lives. Several lines of machine learning research attempt to capture these properties. We propose a simple yet powerful unified framework that supports almost all of these properties and approaches through one central mechanism.
0
0
0