Published on Thu Jun 17 2021

Disentangling Identifiable Features from Noisy Data with Structured Nonlinear ICA

Hermanni Hälvä, Sylvain Le Corff, Luc Lehéricy, Jonathan So, Yongjie Zhu, Elisabeth Gassiat, Aapo Hyvarinen

We introduce a new general identifiable framework for principled disentanglement. We establish the major result that identifiability for this framework holds even in the presence of noise of unknown distribution. We introduce the first nonlinear ICA model for time-series that combines the following very useful properties.

0
0
0
Abstract

We introduce a new general identifiable framework for principled disentanglement referred to as Structured Nonlinear Independent Component Analysis (SNICA). Our contribution is to extend the identifiability theory of deep generative models for a very broad class of structured models. While previous works have shown identifiability for specific classes of time-series models, our theorems extend this to more general temporal structures as well as to models with more complex structures such as spatial dependencies. In particular, we establish the major result that identifiability for this framework holds even in the presence of noise of unknown distribution. The SNICA setting therefore subsumes all the existing nonlinear ICA models for time-series and also allows for new much richer identifiable models. Finally, as an example of our framework's flexibility, we introduce the first nonlinear ICA model for time-series that combines the following very useful properties: it accounts for both nonstationarity and autocorrelation in a fully unsupervised setting; performs dimensionality reduction; models hidden states; and enables principled estimation and inference by variational maximum-likelihood.

Thu May 23 2019
Machine Learning
Unsupervised Discovery of Temporal Structure in Noisy Data with Dynamical Components Analysis
Linear dimensionality reduction methods are commonly used to extract low-dimensional structure from high-dimensional data. We introduce Dynamical Components Analysis(DCA) which discovers a subspace of high-dimensional time series data with maximal predictive information. DCA robustly extracts dynamical structure in noisy,
0
0
0
Mon Jun 22 2020
Machine Learning
Hidden Markov Nonlinear ICA: Unsupervised Learning from Nonstationary Time Series
Recent advances in nonlinear Independent Component Analysis provide a principled framework for unsupervised feature learning and disentanglement. We remedy these shortcomings by combining nonlinear ICA with a Hidden Markov Model, resulting in a model where a latent state acts in place of the observed segment-index.
0
0
0
Mon Mar 04 2019
Machine Learning
Time Series Source Separation using Dynamic Mode Decomposition
The Dynamic Mode Decomposition (DMD) extracted dynamic modes are the non-orthogonal eigenvectors of the matrix that best approximates the one-step temporal evolution of the multivariate samples. DMD is a time series blind source separation algorithm in disguise.
0
0
0
Mon May 21 2012
Machine Learning
Forecastable Component Analysis (ForeCA)
I introduce Forecastable Component Analysis (ForeCA), a novel dimension reduction technique for temporally dependent signals. The R package ForeCA(http://cran.r-project.org/web/packages/ForeCA/index.html) accompanies this work.
0
0
0
Mon Jul 20 2020
Machine Learning
Time Series Source Separation with Slow Flows
Slow feature analysis (SFA), a common time series decomposition method, naturally fits into the flow-based models (FBM) framework. Building upon advances on blind source separation, we show that such a fit makes the time-series decomposition identifiable.
0
0
0
Thu May 30 2019
Machine Learning
Efficient Covariance Estimation from Temporal Data
Estimating the covariance structure of multivariate time series is a fundamental problem with a wide-range of real-world applications. Current state-of-the-art methods are still severely limited in terms of severely scalability. We propose a novel method called Temporal Correlation
0
0
0
Fri Nov 06 2020
Machine Learning
Underspecification Presents Challenges for Credibility in Modern Machine Learning
ML models often exhibit unexpectedly poor behavior when they are deployed in real-world domains. We identify underspecification as a key reason for these behaviors. This ambiguity can lead to instability and poor model behavior in practice.
10
277
1,080
Tue Jul 21 2020
Machine Learning
Towards Nonlinear Disentanglement in Natural Data with Temporal Sparse Coding
We construct an unsupervised learning model that achieves nonlinear disentanglement of underlying factors of variation in naturalistic videos. The model reliably learns disentangled representations on several established benchmark datasets, often surpassing the current state-of-the-art.
2
197
651
Wed Jul 10 2019
Machine Learning
Variational Autoencoders and Nonlinear ICA: A Unifying Framework
We build on recent developments in nonlinear ICA, which we extend to the case with noisy,undercomplete or discrete observations, integrated in a maximum likelihood framework. The result also trivially contains identifiable flow-based generative models as a special case.
1
25
134
Fri Dec 20 2013
Machine Learning
Auto-Encoding Variational Bayes
We introduce a stochastic variational inference and learning algorithm that scales to large datasets. We show that a reparameterization of the variational lower bound yields a lower bound estimator.
2
0
3
Thu Feb 04 2021
Machine Learning
Nonlinear Independent Component Analysis for Continuous-Time Signals
0
0
0
Tue May 23 2017
Machine Learning
Look, Listen and Learn
We consider the question: what can be learnt by looking at and listening to unlabelled videos? There is a valuable, but so far untapped,source of information contained in the video itself. We introduce a novel"Audio-Visual Correspondence" learning task that makes use of this.
0
0
0