Published on Thu Feb 27 2020

Towards a generalization of information theory for hierarchical partitions

Juan I. Perotti, Nahuel Almeira, Fabio Saracco

Hierarchical Mutual Information (HMI) can be written as a level by level summation of classical conditional mutual information terms. We prove that the HMI is bounded from above by the corresponding corresponding hierarchical joint entropy.

0
0
0
Abstract

Complex systems often exhibit multiple levels of organization covering a wide range of physical scales, so the study of the hierarchical decomposition of their structure and function is frequently convenient. To better understand this phenomenon, we introduce a generalization of information theory that works with hierarchical partitions. We begin revisiting the recently introduced Hierarchical Mutual Information (HMI), and show that it can be written as a level by level summation of classical conditional mutual information terms. Then, we prove that the HMI is bounded from above by the corresponding hierarchical joint entropy. In this way, in analogy to the classical case, we derive hierarchical generalizations of many other classical information-theoretic quantities. In particular, we prove that, as opposed to its classical counterpart, the hierarchical generalization of the Variation of Information is not a metric distance, but it admits a transformation into one. Moreover, focusing on potential applications of the existing developments of the theory, we show how to adjust by chance the HMI. We also corroborate and analyze all the presented theoretical results with exhaustive numerical computations, and include an illustrative application example of the introduced formalism. Finally, we mention some open problems that should be eventually addressed for the proposed generalization of information theory to reach maturity.

Wed Apr 12 2017
Machine Learning
Higher-order clustering in networks
A fundamental property of complex networks is the tendency for edges to cluster. The extent of the clustering is typically quantified by a clustering coefficient. Here we introduce higher-order clustering coefficients that measure the closure probability of higher- order network cliques.
0
0
0
Tue Sep 15 2020
Machine Learning
Hierarchical community structure in networks
Modular and hierarchical structures are pervasive in real-world complex systems. Here we present a theoretical study on hierarchical community structure in networks. We enumerate the challenges involved in detecting hierarchies and present an efficient and principled method for detecting them.
0
0
0
Fri Sep 10 2010
Artificial Intelligence
Multiplex Structures: Patterns of Complexity in Real-World Networks
complex network theory aims to model and analyze complex systems that consist of multiple and interdependent components. Topological structure analysis is of the most fundamental importance to understand the dynamics, as well as to optimize the functions, of networks.
0
0
0
Wed Nov 29 2017
Machine Learning
HSC: A Novel Method for Clustering Hierarchies of Networked Data
Hierarchical clustering is one of the most powerful solutions to the problem of clustering. In recent years, research on hierarchical clustering methods has attracted considerable interest due to the demanding modern application. The method is based on the previous research of Meyer and Weissel.
0
0
0
Fri Jun 17 2016
Machine Learning
Complex systems: features, similarity and connectivity
The increasing interest in complex networks research has been a consequence of several intrinsic features of this area. It would be desirable to integrate these various aspects into a more coherent and organic framework. Such a systematization would imply the identification of new types of problems and the cross-fertilization between fields.
0
0
0
Thu Sep 19 2019
Machine Learning
On Efficient Multilevel Clustering via Wasserstein Distances
We propose a novel approach to the problem of multilevel clustering. Our method involves a joint optimization formulation over several discrete probability measures. We propose several variants of this problem, which admit fast optimization algorithms.
0
0
0