Published on Wed Nov 13 2019

Anomaly Detection in Large Scale Networks with Latent Space Models

Wesley Lee, Tyler H. McCormick, Joshua Neil, Cole Sodja, Yanran Cui

We develop a real-time anomaly detection algorithm for directed activity on large, sparse networks. We model the propensity for future activity using a dynamic logistic model. We run our algorithm on network event records collected from an enterprise network of over 25,000 computers.

0
0
0
Abstract

We develop a real-time anomaly detection algorithm for directed activity on large, sparse networks. We model the propensity for future activity using a dynamic logistic model with interaction terms for sender- and receiver-specific latent factors in addition to sender- and receiver-specific popularity scores; deviations from this underlying model constitute potential anomalies. Latent nodal attributes are estimated via a variational Bayesian approach and may change over time, representing natural shifts in network activity. Estimation is augmented with a case-control approximation to take advantage of the sparsity of the network and reduces computational complexity from to , where is the number of nodes and is the number of observed edges. We run our algorithm on network event records collected from an enterprise network of over 25,000 computers and are able to identify a red team attack with half the detection rate required of the model without latent interaction terms.