Published on Sun Apr 11 2021

ALT-MAS: A Data-Efficient Framework for Active Testing of Machine Learning Algorithms

Huong Ha, Sunil Gupta, Santu Rana, Svetha Venkatesh
0
0
0
Abstract

Machine learning models are being used extensively in many important areas, but there is no guarantee a model will always perform well or as its developers intended. Understanding the correctness of a model is crucial to prevent potential failures that may have significant detrimental impact in critical application areas. In this paper, we propose a novel framework to efficiently test a machine learning model using only a small amount of labeled test data. The idea is to estimate the metrics of interest for a model-under-test using Bayesian neural network (BNN). We develop a novel data augmentation method helping to train the BNN to achieve high accuracy. We also devise a theoretic information based sampling strategy to sample data points so as to achieve accurate estimations for the metrics of interest. Finally, we conduct an extensive set of experiments to test various machine learning models for different types of metrics. Our experiments show that the metrics estimations by our method are significantly better than existing baselines.

Wed Dec 11 2019
Machine Learning
Callisto: Entropy based test generation and data quality assessment for Machine Learning Systems
CALLISTO is the first blackbox framework to leverage the uncertainty in the prediction and generate new test cases for ML classifiers. It has the capability to detect low quality data in the datasets that may contain mislabelled data.
0
0
0
Wed Jul 08 2020
Machine Learning
URSABench: Comprehensive Benchmarking of Approximate Bayesian Inference Methods for Deep Neural Networks
Deep learning methods continue to improve in predictive accuracy on a wide range of application domains. However, significant issues remain with their ability to quantify uncertainty and their robustness. We describe initial work on the development ofURSABench.
0
0
0
Tue Dec 29 2020
Machine Learning
Uncertainty-Wizard: Fast and User-Friendly Neural Network Uncertainty Quantification
Uncertainty and confidence have been shown to be useful metrics in a wide variety of techniques proposed for deep learning testing. We present uncertainty-wizard, a tool that allows to quantify such uncertainty and confidence in artificial neural networks.
0
0
0
Tue May 05 2020
Machine Learning
Interpreting Deep Models through the Lens of Data
The classifier is a combination of both the data and the model, therefore, it is essential to also analyze these influences for the interpretability of deep learning models. Analysis of the results shows that some interpretability methods can detect mislabels better than using a random approach.
0
0
0
Mon Apr 01 2019
Machine Learning
Adaptive Bayesian Linear Regression for Automated Machine Learning
The goal of automated machine learning (AutoML) is to design methods that can automatically perform model selection and hyperparameter optimization without human interventions for a given dataset. The method combines an adaptive Bayesian regression model with a neural network function and the acquisition function from Bayesian optimization.
0
0
0
Tue Feb 11 2020
Machine Learning
Debugging Machine Learning Pipelines
Machine learning tasks entail the use of complex computational pipelines. If some of the activities in a pipeline produce erroneous or uninformative outputs, the pipeline may fail or produce incorrect results. Inferring the root cause of failures and unexpected behavior is challenging. We propose a new approach that makes use
0
0
0