Part of the Institute of Computer Graphics @ Johannes Kepler University Linz, Austria

ConfusionFlow: A Model-Agnostic Visualization for Temporal Analysis of Classifier Confusion

ConfusionFlow Teaser

Abstract

Classifiers are among the most widely used supervised machine learning algorithms. Many classification models exist, and choosing the right one for a given task is difficult. During model selection and debugging, data scientists need to assess classifiers’ performances, evaluate their learning behavior over time, and compare different models. Typically, this analysis is based on single-number performance measures such as accuracy. A more detailed evaluation of classifiers is possible by inspecting class errors. The confusion matrix is an established way for visualizing these class errors, but it was not designed with temporal or comparative analysis in mind. More generally, established performance analysis systems do not allow a combined temporal and comparative analysis of class-level information. To address this issue, we propose ConfusionFlow, an interactive, comparative visualization tool that combines the benefits of class confusion matrices with the visualization of performance characteristics over time. ConfusionFlow is model-agnostic and can be used to compare performances for different model types, model architectures, and/or training and test datasets. We demonstrate the usefulness of ConfusionFlow in a case study on instance selection strategies in active learning. We further assess the scalability of ConfusionFlow and present a use case in the context of neural network pruning.


Citation

Andreas Hinterreiter, Peter Ruch, Holger Stitz, Martin Ennemoser, Jürgen Bernard, Hendrik Strobelt, Marc Streit
ConfusionFlow: A Model-Agnostic Visualization for Temporal Analysis of Classifier Confusion
IEEE Transactions on Visualization and Computer Graphics, 28(2): 1222-1236, doi:10.1109/TVCG.2020.3012063, 2022.

BibTeX

@article{,
    title = {ConfusionFlow: A Model-Agnostic Visualization for Temporal Analysis of Classifier Confusion},
    author = {Andreas Hinterreiter and Peter Ruch and Holger Stitz and Martin Ennemoser and Jürgen Bernard and Hendrik Strobelt and Marc Streit},
    journal = {IEEE Transactions on Visualization and Computer Graphics},
    doi = {10.1109/TVCG.2020.3012063},
    volume = {28},
    number = {2},
    pages = {1222-1236},
    year = {2022}
}

Acknowledgements

This work was supported in part by the State of Upper Austria (FFG 851460, Human-Interpretable Machine Learning) and the Austrian Science Fund (FWF P27975-NBL).