Abstract

Trust calibration is essential in AI-assisted decision-making. If human users understand the rationale on which an AI model has made a prediction, they can decide whether they consider this prediction reasonable. Especially in high-risk tasks such as mushroom hunting (where a wrong decision may be fatal), it is important that users make correct choices to trust or overrule the AI. Various explainable AI (XAI) methods are currently being discussed as potentially useful for facilitating understanding and subsequently calibrating user trust. So far, however, it remains unclear which approaches are most effective. In this article, the effects of XAI methods on human AI-assisted decision-making in the high-risk task of mushroom picking were tested. For that endeavor, the effects of (i) Grad-CAM attributions, (ii) nearest-neighbor examples, and (iii) network-dissection concepts were compared in a between-subjects experiment with N=501 participants representing end-users of the system. In general, nearest-neighbor examples improved decision correctness the most. However, varying effects for different task items became apparent. All explanations seemed to be particularly effective when they revealed reasons to (i) doubt a specific AI classification when the AI was wrong and (ii) trust a specific AI classification when the AI was correct. Our results suggest that well-established methods, such as Grad-CAM attribution maps, might not be as beneficial to end users as expected and that XAI techniques for use in real-world scenarios must be chosen carefully.

Citation

BibTeX

@article{,
    title = {Reassuring, Misleading, Debunking: Comparing Effects of XAI Methods on Human Decisions},
    author = {Christina Humer and Andreas Hinterreiter and Benedikt Leichtmann and Martina Mara and Marc Streit},
    journal = {ACM Transactions on Interactive Intelligent Systems},
    doi = {10.1145/3665647},
    url = {https://doi.org/10.1145/3665647},
    volume = {14},
    number = {3},
    month = {September},
    year = {2024}
}

Acknowledgements

We thank Moritz Heckmann for helping with the implementation of the AI Forest—The Schwammerl Hunting Game and Stefan Eibelwimmer for the graphic design of the game. We thank Dr. Otto Stoik, the members of the Mycological Working Group (MYAG) at the Biology Center Linz, Austria, and the German Mycological Society (DGfM) for providing mushroom images for this study. Finally, we thank Alfio Ventura for helping with the study setup.