From feature importance to natural language explanations using LLMs with RAG

Tekkesinoglu, Sule; Kunze, Lars

Abstract

As machine learning becomes increasingly integral to autonomous decision-making processes involving human interaction, the necessity of comprehending the model's outputs through conversational means increases. Most recently, foundation models are being explored for their potential as post hoc explainers, providing a pathway to elucidate the decision-making mechanisms of predictive models. In this work, we introduce traceable question-answering, leveraging an external knowledge repository to inform the responses of Large Language Models (LLMs) to user queries within a scene understanding task. This knowledge repository comprises contextual details regarding the model's output, containing high-level features, feature importance, and alternative probabilities. We employ subtractive counterfactual reasoning to compute feature importance, a method that entails analysing output variations resulting from decomposing semantic features. Furthermore, to maintain a seamless conversational flow, we integrate four key characteristics-social, causal, selective, and contrastive-drawn from social science research on human explanations into a single-shot prompt, guiding the response generation process. Our evaluation demonstrates that explanations generated by the LLMs encompassed these elements, indicating its potential to bridge the gap between complex model outputs and natural language expressions.

• Presentation Conference Type: Conference Paper (published)
• Conference Name: Multimodal, Affective and Interactive eXplainable AI Workshop (ECAI 2024)
• Start Date: Oct 19, 2024
• End Date: Oct 24, 2024
• Acceptance Date: Jul 18, 2024
• Online Publication Date: Oct 25, 2024
• Publication Date: Oct 25, 2024
• Deposit Date: Apr 14, 2025
• Publicly Available Date: Apr 15, 2025
• Peer Reviewed: Peer Reviewed
• Pages: 114-132
• Book Title: Proceedings of the First Multimodal, Affective and Interactive eXplainable AI Workshop (MAI-XAI24 2024) co-located with 27th European Conference On Artificial Intelligence 19-24 October 2024 (ECAI 2024)
• Keywords: Explainable AI; LLMs; Subtractive counterfactual reasoning; Retrieval-Augmented Generation (RAG)
• Public URL: https://uwe-repository.worktribe.com/output/14304658
• Publisher URL: https://ceur-ws.org/Vol-3803/