Explainable AI-Based Dashboards for Data-Driven Decision Making

Abstract:

Operational dashboards increasingly surface machine learning predictions to support high-stakes decisions, but a dashboard that shows only a prediction and confidence score gives decision-makers no way to distinguish a confidently correct output from a confidently wrong one. Decades of research on automation bias and algorithm aversion show that this blindness causes both over-reliance on incorrect recommendations and unwarranted override of correct ones. This paper proposes a self-calibrating explainable-AI dashboard architecture that combines a predictive model with a layered explanation stack, global feature-importance summaries, local instance-level attributions, counterfactual recommendations, and synthesized natural-language rationale, coupled to an adaptive trust-calibration layer that monitors decision-maker reliance patterns against ground-truth outcomes and adjusts explanation presentation accordingly. Drawing on established explainable-AI and human-AI decision-making literature, the study defines seven evaluation criteria — Decision Accuracy, Explanation-Guided Precision, Appropriate Reliance Recall, F1 Score, Automation Bias Rate, trust-calibration-layer AUC-ROC, and a combined Decision Support Reliability Score — and applies them to an illustrative case study of a high-stakes operational review dashboard. A five-configuration ablation study, spanning a score-only baseline through a deliberately over-engineered explanation stack, is used to identify the point of diminishing, and even negative, returns from additional explanation detail, and the recommended configuration is benchmarked directly against the score-only baseline, including performance under a stress test of model-error edge cases. The illustrative results indicate substantially higher recall, F1 Score, and a markedly lower automation bias rate for the proposed dashboard relative to the baseline, while a configuration that maximizes explanation detail without adaptive filtering performs measurably worse than the proposed configuration, indicating a genuine information-overload ceiling. The findings position adaptive, reliance-aware explanation as a foundational design practice for organizations deploying AI-based dashboards in consequential, data-driven decision-making contexts.