Uncertainty-Calibrated Explainable AI for Cybersecurity Alert Prioritization: Improving Detection Reliability and Human Decision-Making

Abstract

Security operations centres receive far more alerts than analysts can review, and the probabilities, rankings and explanations that machine learning attaches to those alerts are rarely checked for reliability. This article designs a data-driven analytics framework that makes alert scores mean what they say, carries uncertainty alongside them, and explains them in a way whose quality is measured, and it evaluates the framework on a transparent synthetic testbed: 1,200 alerts per day from eight families, a rare and drifting incident class, four asset-value tiers, a fixed review capacity of 12.5%, and a stylised analyst model. Raw scores from deep boosted trees ranked incidents well but under-predicted risk by a factor of five (mean predicted probability 0.84% against a true rate of 4.2%); every post-hoc calibrator cut the expected calibration error from 0.034 to between 0.006 and 0.009, drift raised it to 0.017, and a sliding-window recalibrator using delayed labels held it at 0.008. Calibration did not change a ranking by probability alone, but it changed the queue once probability was multiplied by asset loss: loss captured rose from 59.7% to 68.3%, Tier-3 incidents missed fell from 27.6% to 6.9%, and net value rose from 29.4 to 34.5 thousand USD per day. Conformal sets, ensemble spread and explanations behaved less favourably: spread did not signal drift, explanation stability (0.65) was unrelated to uncertainty, and in the analyst model uncertainty flags and explanations traded recall against precision without raising net value. In a 2,000-draw Monte Carlo analysis the calibrated queue beat the raw-score baseline in every draw, and Sobol indices attribute the gain mainly to loss scale and attack prevalence. All results describe the testbed, and the analyst findings are hypotheses for studies with analysts, not findings about them. A governance record and a four-phase roadmap are proposed.