Verifiability
In many applications of AI-based systems such as medical healthcare and military services, the lack of verification of code may not be tolerable... due to some characteristics such as the non-linear and complex structure of AI-based solutions, existing solutions have been generally considered “black boxes”, not providing any information about what exactly makes them appear in their predictions and decision-making processes.
ENTITY
2 - AI
INTENT
2 - Unintentional
TIMING
2 - Post-deployment
Risk ID
mit605
Domain lineage
7. AI System Safety, Failures, & Limitations
7.4 > Lack of transparency or interpretability
Mitigation strategy
1. Prioritize the deployment of intrinsically interpretable (white-box) models, such as generalized linear models or decision trees, in safety-critical and high-stakes application domains to ensure decisions are justifiable *a priori* and to minimize the reliance on post-hoc explanation methods. 2. Implement post-hoc Explainable AI (XAI) techniques, particularly SHAP (SHapley Additive exPlanations) and LIME (Local Interpretable Model-agnostic Explanations), to provide robust, instance-level, and global explanations for predictions generated by any opaque black-box models used in production. 3. Establish a stringent AI Governance framework that mandates end-to-end traceability, including logging of all model inputs, outputs, and version histories, to facilitate independent auditing, compliance with regulatory requirements (e.g., GDPR, EU AI Act), and continuous monitoring for bias and unexpected operational drift.