Governance of autonomous intelligence systems
Governance of autonomous intelligence systemaddresses the question of how to control autonomous systems in general. Since nowadays it is very difficult to conceive automated decisions based on AI, the latter is often referred to as a ‘black box’ (Bleicher, 2017). This black box may take unforeseeable actions and cause harm to humanity.
ENTITY
3 - Other
INTENT
3 - Other
TIMING
2 - Post-deployment
Risk ID
mit322
Domain lineage
6. Socioeconomic and Environmental
6.5 > Governance failure
Mitigation strategy
1. Establish Mandatory Human-in-the-Loop Governance and Override Mechanisms. Implement a structured human-in-the-loop (HITL) framework requiring mandatory human review and sign-off for all critical or high-stakes autonomous decisions. Crucially, embed non-negotiable override controls and "kill switches" to enable immediate human intervention and cessation of operation when an autonomous system exhibits behavior that deviates from safety protocols, ethical constraints, or established intent, thereby preventing catastrophic harm. 2. Mandate Explainable AI (XAI) and Comprehensive Audit Trails. Implement Explainable AI (XAI) techniques, such as SHAP or LIME, to ensure that the decision-making processes of autonomous "black box" systems are fully transparent and interpretable. Maintain robust, immutable audit trails logging all inputs, outputs, model versions, and internal metrics to facilitate post-hoc analysis, regulatory compliance, legal accountability, and the identification of algorithmic bias or failure modes. 3. Deploy Continuous Real-time Behavioral Monitoring. Establish continuous, real-time monitoring of the autonomous system's performance, behavior, and environment. Define and track deviations from established behavioral baselines, ethical guardrails, and key performance indicators to enable the early detection of model drift, anomalies, or unintended emergent behavior, triggering automated alerts for immediate human investigation and remediation.
ADDITIONAL EVIDENCE
For instance, if an autonomous AI weapon system learned that it is necessary to prevent all threats to obtain security, it might also attack civilians or even children classified as armed by the opaque algorithm (Heyns, 2014).