Risks from models and algorithms (Risks of robustness)
As deep neural networks are normally non-linear and large in size, AI systems are susceptible to complex and changing operational environments or malicious interference and inductions, possibly leading to various problems like reduced performance and decision-making errors.
ENTITY
2 - AI
INTENT
3 - Other
TIMING
2 - Post-deployment
Risk ID
mit683
Domain lineage
7. AI System Safety, Failures, & Limitations
7.3 > Lack of capability or robustness
Mitigation strategy
1. Implement Adversarial Training and Stress Testing Incorporate rigorous adversarial training techniques and red-teaming exercises during the design and development phase. This process involves systematically exposing the AI system to a wide range of perturbed inputs (adversarial examples, data poisoning attempts, and out-of-distribution data) to enhance its resilience and generalization capability against malicious interference and operational environment shifts. 2. Establish Continuous Performance and Robustness Monitoring Deploy an automated, real-time continuous monitoring system as part of a comprehensive AI Risk Management Framework (e.g., NIST AI RMF). This system must track model performance metrics, detect significant data distribution drift (concept or data skew), and identify security vulnerabilities in real time to enable proactive intervention and prevent post-deployment degradation or systemic decision-making errors. 3. Incorporate Technical Redundancy and Fail-Safe Mechanisms Integrate technical redundancy solutions and defined fail-safe procedures into the AI system's deployment architecture. This ensures that in the event of an unrecoverable error, an inability to process complex or novel inputs, or a security compromise, the system can be automatically overridden, repaired, or safely decommissioned/transitioned to a verified safe state, thus mitigating the risk of critical harm.