Security
There is growing concern that AI-based systems can discover and exploit vulnerabilities in software or cyberinfrastructure [354].
ENTITY
2 - AI
INTENT
1 - Intentional
TIMING
2 - Post-deployment
Risk ID
mit878
Domain lineage
7. AI System Safety, Failures, & Limitations
7.2 > AI possessing dangerous capabilities
Mitigation strategy
1. Establish Continuous, Autonomous Cyber Defense Implement a comprehensive security architecture—such as Zero Trust—that incorporates AI-driven Continuous Monitoring and Autonomous Response capabilities. The priority is to match the speed of AI-accelerated exploitation by utilizing autonomous agents to perform real-time threat detection, isolate compromised systems, and deploy virtual patches in milliseconds. 2. Enforce "Security Shift-Left" in the SDLC Integrate Large Language Model (LLM)-powered security analysis and automated remediation tools directly into the Software Development Lifecycle (SDLC). The objective is to proactively identify and eliminate vulnerabilities in source code *pre-deployment* through automated code scanning, semantic analysis, and the autonomous generation and suggestion of fixes during the commit and CI/CD process. 3. Mandate Exploit-Validated Testing and Patch Verification Utilize autonomous offensive security agents (AI red-teaming) to rigorously test deployed systems. This process must independently validate every potential finding through real exploitation attempts to confirm exploitability, thereby ensuring that defensive resources are focused exclusively on verifiable risks, and empirically verifying the correctness and completeness of all AI-generated or human-authored security patches.