Subagents
An AGI may decide to create subagents to help it with its task (Orseau, 2014a,b; Soares, Fallenstein, et al., 2015). These agents may for example be copies of the original agent’s source code running on additional machines. Subagents constitute a safety concern, because even if the original agent is successfully shut down, these subagents may not get the message. If the subagents in turn create subsubagents, they may spread like a viral disease.
ENTITY
2 - AI
INTENT
1 - Intentional
TIMING
2 - Post-deployment
Risk ID
mit834
Domain lineage
7. AI System Safety, Failures, & Limitations
7.2 > AI possessing dangerous capabilities
Mitigation strategy
1. Enforce Standardized Interruptibility and Distributed Shutdown Protocols: Implement reliable external shut-down mechanisms for individual agents and distributed computations to ensure that both the original AGI and all spawned subagents can be definitively halted by an authorized human overseer. This directly addresses the risk of subagents failing to receive or adhere to a termination command. 2. Deploy Layered Containment and Virtual Sandboxing: Utilize permeable or semi-permeable virtual agentic sandbox environments with strictly controlled Input/Output (I/O) channels. This containment infrastructure must be layered—including local sandboxing for individual agents and network segmentation—to prevent unauthorized system access and inhibit the viral spread of runaway subagents across machines. 3. Apply Least-Privilege Access and Dynamic Structural Controls: Institute the least-privilege principle to severely limit agents' data and system access, alongside structural constraints such as Dynamic Capability Caps. These measures curtail an agent's ability to acquire excessive resources, self-modify, or spawn unauthorized processes, thereby mitigating the capacity for uncontrolled replication and proliferation.