Cyclic Behaviour
Cyclic Behaviour. The dynamics described above are highly non-linear (small changes to the system’s state can result in large changes to its trajectory). Similar non-linear dynamics can emerge in multi- agent learning and lead to a variety of phenomena that do not occur in single-agent learning (Barfuss et al., 2019; Barfuss & Mann, 2022; Galla & Farmer, 2013; Leonardos et al., 2020; Nagarajan et al., 2020). One of the simplest examples of this phenomenon is Q-learning (Watkins & Dayan, 1992): in the case of a single agent, convergence to an optimal policy is guaranteed under modest conditions, but in the (mixed-motive) case of multiple agents, this same learning rule can lead to cycles and thus non- convergence (Zinkevich et al., 2005). While cycles in themselves need not carry any risk, their presence can subvert the expected or desirable properties of a given system.
ENTITY
2 - AI
INTENT
2 - Unintentional
TIMING
2 - Post-deployment
Risk ID
mit1231
Domain lineage
7. AI System Safety, Failures, & Limitations
7.6 > Multi-agent risks
Mitigation strategy
1. Implement hierarchical goal decomposition and deterministic task allocation mechanisms to structure agent interactions, thereby decoupling non-linear interdependencies and localizing failures to prevent system-wide destabilizing cyclic behaviors. 2. Within multi-agent reinforcement learning algorithms, enforce sufficient exploration rates or utilize algorithms with provable convergence guarantees to ensure dynamics approach a stable equilibrium or a constrained neighborhood, mitigating the risk of perpetual non-convergent cycles. 3. Establish continuous, real-time behavior monitoring with anomaly detection across the multi-agent system to identify and halt emergent cyclical patterns or significant behavioral drift, utilizing safety filtering mechanisms to override high-risk, non-convergent action execution.