Feedback Loops
Feedback Loops. One of the best-known historical examples to illustrate destabilising dynamics in the context of autonomous agents is the 2010 flash crash, in which algorithmic trading agents entered into an unexpected feedback loop (Commission & Commission, 2010, see also Case Study 10).37 More generally, a feedback loop occurs when the output of a system is used as part of its input, creating a cycle that can either amplify or dampen the system’s behaviour. In multi-agent settings, feedback loops often arise from the interactions between agents, as each agent’s actions affect the environment and the behaviour of other agents, which in turn affect their own subsequent actions. Feedback loops can lead not only to financial crashes but to military conflicts (Richardson, 1960, see also ??) and ecological disasters (Holling, 1973).
ENTITY
2 - AI
INTENT
2 - Unintentional
TIMING
2 - Post-deployment
Risk ID
mit1230
Domain lineage
7. AI System Safety, Failures, & Limitations
7.6 > Multi-agent risks
Mitigation strategy
1. Prioritize Dynamic and Longitudinal System Evaluation: Implement systematic dynamic testing, such as multi-round simulations or live A/B testing, that explicitly models the feedback loop's effect across retraining cycles to detect bias amplification or destabilizing emergent behaviors. This must be complemented by continuous, real-time monitoring and runtime safety enforcement (guardrails) to prevent the manifestation of hazardous outputs in production. 2. Employ Robust Design and Data Controls: Redesign the AI system architecture and training regimen to mitigate endogenous feedback. This involves ensuring the training data for successive iterations is diverse and fully representative, utilizing bias-aware algorithms or causal inference-based techniques that explicitly model and decouple the system's own actions/predictions from future inputs, and establishing non-engagement signals for ground-truth feedback. 3. Integrate Human and Ethical Oversight Mechanisms: Establish a mandatory Human-in-the-Loop (HITL) protocol, particularly in critical decision-making workflows, supported by diverse cross-functional teams for expert validation and ethical review. Furthermore, prioritize transparency and explainability to allow human operators to trace model predictions back to their data and logic origins, facilitating timely identification and correction of unintended system drift.