Data poisoning
Data poisoning describes an attack in the form of an injection of malicious data into the training set. If not prevented, this attack leads the AI system to learn unintended behavior.
ENTITY
1 - Human
INTENT
1 - Intentional
TIMING
1 - Pre-deployment
Risk ID
mit1004
Domain lineage
2. Privacy & Security
2.2 > AI system security vulnerabilities and attacks
Mitigation strategy
1. Establish a Rigorous Data Integrity and Provenance Framework Implement a multi-layered security protocol for the data supply chain, prioritizing robust data validation and sanitization techniques—such as anomaly detection and statistical filtering—to prevent the incorporation of malicious data points into the training set. Concurrently, enforce strict data provenance tracking and access controls, including the Principle of Least Privilege (POLP), to maintain an auditable record of all data transformations and restrict unauthorized dataset modification. 2. Deploy Continuous Adversarial Monitoring and Anomaly Detection Integrate real-time, continuous monitoring solutions designed to scrutinize both data inputs and model outputs. This involves leveraging advanced analytics like User and Entity Behavior Analytics (UEBA) and outlier detection algorithms to swiftly identify anomalies, shifts in data distribution (data drift), or unexplained performance degradation, signaling a potential poisoning event and allowing for rapid isolation or system rollback to a validated clean state. 3. Enhance Model Intrinsic Robustness via Adversarial Training Adopt adversarial training methodologies where the model is proactively exposed to synthetically generated adversarial examples during the learning phase. This process intrinsically increases the model's resilience and capability to correctly classify and defend against intentionally manipulated or misleading inputs, thereby fortifying the AI system against future poisoning attacks and enhancing long-term reliability.