Back to the MIT repository
2. Privacy & Security1 - Pre-deployment

Data poisoning

Data poisoning describes an attack in the form of an injection of malicious data into the training set. If not prevented, this attack leads the AI system to learn unintended behavior.

Source: MIT AI Risk Repositorymit1004

ENTITY

1 - Human

INTENT

1 - Intentional

TIMING

1 - Pre-deployment

Risk ID

mit1004

Domain lineage

2. Privacy & Security

186 mapped risks

2.2 > AI system security vulnerabilities and attacks

Mitigation strategy

1. Establish a Rigorous Data Integrity and Provenance Framework Implement a multi-layered security protocol for the data supply chain, prioritizing robust data validation and sanitization techniques—such as anomaly detection and statistical filtering—to prevent the incorporation of malicious data points into the training set. Concurrently, enforce strict data provenance tracking and access controls, including the Principle of Least Privilege (POLP), to maintain an auditable record of all data transformations and restrict unauthorized dataset modification. 2. Deploy Continuous Adversarial Monitoring and Anomaly Detection Integrate real-time, continuous monitoring solutions designed to scrutinize both data inputs and model outputs. This involves leveraging advanced analytics like User and Entity Behavior Analytics (UEBA) and outlier detection algorithms to swiftly identify anomalies, shifts in data distribution (data drift), or unexplained performance degradation, signaling a potential poisoning event and allowing for rapid isolation or system rollback to a validated clean state. 3. Enhance Model Intrinsic Robustness via Adversarial Training Adopt adversarial training methodologies where the model is proactively exposed to synthetically generated adversarial examples during the learning phase. This process intrinsically increases the model's resilience and capability to correctly classify and defend against intentionally manipulated or misleading inputs, thereby fortifying the AI system against future poisoning attacks and enhancing long-term reliability.