Privacy Violation
machine learning models are known to be vulnerable to data privacy attacks, i.e. special techniques of extracting private information from the model or the system used by attackers or malicious users, usually by querying the models in a specially designed way
ENTITY
2 - AI
INTENT
1 - Intentional
TIMING
2 - Post-deployment
Risk ID
mit486
Domain lineage
2. Privacy & Security
2.1 > Compromise of privacy by leaking or correctly inferring sensitive information
Mitigation strategy
1. Implement Differential Privacy (DP) mechanisms, such as noise injection during training, to provide a quantifiable guarantee against the leakage of individual training data points, thereby mitigating the risk of memorization and membership inference attacks. 2. Employ Adversarial Training methodologies to enhance model robustness. This involves training the model on intentionally altered data to improve its resilience against targeted privacy threats, including Model Inversion and Data Reconstruction attacks. 3. Enforce foundational data security measures including robust encryption (data at rest and in transit), and stringent access control protocols such-as role-based access control and multi-factor authentication to prevent unauthorized system access, which is often a prerequisite for data exfiltration and privacy breaches.
ADDITIONAL EVIDENCE
The private information includes training data, training data property, instance's membership belonging to the training data, model weights, model architecture, and even the training hyperparameters. The memorization effect in deep neural network models make them even more vulnerable to privacy attacks than simple models.