Privacy
The risk of loss or harm from leakage of personal information via the ML system.
ENTITY
2 - AI
INTENT
2 - Unintentional
TIMING
2 - Post-deployment
Risk ID
mit201
Domain lineage
2. Privacy & Security
2.1 > Compromise of privacy by leaking or correctly inferring sensitive information
Mitigation strategy
1. Implement Differential Privacy (DP) mechanisms, such as Differentially Private Stochastic Gradient Descent (DP-SGD) or the Laplace/Gaussian mechanisms, during the model training and inference phases to provide a rigorous, formal bound ($\\epsilon, \\delta$) on the privacy loss attributable to any single individual's data. 2. Apply the principle of data minimization by systematically evaluating feature requirements and implementing data expiration policies to ensure only the minimum necessary amount of identifying or sensitive data is collected, processed, and retained. 3. Conduct continuous Privacy Risk Assessments and Audits, leveraging techniques like Membership Inference Attacks (MIAs) to empirically quantify data leakage and identify high-risk training samples for targeted safeguarding strategies or model architecture adjustments.
ADDITIONAL EVIDENCE
Although we only focus on privacy in this section, we use the GDPR’s definition of personal data due to its broad coverage: “any information relating to an identified or identifiable natural person”.8 Privacy breaches often result from compromised databases [133] and may be mitigated with proper data governance and stewardship [152]. However, we wish to highlight privacy risks that are specific to ML systems. Although federated learning [164] has been proposed to avoid storing training data in a central location (avoiding the problem of compromised databases), it may still be possible to recover training examples from a model learned in this manner [77, 78]. Researchers have also demonstrated that information about the training data can be retrieved from an ML model [37, 70, 165], and in some cases, the training examples themselves can even be extracted [31]. Therefore, simply securing the training data is now insufficient