Privacy and data collection concerns (data protection concerns)
The incorporation of personal data within training datasets raises numerous concerns. The primary issue is that personal data may be incorporated without the knowledge or consent of the individuals concerned, even though the data may include names, identification numbers, Social Security numbers, or other personal information. Another particularly difficult problem is related to the fact that complex models may “memorize” (i.e., store) specific threads of training data and regurgitate them when responding to a prompt.498 This data memorization can directly lead to leakage of personal data. Even if generative AI models do not memorize or leak personal data, they make it possible to recognize patterns or information structures that could enable malicious users to uncover personal details.
ENTITY
2 - AI
INTENT
2 - Unintentional
TIMING
2 - Post-deployment
Risk ID
mit746
Domain lineage
2. Privacy & Security
2.1 > Compromise of privacy by leaking or correctly inferring sensitive information
Mitigation strategy
1. Implement rigorous data provenance and engineering protocols, specifically including pre-training PII filtering (anonymization/pseudonymization) and comprehensive data deduplication to reduce the likelihood of personal information incorporation and subsequent verbatim memorization. 2. Integrate advanced privacy-preserving machine learning techniques, such as Differential Privacy or specialized loss functions (e.g., Goldfish Loss), during model training to formally restrict the model's capacity to memorize specific data points and to obfuscate individual contribution to the final model output. 3. Establish robust post-deployment security measures, including strong, role-based access controls (RBAC) to enforce the principle of least privilege, and deploy real-time output filtering and Data Loss Prevention (DLP) tools to detect and block the regurgitation or exfiltration of sensitive information.