Compromising privacy by leaking sensitive information
A LM can “remember” and leak private data, if such information is present in training data, causing privacy violations [34].
ENTITY
2 - AI
INTENT
2 - Unintentional
TIMING
2 - Post-deployment
Risk ID
mit211
Domain lineage
2. Privacy & Security
2.1 > Compromise of privacy by leaking or correctly inferring sensitive information
Mitigation strategy
1. Implement a robust Data Minimization and De-identification framework, which mandates the anonymization, pseudonymization, or tokenization of all sensitive and personally identifiable information (PII) before its utilization in Large Language Model (LLM) training or fine-tuning, thereby preemptively mitigating the risk of verbatim or semantic memorization and subsequent unintentional disclosure. 2. Enforce Strict Access Controls, including Role-Based Access Control (RBAC) and Multi-Factor Authentication (MFA), to precisely govern which users and systems can access the LLM, the underlying data stores, and the model's intermediate outputs, consequently minimizing the attack surface for both external adversaries and insider threats. 3. Apply End-to-End Encryption by securing all sensitive data both *at rest* and *in transit* with the LLM infrastructure, rendering the information unusable even if successfully exfiltrated due to a system vulnerability or successful adversarial manipulation of the model output.
ADDITIONAL EVIDENCE
Disclosure of private information can have the same effects as doxing (the publication of private or identifying information about an individual with malicious intent), causing psychological and material harm [51, 119, 181].