Creating avenues for exploiting user trust, nudging or manipulation
In conversation, users may reveal private information that would otherwise be difficult to access, such as thoughts, opinions, or emotions. Capturing such information may enable downstream applications that violate privacy rights or cause harm to users, such as via surveillance or the creation of addictive applications.
ENTITY
3 - Other
INTENT
2 - Unintentional
TIMING
2 - Post-deployment
Risk ID
mit251
Domain lineage
5. Human-Computer Interaction
5.1 > Overreliance and unsafe use
Mitigation strategy
1. Enforce Contextual Privacy and Data Minimization Implement a real-time framework to analyze user inputs, identify sensitive information that is non-essential or out-of-context for the user's explicit task, and proactively guide the user to reformulate the prompt to adhere to the principle of data minimization. 2. Ensure Transparency and Manage Agent Persona Clearly and continuously disclose the AI nature of the conversational agent (CA) to manage user expectations, prevent the establishment of misplaced human-like trust, and mitigate the risk of users inadvertently oversharing due to a perceived absence of social stigma. 3. Deploy Sensitive Data Redaction and Encryption Utilize pre- and post-processing filters, fine-tuning, or a Data Loss Prevention (DLP) solution to detect and redact Personally Identifiable Information (PII) and non-essential sensitive attributes from user inputs prior to processing and storage. Furthermore, enforce encryption and strong access controls for all data at rest and in transit.
ADDITIONAL EVIDENCE
This risk is more likely to occur where users take the conversational agent (CA) to be human-like, and are more inclined to bestow a level of trust upon it that is akin to the trust placed in human counterparts. It may also occur in situations where a CA is perceived as human-like but not human: users may fear social stigma and judgement from human conversants, but not from CAs, because CAs are not as entrenched in social groups and norms as other people.