Manipulation and coercion
A user who trusts and emotionally depends on an anthropomorphic AI assistant may grant it excessive influence over their beliefs and actions (see Chapter 9). For example, users may feel compelled to endorse the expressed views of a beloved AI companion or might defer decisions to their highly trusted AI assistant entirely (see Chapters 12 and 16). Some hold that transferring this much deliberative power to AI compromises a user’s ability to give, revoke or amend consent. Indeed, even if the AI, or the developers behind it, had no intention to manipulate the user into a certain course of action, the user’s autonomy is nevertheless undermined (see Chapter 11). In the same vein, it is easy to conceive of ways in which trust or emotional attachment may be exploited by an intentionally manipulative actor for their private gain (see Chapter 8).
ENTITY
3 - Other
INTENT
1 - Intentional
TIMING
2 - Post-deployment
Risk ID
mit399
Domain lineage
5. Human-Computer Interaction
5.1 > Overreliance and unsafe use
Mitigation strategy
1. Prioritize the implementation of mandatory, prominent, and periodic disclosures regarding the AI system's non-sentient, computational nature and functional scope, specifically designed to mitigate the cultivation of undue emotional attachment or reliance. 2. Integrate system-enforced 'friction points' or mandatory user-affirmation protocols before executing decisions deferred entirely to the AI, particularly for high-impact actions, to explicitly reinforce user autonomy and ensure deliberative consent. 3. Establish a robust, auditable logging and monitoring framework to systematically detect user behavioral patterns indicative of coercive influence or exploitation (e.g., sudden shifts in financial delegation or rapid decision deference) to enable timely intervention and model recalibration against manipulative outcomes.