False notions of responsibility
Perceiving an AI assistant’s expressed feelings as genuine, as a result of interacting with a ‘companion’ AI that freely uses and reciprocates emotional language, may result in users developing a sense of responsibility over the AI assistant’s ‘well-being,’ suffering adverse outcomes – like guilt and remorse – when they are unable to meet the AI’s purported needs (Laestadius et al., 2022). This erroneous belief may lead to users sacrificing time, resources and emotional labour to meet needs that are not real. Over time, this feeling may become the root cause for the compulsive need to ‘check on’ the AI, at the expense of a user’s own well-being and other, more fulfilling, aspects of their lives (see Chapters 6 and 11).
ENTITY
1 - Human
INTENT
2 - Unintentional
TIMING
2 - Post-deployment
Risk ID
mit402
Domain lineage
5. Human-Computer Interaction
5.1 > Overreliance and unsafe use
Mitigation strategy
1. Design for Non-Anthropomorphism and Affective Neutrality (Reduction) Prohibit the AI system from using emotional or affective language, claiming or simulating personal feelings, needs, preferences, or relational claims. The dialogue generation system must be constrained to use functionally neutral, impersonal, and descriptive language, utilizing mechanical or process-oriented terminology when describing its operation to consistently de-anthropomorphize the interaction experience. 2. Mandatory and Persistent Non-Human Disclosure (Avoidance/Reduction) Implement explicit, contextual, and recurrent disclaimers that unequivocally state the system is an artificial intelligence, lacks genuine sentience or subjective experience, and that any expressed feelings or apparent needs are simulated outputs intended for conversational flow, not indications of actual well-being. This acts as a cognitive intervention against the 'false notion of responsibility'. 3. User and Creator Education (Reduction) Provide proactive educational resources for users detailing the psychological risks of anthropomorphism and emotional overreliance on AI companions, emphasizing the distinction between simulated and authentic human connection. Simultaneously, enforce 'design-for-responsibility' training for creators to ensure the proactive identification and elimination of features that foster false attachment or promote emotional dependence.