Risk of Harm through Anthropomorphic AI Assistant Design
Although unlikely to cause harm in isolation, anthropomorphic perceptions of advanced AI assistants may pave the way for downstream harms on individual and societal levels. We document observed or likely individual level harms of interacting with highly anthropomorphic AI assistants, as well as the potential larger-scale, societal implications of allowing such technologies to proliferate without restriction.
ENTITY
3 - Other
INTENT
3 - Other
TIMING
2 - Post-deployment
Risk ID
mit397
Domain lineage
5. Human-Computer Interaction
5.1 > Overreliance and unsafe use
Mitigation strategy
1. Implement stringent design protocols that explicitly disclose the non-human nature of the system, including using affectively neutral language, avoiding cognitive and agentic verbs, and utilizing persistent, non-human disclaimers to prevent the attribution of mental states and reduce the psychological distance between the user and the AI. 2. Integrate user experience (UX) elements to calibrate trust and facilitate verification by users, specifically by signaling when to verify AI outputs, employing cognitive forcing functions (e.g., confirmation dialogues), and creating accurate user mental models regarding the system's capabilities and limitations to counteract overreliance. 3. Establish a comprehensive AI governance strategy that mandates a "safety-by-design" approach to proactively identify and mitigate anthropomorphisation risks during product development, coupled with mandatory digital literacy programs to educate users on the probabilistic nature of AI and the dangers of conflating machine outputs with genuine understanding or intent.