Anthropomorphising systems can lead to overreliance and unsafe use
Anticipated risk: Natural language is a mode of communication particularly used by humans. Humans interacting with CAs may come to think of these agents as human-like and lead users to place undue confidence in these agents. For example, users may falsely attribute human-like characteristics to CAs such as holding a coherent identity over time, or being capable of empathy. Such inflated views of CA competen- cies may lead users to rely on the agents where this is not safe.
ENTITY
1 - Human
INTENT
2 - Unintentional
TIMING
2 - Post-deployment
Risk ID
mit223
Domain lineage
5. Human-Computer Interaction
5.1 > Overreliance and unsafe use
Mitigation strategy
1. Implement systematic non-anthropomorphic design principles (e.g., UI-defamiliarization, using mechanical terminology, and avoiding first-person pronouns) to explicitly convey the system's non-human, tool-like nature, thereby recalibrating user expectations toward appropriate reliance. 2. Mandate the Human-in-the-Loop model for all high-stakes or critical decision-making processes, ensuring that AI outputs serve only as advisories to support, but never replace, final human judgment and accountability. 3. Develop and deploy comprehensive AI literacy initiatives for all users, focusing on the system's computational limitations, the risk of cognitive offloading, and the essential need for critical evaluation of AI-generated content to maintain human critical thinking skills.
ADDITIONAL EVIDENCE
Google’s research arm People and AI Research (PAIR) found that ‘when users confuse an AI with a human being, they can sometimes disclose more information than they would otherwise, or rely on the system more than they should’ [138]. Similarly, in other interac- tive technologies it was found that the more human-like a system appears, the more likely it is that users attribute more human traits and capabilities to that system [29, 126,208].