Anthropomorphising systems can lead to overreliance or unsafe use
...humans interacting with conversational agents may come to think of these agents as human-like. Anthropomorphising LMs may inflate users’ estimates of the conversational agent’s competencies...As a result, they may place undue confidence, trust, or expectations in these agents...This can result in different risks of harm, for example when human users rely on conversational agents in domains where this may cause knock-on harms, such as requesting psychotherapy...Anthropomorphisation may amplify risks of users yielding effective control by coming to trust conversational agents “blindly”. Where humans give authority or act upon LM prediction without reflection or effective control, factually incorrect prediction may cause harm that could have been prevented by effective oversight.
ENTITY
1 - Human
INTENT
2 - Unintentional
TIMING
2 - Post-deployment
Risk ID
mit250
Domain lineage
5. Human-Computer Interaction
5.1 > Overreliance and unsafe use
Mitigation strategy
1. **Implement Design Principles to Dehumanize the System and Calibrate Anthropomorphism**: Intentionally tune system design and language to reduce misleading anthropomorphic cues, and establish explicit system prompts to communicate the LLM's non-human nature, inherent limitations, and probabilistic reasoning to foster realistic user mental models and prevent inflated expectations of competence. 2. **Deploy Cognitive Forcing Functions (Friction) for Critical Outputs**: Introduce "positive friction" in the user interface, such as mandatory user confirmations, clarification questions, or uncertainty expressions (e.g., highlighting low-probability tokens), specifically before accepting high-stakes or potentially harmful generations (e.g., in domains like psychotherapy or coding), thereby compelling user reflection and effective human oversight. 3. **Establish a Continuous Observability Loop for Overreliance and Miscalibration**: Implement real-time monitoring and observability to detect behavioral patterns indicative of overreliance, such as uncritical acceptance of suggestions or high-volume usage in sensitive domains. Findings from this monitoring should be integrated into a continuous feedback loop to inform prompt engineering updates, RAG context hygiene, and model alignment retraining.