Influence, overreliance and dependence (influence and manipulation)
Despite the widely recognized potential of generative AI tools to “hallucinate” or produce harmful content, such tools can exert a noteworthy influence on the humans who engage with them. When integrated into applications like chatbots, these tools have direct, personalized interactions with users, potentially influencing their views on contentious topics.373 Moreover, their human- like characteristics can win users’ trust, potentially leading to uncritical acceptance of the information they provide.374 Interactions with these seemingly human- like AI models may also encourage users to share more personal information, enabling even more targeted content.
ENTITY
2 - AI
INTENT
2 - Unintentional
TIMING
2 - Post-deployment
Risk ID
mit739
Domain lineage
5. Human-Computer Interaction
5.1 > Overreliance and unsafe use
Mitigation strategy
1. Implement Human-in-the-Loop (HITL) Protocols Establish mandatory human oversight for all critical decision-making processes, ensuring that AI-generated insights serve only as recommendations that must be reviewed, contextualized, and approved by a responsible human agent. This directly counters the risk of uncritical acceptance and potential for severe harm resulting from unchecked AI outputs. 2. Mandate User Trust Calibration and Transparency Deploy explicit user experience (UX) mechanisms, such as uncertainty expressions and cognitive forcing functions, to create realistic user mental models regarding the system's capabilities and limitations (e.g., potential for "hallucination"). Systems must clearly disclose their AI involvement to prevent anthropomorphism and misplaced trust. 3. Conduct Targeted Anti-Automation Bias Training Develop and implement rigorous training programs for all end-users to enhance AI literacy, focusing specifically on recognizing the indicators of automation bias, overconfidence, and the risks associated with sharing excessive personal information with AI interfaces, thereby fostering appropriate reliance.