Overreliance
If a user begins to excessively trust an LLM, this may cause them to develop an overreliance on the LLM. Overreliance can result in automation bias (Kupfer et al., 2023), and can cause errors of omission (user choosing not to verify the validity of a response) and errors of commission (user believing and acting on the basis of the LLM’s response, even if it contradicts their own knowledge) (Skitka et al., 1999). It can be particularly dangerous in domains where the user may lack relevant expertise to robustly scrutinize the LLM responses. This is particularly a source of risk for LLMs because LLMs can often generate plausible, yet incorrect or unfaithful, rationalizations of their actions (c.f. Section 3.4.10), which can mistakenly cause the user to develop the belief that LLM has the relevant expertise and has provided a valid response
ENTITY
1 - Human
INTENT
2 - Unintentional
TIMING
2 - Post-deployment
Risk ID
mit1497
Domain lineage
5. Human-Computer Interaction
5.1 > Overreliance and unsafe use
Mitigation strategy
1. Implement Model- and System-Level Trust Calibration Mechanisms This strategy requires developing models capable of accurately conveying epistemic uncertainty (e.g., via confidence scores or natural language hedges) and integrating interface design features that facilitate output verification. Key mechanisms include source attribution, factual consistency checks, and clearly labeling all AI-generated content to aid users in critically calibrating their reliance. 2. Integrate Cognitive Forcing Functions at Critical Junctures Apply system-level interventions, such as friction mechanisms (e.g., mandatory disclaimers, extra clicks, or confirmation dialogues) and priming statements, at points where the risk of overreliance and high-stakes error is maximal. These functions are designed to disrupt automation bias and compel users to slow down, actively scrutinize, and verify the LLM's output before committing to an action. 3. Mandate Comprehensive User Education and Accountability Frameworks Establish structured training and continuous education programs that focus on the inherent limitations, potential failure modes (e.g., hallucination, plausible rationalizations), and intended field-of-use restrictions of the LLM system. This must be coupled with defining clear user accountability protocols that emphasize the human-in-the-loop's ultimate responsibility for validating and acting upon AI-generated advice, especially in domains requiring specialized expertise.