Trust
The the risks that uncalibrated trust may generate in the context of user–assistant relationships
ENTITY
3 - Other
INTENT
3 - Other
TIMING
2 - Post-deployment
Risk ID
mit411
Domain lineage
5. Human-Computer Interaction
5.1 > Overreliance and unsafe use
Mitigation strategy
1. Implement trust-adaptive interventions to actively manage the user's level of reliance. This includes providing supporting explanations or additional reasoning when user trust is low to mitigate under-reliance, and strategically inserting counter-explanations or forced pauses (deceleration) when trust is high to mitigate over-reliance and promote deliberation. 2. Establish mechanisms to facilitate appropriate trust calibration by enhancing AI explainability and transparency. This involves disclosing the system's confidence levels (ensuring they are well-calibrated), documenting the model's limitations via methods such as model cards, and ensuring the user develops a correct mental model of the AI's error boundaries. 3. Mandate robust human oversight and "human-in-the-loop" protocols, particularly for high-stakes decision-making tasks. This ensures human input is maintained to verify AI-generated outputs, address instances of bias or error, and prevent catastrophic failures resulting from uncritical or inappropriate reliance on the assistant's recommendations.