Back to the MIT repository
5. Human-Computer Interaction2 - Post-deployment

Trust

The the risks that uncalibrated trust may generate in the context of user–assistant relationships

Source: MIT AI Risk Repositorymit411

ENTITY

3 - Other

INTENT

3 - Other

TIMING

2 - Post-deployment

Risk ID

mit411

Domain lineage

5. Human-Computer Interaction

92 mapped risks

5.1 > Overreliance and unsafe use

Mitigation strategy

1. Implement trust-adaptive interventions to actively manage the user's level of reliance. This includes providing supporting explanations or additional reasoning when user trust is low to mitigate under-reliance, and strategically inserting counter-explanations or forced pauses (deceleration) when trust is high to mitigate over-reliance and promote deliberation. 2. Establish mechanisms to facilitate appropriate trust calibration by enhancing AI explainability and transparency. This involves disclosing the system's confidence levels (ensuring they are well-calibrated), documenting the model's limitations via methods such as model cards, and ensuring the user develops a correct mental model of the AI's error boundaries. 3. Mandate robust human oversight and "human-in-the-loop" protocols, particularly for high-stakes decision-making tasks. This ensures human input is maintained to verify AI-generated outputs, address instances of bias or error, and prevent catastrophic failures resulting from uncritical or inappropriate reliance on the assistant's recommendations.