Back to the MIT repository
5. Human-Computer Interaction2 - Post-deployment

Diminished health & well-being

algorithmic behavioral exploitation [18, 209], emotional manipulation [202] whereby algorithmic designs exploit user behavior, safety failures involving algorithms (e.g., collisions) [67], and when systems make incorrect health inferences

Source: MIT AI Risk Repositorymit150

ENTITY

2 - AI

INTENT

3 - Other

TIMING

2 - Post-deployment

Risk ID

mit150

Domain lineage

5. Human-Computer Interaction

92 mapped risks

5.1 > Overreliance and unsafe use

Mitigation strategy

1. **Implement Algorithmic Recourse and Transparency Mechanisms (Highest Priority):** Establish robust, accessible, and high-agency user interfaces that allow individuals to easily understand the key factors driving sensitive algorithmic inferences and to immediately challenge, correct, or opt-out of personalized content streams (e.g., ads, recommendations) that are based on potentially harmful or incorrect health/emotional status assumptions. 2. **Mandate Sensitive Data Governance and Proactive Bias Auditing:** Develop a stringent data governance strategy that explicitly identifies, inventories, and protects features—including latent proxies—capable of inferring protected or vulnerable attributes (such as recent health events or psychological states). This must be accompanied by mandatory, periodic audits to assess for and mitigate disparate impact and unfairness metrics across sensitive demographic or health-related subgroups to prevent algorithmic exploitation. 3. **Integrate Human-in-the-Loop Systems for High-Consequence Decisions:** Implement a Human-in-the-Loop (HIL) protocol to ensure human oversight and contextual judgment for any algorithmic inference or automated action that carries a high risk of emotional distress, diminished well-being, or physical safety failure. This system should triage sensitive predictions (e.g., health status changes) to a human reviewer before triggering downstream exploitative or potentially harmful outputs.

ADDITIONAL EVIDENCE

I was getting ads for maternity clothes. I was like, 'Oh please stop.' . . . there's no way to tell your app, 'I had a miscarriage. Please stop sending me these updates'