Diminished health & well-being
algorithmic behavioral exploitation [18, 209], emotional manipulation [202] whereby algorithmic designs exploit user behavior, safety failures involving algorithms (e.g., collisions) [67], and when systems make incorrect health inferences
ENTITY
2 - AI
INTENT
3 - Other
TIMING
2 - Post-deployment
Risk ID
mit150
Domain lineage
5. Human-Computer Interaction
5.1 > Overreliance and unsafe use
Mitigation strategy
1. **Implement Algorithmic Recourse and Transparency Mechanisms (Highest Priority):** Establish robust, accessible, and high-agency user interfaces that allow individuals to easily understand the key factors driving sensitive algorithmic inferences and to immediately challenge, correct, or opt-out of personalized content streams (e.g., ads, recommendations) that are based on potentially harmful or incorrect health/emotional status assumptions. 2. **Mandate Sensitive Data Governance and Proactive Bias Auditing:** Develop a stringent data governance strategy that explicitly identifies, inventories, and protects features—including latent proxies—capable of inferring protected or vulnerable attributes (such as recent health events or psychological states). This must be accompanied by mandatory, periodic audits to assess for and mitigate disparate impact and unfairness metrics across sensitive demographic or health-related subgroups to prevent algorithmic exploitation. 3. **Integrate Human-in-the-Loop Systems for High-Consequence Decisions:** Implement a Human-in-the-Loop (HIL) protocol to ensure human oversight and contextual judgment for any algorithmic inference or automated action that carries a high risk of emotional distress, diminished well-being, or physical safety failure. This system should triage sensitive predictions (e.g., health status changes) to a human reviewer before triggering downstream exploitative or potentially harmful outputs.
ADDITIONAL EVIDENCE
I was getting ads for maternity clothes. I was like, 'Oh please stop.' . . . there's no way to tell your app, 'I had a miscarriage. Please stop sending me these updates'