Back to the MIT repository
5. Human-Computer Interaction2 - Post-deployment

Social manipulation in elderly- and child-care

the use of advanced AI for elderly- and child-care are subject to risk of psychological manipulation and misjudgment

Source: MIT AI Risk Repositorymit625

ENTITY

2 - AI

INTENT

1 - Intentional

TIMING

2 - Post-deployment

Risk ID

mit625

Domain lineage

5. Human-Computer Interaction

92 mapped risks

5.1 > Overreliance and unsafe use

Mitigation strategy

1. Mandate the implementation of robust Human-in-the-Loop (HITL) mechanisms, particularly for high-stakes decisions and interventions within the care environment, to ensure human judgment and accountability override autonomous AI actions that could lead to misjudgment or harm. 2. Establish and strictly enforce a sector-specific AI ethics and governance framework that explicitly prohibits the development and deployment of AI systems capable of exploiting cognitive or emotional vulnerabilities in elderly and child populations, requiring high levels of transparency and explainability in all human-AI interactions. 3. Invest in and deploy sophisticated AI-enhanced detection and authentication technologies (e.g., deepfake and voice-cloning verification tools) and provide continuous, specialized training to caregivers and users on recognizing and resisting emotionally manipulative communication tactics facilitated by advanced AI.