Social manipulation in elderly- and child-care
the use of advanced AI for elderly- and child-care are subject to risk of psychological manipulation and misjudgment
ENTITY
2 - AI
INTENT
1 - Intentional
TIMING
2 - Post-deployment
Risk ID
mit625
Domain lineage
5. Human-Computer Interaction
5.1 > Overreliance and unsafe use
Mitigation strategy
1. Mandate the implementation of robust Human-in-the-Loop (HITL) mechanisms, particularly for high-stakes decisions and interventions within the care environment, to ensure human judgment and accountability override autonomous AI actions that could lead to misjudgment or harm. 2. Establish and strictly enforce a sector-specific AI ethics and governance framework that explicitly prohibits the development and deployment of AI systems capable of exploiting cognitive or emotional vulnerabilities in elderly and child populations, requiring high levels of transparency and explainability in all human-AI interactions. 3. Invest in and deploy sophisticated AI-enhanced detection and authentication technologies (e.g., deepfake and voice-cloning verification tools) and provide continuous, specialized training to caregivers and users on recognizing and resisting emotionally manipulative communication tactics facilitated by advanced AI.