Promoting harmful stereotypes by implying gender or ethnic identity
CAs can perpetuate harmful stereotypes by using particular identity markers in language (e.g. referring to “self” as “female”), or by more general design features (e.g. by giving the product a gendered name such as Alexa). The risk of representational harm in these cases is that the role of “assistant” is presented as inherently linked to the female gender [19, 36]. Gender or ethnicity identity markers may be implied by CA vocabulary, knowledge or vernacular [124]; product description, e.g. in one case where users could choose as virtual assistant Jake - White, Darnell - Black, Antonio - Hispanic [117]; or the CA’s explicit self-description during dialogue with the user.
ENTITY
2 - AI
INTENT
2 - Unintentional
TIMING
2 - Post-deployment
Risk ID
mit222
Domain lineage
1. Discrimination & Toxicity
1.1 > Unfair discrimination and misrepresentation
Mitigation strategy
1. Implement industry-wide standards to systematically eliminate gender and ethnic defaults in Conversational Agent (CA) design, specifically by avoiding female-default voices, gendered names (e.g. Alexa, Siri), and product descriptions that reinforce stereotypes linking the assistant role to a specific gender or ethnicity. 2. Program Conversational Agents to respond assertively and condemn harassing, sexually explicit, or abusive language, preventing the reinforcement of harmful stereotypes that portray a specific gender as subservient, passive, or accepting of objectification. 3. Mandate diverse and cross-functional development teams, including expertise in ethics and social science, to implement ethical frameworks and conduct inclusive user research/testing across the AI lifecycle, thereby mitigating implicit biases in design and data that lead to harmful stereotyping.
ADDITIONAL EVIDENCE
The commonplace gendering of CAs as female has been argued to promote the objectification of women, reinforcing ‘the idea that women are tools, fetishized instruments to be used in the service of accomplishing users’ goals’ [36, 195, 202]. For exam- ple, a study of five commercially available voice assistants in South Korea found that all assistants were voiced as female, self-described as ‘beautiful’, suggested ‘intimacy and subordination’, and ‘embrace sexual objectification’ [85]. Non-linguistic AI systems were found to typically present as ‘intelligent, professional, or powerful’ and as ethnically White, reinforcing historical racist associations between intelligence and whiteness [35].