Reifying essentialist categories
algorithmic systems that reify essentialist social categories can be understood as when systems that classify a person’s membership in a social group based on narrow, socially constructed criteria that reinforce perceptions of human difference as inherent, static and seemingly natural... especially likely when ML models or human raters classify a person’s attributes – for instance, their gender, race, or sexual orientation – by making assumptions based on their physical appearance
ENTITY
2 - AI
INTENT
2 - Unintentional
TIMING
2 - Post-deployment
Risk ID
mit139
Domain lineage
1. Discrimination & Toxicity
1.1 > Unfair discrimination and misrepresentation
Mitigation strategy
1. Mandate a Conceptual Shift in Model Design and Output Interpretation Prohibit the conflation of complex human identity categories, such as gender or race, with static, observable proxies (e.g., morphology, sex characteristics). System outputs must be re-framed and communicated to users as inferences about *observable features* rather than affirmations of a person's inherent *identity*, thereby de-essentializing the classification. 2. Implement Targeted Pre-processing and Fair Representation Learning Employ advanced pre-processing techniques, such as reweighting or transforming training data, to correct for representation bias. Simultaneously, utilize fair representation learning strategies to ensure the learned features in the model are statistically invariant to or decoupled from the sensitive attributes that lead to essentialist categorization. 3. Establish User-Centric and Algorithmic Feedback Mechanisms Integrate mandatory post-deployment feedback loops, including user-correction mechanisms for miscategorization and regular algorithmic impact assessments/audits, to continuously monitor for, measure, and mitigate representational harms that emerge from the system's reification of social categories in real-world contexts.
ADDITIONAL EVIDENCE
[Automatic gender recognition] aim(s) to capture the morphological sexual differences between male and female faces by comparing their shape differences to a defined face template. We assume that such differences change with the face gender