Discrimination and Stereotype Reproduction
General purpose AI models interpret and respond to inputs based on their training data, potentially causing Discrimination and Stereotype Reproduction. Since they are “black-box” models, the exact mechanism behind decisions remains opaque and attempts to mitigate harmful outputs are not fully reliable yet. These models have the capacity to influence a multitude of downstream applications, decisions, and processes, thereby affecting many individuals simultaneously. The extent of this impact could outstrip the range of any single human or group of humans, amplifying the potential consequences of embedded biases or stereotypes.
ENTITY
2 - AI
INTENT
2 - Unintentional
TIMING
2 - Post-deployment
Risk ID
mit837
Domain lineage
1. Discrimination & Toxicity
1.1 > Unfair discrimination and misrepresentation
Mitigation strategy
1. Implement a Comprehensive Governance and 'Fairness-by-Design' Framework: Establish mandatory governance structures that define explicit fairness objectives (e.g., equal opportunity, demographic parity) and accountability throughout the AI lifecycle. This includes ensuring diverse, interdisciplinary teams are involved from the initial problem-framing stage to proactively mitigate systemic biases. 2. Mandate Explainable AI (XAI) and Continuous Bias Auditing: Employ interpretability techniques such as SHAP or LIME to transform the opaque "black-box" model into an auditable system, enabling the justification of automated decisions. This must be coupled with continuous, rigorous testing using a suite of algorithmic fairness metrics (e.g., statistical parity) and adversarial testing methods (e.g., Metamorphic Relations) to detect and quantify disparate outcomes across demographic groups. 3. Optimize Data Quality and Representativeness: Institute stringent, regular audits of training datasets to identify and remove sources of historical and stereotyping bias. This requires implementing data augmentation strategies, such as the collection of representative data for under-represented groups or the judicious use of synthetic data, to ensure dataset balance and reduce representational bias.