AI discrimination
AI discrimination is a challenge raised by many researchers and governments and refers to the prevention of bias and injustice caused by the actions of AI systems (Bostrom & Yudkowsky, 2014; Weyerer & Langer, 2019). If the dataset used to train an algorithm does not reflect the real world accurately, the AI could learn false associations or prejudices and will carry those into its future data processing. If an AI algorithm is used to compute information relevant to human decisions, such as hiring or applying for a loan or mortgage, biased data can lead to discrimination against parts of the society (Weyerer & Langer, 2019).
ENTITY
2 - AI
INTENT
2 - Unintentional
TIMING
3 - Other
Risk ID
mit329
Domain lineage
1. Discrimination & Toxicity
1.1 > Unfair discrimination and misrepresentation
Mitigation strategy
1. Prioritize Data Auditing and Enhancement for Representativeness Rigorously audit all training and test datasets to proactively identify and mitigate fundamental biases (historical, representation, and measurement biases). Implement pre-processing techniques, such as stratified sampling, feature engineering, or reweighting, to ensure the data is diverse, balanced, and accurately reflective of the real-world population. This is the most critical step as it prevents the replication and amplification of past societal inequities by the AI system. 2. Employ Algorithmic Fairness Metrics and Continuous Monitoring Utilize mathematically defined fairness metrics (e.g., Demographic Parity, Equalized Odds, Equal Opportunity) and adversarial testing (red-teaming) to quantify and detect disparate impact and differential performance across protected groups. These detection methods must be integrated into a continuous evaluation and monitoring pipeline to identify and correct bias that may emerge or 'drift' post-deployment, ensuring the system’s fairness is maintained throughout its operational lifecycle. 3. Integrate Robust AI Governance and Human-in-the-Loop Oversight Establish a comprehensive AI governance framework and ethical guidelines that mandate transparency, explainability, and fairness standards from the design phase. Institute mandatory human oversight for all high-stakes, consequential decisions (e.g., in hiring, lending, or law enforcement) to review and validate AI outputs. This human-in-the-loop validation is essential for providing contextual judgment and overriding potential algorithmic biases that automated controls may fail to resolve.