Training & validation data
This is the risk posed by the choice of data used for training and validation.
ENTITY
1 - Human
INTENT
3 - Other
TIMING
1 - Pre-deployment
Risk ID
mit191
Domain lineage
7. AI System Safety, Failures, & Limitations
7.0 > AI system safety, failures, & limitations
Mitigation strategy
1. Establish a robust data governance framework, including the mandatory creation of Datasheets for Datasets, to rigorously document the motivation, composition, collection provenance, and known limitations of all training and validation data, thereby proactively identifying representational and measurement bias before model ingestion. 2. Systematically implement bias detection and mitigation strategies by conducting pre-processing techniques (e.g., balancing through re-sampling or synthetic data generation) and performing fairness audits that disaggregate performance metrics across all relevant demographic and sub-groups during the validation phase to ensure equitable model outcomes. 3. Institute continuous monitoring and auditing mechanisms post-deployment to track model performance, detect data drift, and identify the emergence of new societal or contextual biases in the live environment, coupled with a defined human-in-the-loop intervention strategy for sensitive or uncertain decisions.
ADDITIONAL EVIDENCE
Due to their data-driven nature, the behavior of machine learning systems is often heavily influenced by the data used to train them. An ML system trained on data encoding historical or social biases will often exhibit similar biases in its predictions. Separate from the training data, validation datasets are often used to evaluate an ML model’s ability to generalize beyond the training data, to new examples from the same distribution, or to examples with different characteristics (other distributions). Representative validation data can be used to detect potential mismatches between the training data and the deployment environment, such as the presence of social biases or spurious features in the training data. We summarize key data risks specific to ML systems and refer the reader to Demchenko et al. for a detailed discussion of the general issues around big data [50]