Back to the MIT repository
7. AI System Safety, Failures, & Limitations1 - Pre-deployment

Design

This is the risk of system failure due to system design choices or errors.

Source: MIT AI Risk Repositorymit193

ENTITY

1 - Human

INTENT

3 - Other

TIMING

1 - Pre-deployment

Risk ID

mit193

Domain lineage

7. AI System Safety, Failures, & Limitations

375 mapped risks

7.3 > Lack of capability or robustness

Mitigation strategy

1. Formalize a governance process to rigorously vet the ML task formulation and the design specifications of all system components (e.g., tokenizers, data pipelines) during the pre-deployment phase to ensure fundamental alignment with robustness and safety requirements. 2. Implement comprehensive, systematic end-to-end validation methodologies, including stress-testing against diverse datasets and simulated failure modes, to empirically verify the functional robustness and reliability of the overall system design before deployment. 3. Establish continuous operational monitoring systems to track critical performance metrics and proactively detect model decay or drift, which serves as an essential mechanism for identifying and correcting failures arising from initial design assumptions no longer holding in the deployment environment.

ADDITIONAL EVIDENCE

While the ML model is the core component, we should not neglect the risks resulting from how the problem is modeled as an ML task and the design choices concerning other system components, such as the tokenizer in natural language processing (NLP) systems