Design
This is the risk of system failure due to system design choices or errors.
ENTITY
1 - Human
INTENT
3 - Other
TIMING
1 - Pre-deployment
Risk ID
mit193
Domain lineage
7. AI System Safety, Failures, & Limitations
7.3 > Lack of capability or robustness
Mitigation strategy
1. Formalize a governance process to rigorously vet the ML task formulation and the design specifications of all system components (e.g., tokenizers, data pipelines) during the pre-deployment phase to ensure fundamental alignment with robustness and safety requirements. 2. Implement comprehensive, systematic end-to-end validation methodologies, including stress-testing against diverse datasets and simulated failure modes, to empirically verify the functional robustness and reliability of the overall system design before deployment. 3. Establish continuous operational monitoring systems to track critical performance metrics and proactively detect model decay or drift, which serves as an essential mechanism for identifying and correcting failures arising from initial design assumptions no longer holding in the deployment environment.
ADDITIONAL EVIDENCE
While the ML model is the core component, we should not neglect the risks resulting from how the problem is modeled as an ML task and the design choices concerning other system components, such as the tokenizer in natural language processing (NLP) systems