Out-of-domain data
Without proper validation and management on the input data, it is highly probable that the trained AI/ML model will make erroneous predictions with high confidence for many instances of model inputs. The unconstrained inputs together with the lack of definition of the problem domain might cause unintended outcomes and consequences, especially in risk-sensitive contexts....For example, with respect to the example shown in Fig. 5, if an image with the English letter A is fed to an AI/ML model that is trained to classify digits (e.g., 0, 1, …, 9), no matter how accurate the AI/ML model is, it will fail as the input data is beyond the domain that the AI/ML model is trained with. U
ENTITY
2 - AI
INTENT
2 - Unintentional
TIMING
3 - Other
Risk ID
mit335
Domain lineage
7. AI System Safety, Failures, & Limitations
7.3 > Lack of capability or robustness
Mitigation strategy
1. Deploy and Continuously Monitor Out-of-Distribution (OOD) Detection Mechanisms Implement robust OOD detection techniques—such as Mahalanobis distance in feature space, energy-based scoring, or ensemble uncertainty estimation—to quantify and flag inputs that deviate significantly from the model's training data distribution. This enables the system to either abstain from generating a prediction or to explicitly reject unfamiliar inputs, preventing erroneous, high-confidence outputs. 2. Enforce Strict Data Validation and Preprocessing Pipelines Integrate mandatory, automated data validation and schema checks within the input pipeline (e.g., during Continuous Integration/Continuous Deployment). This ensures that all data adheres to predefined domain constraints and formats before reaching the model for inference, thereby eliminating unconstrained inputs that fall outside the problem domain. Version control for the preprocessing code and configurations is critical for reproducibility. 3. Establish a Structured Abstinence and Human-in-the-Loop Escalation Policy Define an explicit action policy for inputs identified as OOD, utilizing "learning with rejection" to train the model to output an "I don't know" response when uncertainty is high. This process must include a structured escalation path for channeling OOD flags to a human expert for review and contextual disposition, allowing for continuous model improvement and safe operational control.