Back to the MIT repository
7. AI System Safety, Failures, & Limitations2 - Post-deployment

Reproducibility

How a learning model can be reproduced when it is obtained based on various sets of data and a large space of parameters. This problem becomes more challenging in data-driven learning procedures without transparent instructions

Source: MIT AI Risk Repositorymit604

ENTITY

2 - AI

INTENT

2 - Unintentional

TIMING

2 - Post-deployment

Risk ID

mit604

Domain lineage

7. AI System Safety, Failures, & Limitations

375 mapped risks

7.4 > Lack of transparency or interpretability

Mitigation strategy

1. Establish a comprehensive and rigorously enforced version control system encompassing the entire machine learning pipeline. This must include versioning of the source code (e.g., Git), all raw and preprocessed datasets (e.g., DVC), and a dedicated Model Registry to log every artifact, hyperparameter, configuration setting, and evaluation metric of the resulting model, ensuring full auditability and traceability to enable reliable model rollback. 2. Enforce environment standardization through containerization technologies (e.g., Docker) to encapsulate the model's runtime environment, including all operating system dependencies, software libraries, and specified versions. This practice minimizes the discrepancies that arise from heterogeneous computing environments, ensuring consistent execution and identical results across development, testing, and post-deployment stages. 3. Mandate the implementation of deterministic computational methods, primarily by explicitly setting all relevant random seeds (for operating system, hardware, and libraries like NumPy, PyTorch, or TensorFlow) at the initiation of the training process. This eliminates stochasticity as a variable in the model generation process, which is foundational for achieving consistent output when running the experiment with the same inputs and parameters.