Inappropriate data splitting

Mitigation strategy

1. Enforce Strict Procedural and Technical Separation of Data Subsets Institute rigorous governance and access controls to ensure the designated holdout test set is used *exclusively* for final, unbiased model evaluation. This control is paramount to preventing test set contamination, which fundamentally compromises the system's quality assurance metrics and leads to an overestimation of real-world performance. The test set must be isolated from the training, validation, and hyperparameter tuning processes. 2. Employ Structurally Appropriate Data Splitting Methodologies Select a data partitioning strategy—such as time-based splits for temporal data or grouped splits for inherent entity structures (e.g., users, sessions)—that aligns precisely with the data's dependencies and the inference-time constraints. All data transformation and feature engineering steps must be fit *only* on the training set and subsequently applied to the test set to avoid unintentionally bleeding information across partitions. 3. Institute Automated Split Validation and Auditing Mechanisms Implement automated validation tests and audit tools within the MLOps pipeline to continuously inspect for signs of data leakage. Key checks include validating that no overlapping records or identifiers exist between sets, monitoring data lineage, and conducting differential testing (such as ablation studies) to detect anomalous performance gains that may indicate hidden leakage patterns.