Auditor capacity mismatch
Auditors may not be able to address all of the specific safety, performance, or validation needs. Reports of passing audits may be more inclusive than can be justified due to a lack of knowledge of specific risks and how they can be tested, or a lack of capacity to perform sufficiently rigorous testing.
ENTITY
1 - Human
INTENT
2 - Unintentional
TIMING
1 - Pre-deployment
Risk ID
mit1128
Domain lineage
6. Socioeconomic and Environmental
6.5 > Governance failure
Mitigation strategy
1. Prioritize the enhancement of technical and domain-specific expertise: Implement a mandatory, comprehensive training and certification regime for auditors, focusing on the unique technical, ethical, and domain-specific risk landscape pertinent to the audited General-Purpose AI (GPAI) systems. This directly addresses the deficit in knowledge and necessary capacity. 2. Mandate the adoption of rigorous, well-scoped audit standards: Develop and enforce clear, granular auditing standards that explicitly define the scope, necessary testing methodologies (qualitative and quantitative), and minimum requirements for auditor access to relevant data, code, and system architecture. This ensures sufficiently rigorous testing and justifiable audit conclusions. 3. Establish a continuous quality assurance and capacity monitoring framework: Institute an independent oversight body, such as an Audit Quality Governance Committee, tasked with continuously monitoring auditor performance, evaluating the quality of completed work, and proactively assessing the sufficiency of the audit resource pool relative to the evolving systemic risks.