academic analysis

Abstract

Safety frameworks have emerged as a best practice for managing risks from frontier artificial intelligence (AI) systems. However, it may be difficult for stakeholders to know if companies are adhering to their frameworks. This paper explores a potential solution: third-party compliance reviews. During a third-party compliance review, an independent external party assesses whether a frontier AI company is complying with its safety framework. First, we discuss the main benefits and challenges of such reviews. On the one hand, they can increase compliance with safety frameworks and provide assurance to internal and external stakeholders. On the other hand, they can create information security risks, impose additional cost burdens, and cause reputational damage, but these challenges can be partially mitigated by drawing on best practices from other industries. Next, we answer practical questions about third-party compliance reviews, namely: (1) Who could conduct the review? (2) What information sources could the reviewer consider? (3) How could compliance with the safety framework be assessed? (4) What information about the review could be disclosed externally? (5) How could the findings guide development and deployment actions? (6) When could the reviews be conducted? For each question, we evaluate a set of plausible options. Finally, we suggest "minimalist", "more ambitious", and "comprehensive" approaches for each question that a frontier AI company could adopt.