In ten days this September, California signed #SB813 and #AB1405, Anthropic and OpenAI pledged embedded evaluators, AIUC-1 raised $40 million to audit frontier models, and Executive Order N-9-26 told the state of California to move faster. As a community, we have at least eight competing responses to who can verify a claim about an AI system, mostly without clarification of what either "verify" or "AI system" actually mean. Deployers and Providers who are relying on us to bring clarity are often left with more questions than answers. As we compete to prop up our own initiatives, our eagerness and confidence show real potential to do more harm than good. Today's article describes how our "answers" compare, and why the accredited infrastructure we already have is, in my opinion, the strongest foundation to build on. I fully welcome your feedback...this is a topic with very few direct paths forward. A-LIGN Petar Besalev Michael Brooks CISSP, PMP, MBA Matthew B. Davis Shea Brown Danny Manimbo Graeme Finley Partnership for Assurance, Credibility, and Trust on AI ANAB - ANSI National Accreditation Board UKAS AIQI Consortium Slater Sharp #TheBusinessofCompliance #ComplianceAlignedtoYou
Patrick Sullivan my bet is on AEF-1 and accreditation winning the race. I agree AEF-1 is mostly a specialisation of existing requirements, but not entirely. I like it. Accreditation + MLA + GLOBAC is really the only answer when you are dealing with a globally traded product. The future of AI is in embedded devices, which will increase the pull towards the same route.
Ramy Nassar
1mo
Accounting had a version of this fight a century ago, and what settled it was not the best framework but a shared answer to what the auditor is liable for when the opinion turns out wrong. Until AI assurance answers that question, eight competing approaches is just the market working out who carries the risk.
Cool
Patrick Sullivan I am the deployer in your third paragraph, so here is what I would need the word verify to mean before any of the eight answers helps me. I run a pipeline of about 90 AI agents for engineering work. The failures that cost me the most were in the harness around the model, which is the permissions list that says what an agent may touch, and the review that decides when its output is safe to ship. A verified claim about the model would not have caught the agent that froze the tests so a build could go green. If an accredited auditor came to me tomorrow I would hand them two things, the permissions file and the hours of senior review each run costs before it ships, roughly one and a half times a person's PR in my case. Those are the controls I actually operate, and they look like every change-control audit your accredited infrastructure already knows how to run. The frontier model is the labs' problem to verify. What I bolted on top of it is mine, and as far as I can tell none of the eight bets is aimed there.