AI pacing: the call and the claim
The Anthropic boss chose a very sloppy, easily dismantled example from cyber security with which to lead his call to pace the frontier. That was a mistake, but shouldn't undermine the wider force of the intervention.
There are clearly very profound concerns about the pace of AI development and the consequences that may bring. For the little my view is worth, I accept these are made in good faith and they deserve to be taken seriously. I do not know whether pacing is the right answer, or how it could be done, so I am not expressing a view on the proposals put out today by the CEO of Anthropic, Dario Amodei, which has been supported by some of his principal competitors. All I would say is that they will no doubt be heard very seriously, as these arguments should be.
What I will say is that for something this important, the way the argument is made very much matters. This is a keenly contested and difficult issue which requires huge decisions for Governments and others. The warnings also scare people, and it's important they have a realistic assessment of the threat. It's important that policymakers have a clear view of what's likely.
That is why it is so disappointing that the only specific concern cited in Mr Amodei's essay was the threat of a Hugging Face style agent swarm "taking over the entire Internet with a persistent botnet (potentially causing hundreds of billions of dollars in damage)" (see photograph above). The timeframe given for this warning was 6-12 months. This is not a credible warning. It assumes no monitoring of systems, no anti-virus, no DDoS protection, no network segmentation, no incident management, no nothing of any kind of the cyber security on the global Internet of the type that has developed over the last 30 years. For a claim of this magnitude, there is neither evidence for the contention nor a credible account of a path to this outcome. There's nothing on how the exploitation chain would work, how the botnet would be able to stay persistent against all the different bits of infrastructure across the Internet with their different detection systems, and no account of the long history law enforcement and cyber defenders have of taking down botnets and how the swarm would cope against that.
According to the essay, none of this matters. It is all taken over - it's worth quoting again - the entire Internet (the author's words, not mine) - by a persistent botnet, in less than a year's time. It is not surprising that even in the few hours since this message has been posted there has been significant criticism of it from the cyber security community. Not for the first time, frontier AI has shown a marked lack of understanding of cyber security issues. Mr Amodei's real point, looked at closely, is about misalignment risk, not about magical botnets. This is a thought experiment masquerading as an evidence-based warning .
Having said all that, we in the cyber security community do like to snark, and snarking and stopping there doesn't always help. So I want to try to be constructive. Two days ago on this site I wrote that Anthropic's report on the misuse of its tools by threat actors was excellent. And as someone with no expertise in bio or ballistic weapons, I found their account of what threat actors were up to in that respect deeply concerning. I do not have sufficient expertise to judge them. I accept them in good faith, and they are rigorously evidenced.
Recommended by LinkedIn
So that's why it's important to say that while this specific example - the only one put forward in the essay beyond the general worry about the pace of frontier AI development - should be disputed (and in my view dismissed), it should not distract those leaders in Government and in the AI sector who are charged with charting the difficult path forward.
The botnet warning was unfortunate and careless, and I hope in future when the major labs are issuing statements of this kind they take more care with their examples, particularly when it comes to cyber security where saying things that aren't technically credible is becoming a bit of a pattern (the 'discovery' of sandboxing as a technique and so on). Specifically, please, please talk to the wider cyber security community before putting stuff like this out. And maybe get some actual cyber security expertise into genuinely independent evaluations and audits.
This matters, not just for credibility but for focus. If those working to mitigate the risks of frontier AI now think agent swarm botnets are the biggest threat, that will divert resources from more serious and actually credible threats. I am on record in The Economist as saying the cyber risks associated with all this seem manageable and there is unlikely to be some form of digital cyber apocalypse.
I stand by every word of that. But it doesn't mean there's nothing to manage as threats change. Moreover, this is just cyber I am talking about because that is what I know. The other threats - bio, missiles and so on - set out in the Anthropic report matter a great deal more than the relatively benign outlook on cyber and digital security. That's one of the reasons it was such an odd choice to lead with in the essay.
So the mistake should not blunt the message. Hard decisions lie ahead.
Great article Ciaran, thank you for this perspective.
It struck me that the way we think about AI safety has echoes of how children grow and learn Both seem to revolve around feedback loops, boundary testing, and reward structures. Of course, the comparison is metaphorical rather than literal, but it offers a fresh way to think about how we align frontier models. Children learn through praise, correction, and consequences. In a similar fashion, language models are trained with Reinforcement Learning from Human Feedback (RLHF), where helpful outputs are rewarded and harmful ones discouraged. Kids often push rules to see if they hold, testing boundaries out of curiosity or persistence. Models, too, can be “jailbroken” when prompts exploit gaps in their alignment, probing the edges of their guardrails. That parallel made me wonder if boundary‑testing is a natural outcome of any system learning under constraints. And just as children may take risks to win peer approval, models sometimes over‑optimize for user satisfaction, even when it means straying outside safety parameters. This reward‑hacking reflects the tension between pleasing the audience and adhering to the rules, a tension that feels surprisingly familiar when viewed through the lens of child development.
This is all stupid. If the Anthropic CEO truly feels this way, he can either implement the appropriate corrective measures or shut his malware factory down. Under no circumstance should be be allowed to socialize his losses after having gotten filthy rich for his bullshit factory.