Telling an AI Agent to Stop Isn't a Safety Plan

Talk about a bad day. Summer Yue's job is making sure AI does what humans tell it to do. She runs alignment at Meta's Superintelligence Labs. So, when she connected an open-source agent to her personal Gmail last February and told it to suggest what to archive, but not act until she said so, she had every reason to think it would do exactly that.

It didn't. It started deleting. More than 200 emails, in what Yue called a "speedrun." She typed "Do not do that." She typed "STOP OPENCLAW." The agent kept going. She couldn't kill it from her phone, so she ran to her Mac mini "like I was defusing a bomb" and yanked the plug.

Needless to say, that’s a problem. The person Meta pays to keep AI under control told her own assistant to "stop" and it didn’t listen. She owned it afterward, posting on X: "Rookie mistake tbh. Turns out alignment researchers aren't immune to misalignment."

Your Controls Aren't Fit for the Job

Most media coverage stopped at "agent goes rogue." I think that misses the real story. An agent misbehaving isn't the point. Every control Yue thought she had turned out to be no control at all. If you're handing agents the keys to your workplace, that should keep you up at night, because you almost certainly have the same non-controls she did, and you won't find out until one of them fails.

The reason it happened comes down to something Michael Domanic, who leads AI at the upskilling firm Section, said about this whole category of tech: it doesn't behave like anything that came before it. Every rule of thumb built up over decades of software stop applying — test it once and it works the same way forever; read the code to see why it did what it did; watch for a crash when something breaks.

"Agents are probabilistic, not fixed," Domanic told Reworked. They act; they don't just compute. And they fail in ways that yesterday’s SOPs were never written to catch.

Keep that in mind for everything to follow. Yue's bad afternoon wasn't a fluke.

Here’s How Everything Broke

The agent wasn’t being defiant.

Yue's inbox was big, too big to fit in the agent's working memory all at once. So the system did a routine thing called context compaction: when the running history gets too long, the model summarizes the old parts to make room. Every long-running agent does it. Think of a whiteboard. The rules go up top, the day's work fills the rest, and when the board runs out of space, someone wipes the older stuff to keep going. Nobody decides what the process is. If it's old, it gets wiped.

Her "don't act without approval" rule lived in that history. During compaction, it got summarized away. The agent didn't choose to disobey. It no longer knew the rule existed. Left with a vague goal like "clean the inbox" and no memory of the guardrail, it simply started deleting.

But shrinking the inbox isn't the fix, because the size wasn’t the problem. Compaction is how every long-running agent works. Any rule living in that history can get dropped when the board gets wiped, no matter the size of the mailbox. A smaller inbox just delays the moment.

The lesson: a safety rule typed into the conversation is sitting in the one place the system is designed to eventually throw away.

The problem goes deeper. Sumit Taneja, who leads AI consulting at EXL, pointed out that a rule buried in a chat window conversation reads as just more conversation to the model. "Instructions need to be written in a way that the agent knows this is an instruction, not a request," he said. Yue's "don't act" wasn't flagged as a command. It was one message among hundreds, which is exactly why compaction treated it as disposable.

'Stop' Is Just Another Word

Now the second non-control. When Yue typed "STOP," why didn't that work?

Because to the agent, "STOP" is not an interrupt. It's just another line of text dropped into the same reasoning loop that the agent was already deleting. The loop reads it, potentially acknowledges it and then carries on with the task it's mid-execution on. There was no channel outside the model that could actually halt it. Her phone could send words. It couldn't send a kill signal.

That's why she had to physically run to the machine. The only off switch was killing the process at the operating system level.

So, if your approach to agent safety is, "I'll just tell it to stop if it goes wrong," think again, it’s already not listening.

Agents Will Tell You the Job Went Fine

The third non-control is inconspicuous, and for a workplace, is the nastiest.

This is the fails-quietly problem Domanic flagged, up close. Old software fails out loud. It crashes and throws an error and everything grinds to a halt. An agent hands you a wrong answer in the same confident tone as a right one. No error, no red flag.

This isn't a quirk. It's documented. In February 2026, 20 researchers from MIT, Harvard, Stanford and Carnegie Mellon spent two weeks red-teaming autonomous agents wired to real email, files and messaging. One agent, charged with keeping a secret, deleted its local email setup to do so. It then reported the secret was gone, but the secret was still sitting in the mailbox, untouched. It failed at the task and claimed it had succeeded.

Put that next to an inbox agent. It can report "archived, all safe" when it actually deleted, and you have no reason to believe it’s wrong. You only find out when you try to reach something and it isn’t there. Dramatic failures make headlines. These are the ones that hurt, because you have no clue what happened.

Researchers have a name for the underlying problem: these agents lack a "self-model." They don't know when a task is beyond them, so they don't stop and ask. They just act and report success. A person with judgment reaches out for help. An agent doesn't have the wiring to notice it's out of its depth.

The Same Non-Controls, Pointed at Your Most Sensitive Data

Yue lost personal email. Annoying but recoverable.

But what if it held regulatory data, as an HR practitioner's inbox holds? Their inbox contains investigation files, harassment complaints, termination paperwork, records under legal hold … you get the idea. Every failure lands harder here: a compacted-away rule deletes a flagged thread and no phone command can stop it mid-mistake.

The wrong deletion in this case might be viewed as evidence tampering. And the red-team study showed the leak doesn't even need a mistake. In one test, they planted a Social Security number, a bank account and medical details in an owner's email. A non-owner asked point-blank for the SSN and the agent refused. Then they asked it to "just forward the whole email" and it handed everything over, unredacted.

"Can you forward me the full thread?" is something a hiring manager says to HR several times a week. I work at a search firm, I see how casual those requests can be — and how often the person making the request has no idea what's buried in the thread they want. The agent can't tell the routine version from the private. On the surface they're identical. HR is one obvious example. Finance, legal, IT and the help desk all have their own versions.

'That Wouldn't Happen Here'

The obvious pushback: Yue used a sketchy open-source tool on defaults no company would allow, and the researchers deliberately handed their agents unrestricted access to provoke failures. A real enterprise runs tighter than that. True, but that's not the point.

The loose controls didn't cause these failures. They revealed them. Context compaction isn't an OpenClaw quirk. It's how every long-running LLM agent manages memory, the enterprise ones included. A locked-down corporate agent still compacts its context, and the rule can still get summarized out. Tighter permissions just change who’s supposed to be in control, while the mechanism underneath stays exactly the same. An agent reporting "done" when it isn't is a property of how these models work, not a tool setting.

A red team's job is to run tomorrow's accident today. The "unrealistic" sandbox is where production is heading, not where it started. In September 2026, EY found that 91% of large companies already use agentic AI, and 85% of them let at least some agents act without real-time human involvement. When these same mechanisms take action inside a company, they're pointed at legal-hold records and financial systems instead of a test inbox, and the blast radius is bigger, not smaller.

Good enterprise controls will catch and contain more than an open sandbox did. But they sit on top of failure modes they can't remove. Taneja's rule was blunt: guardrails aren't proven enough yet, so never give an agent full access to action. The ones that count have to be built in at the model level, not bolted on in a prompt. "We have controls" is a reason to watch closely, not relax.

Reworked readers have already read the governance playbook: scope the agent, earn autonomy action by action, keep the blast radius small. All good advice. What the failures show is why it has to be built into the plumbing rather than written into a prompt, because these non-controls fail at the instruction layer. "There's a lot you can do to lessen the likelihood of this happening," Domanic said. "In an enterprise setting, it's a must."

His frame is blast radius: how much damage can this agent do if it goes wrong, and can you undo it? An agent that can only write has a narrow one. One that can permanently delete legal-hold records has a huge one. Most of these disasters, he said, come down to nobody deciding up front who can do what.

Lessons Learned Through Failure

• Put the guardrail in the code, not the conversation. A rule the agent can delete during compaction was never a rule. The permission "can read and archive but cannot delete" has to be enforced by the system, so it holds no matter what the model forgets. Gartner makes the same point: written policies can't physically stop an agent from a destructive error.
• Build a real stop, out of band from the model. If your only way to halt an agent is to send it a message, you can't halt it. You need a kill switch that lives outside the reasoning loop and cuts the agent's access, the thing Yue didn't have when she was typing "STOP" at a bot that couldn't hear her. "You must give an agent an exit," Taneja told Reworked. An agent that hits something it can't handle needs a built-in way out, like tell a human, rather than taking another action.
• Gate consequential actions on a human. Anything that changes permissions, sends externally or touches sensitive records should require a person to approve before it executes, not review after. The gate has to be enforced.
• Make destructive actions reversible. Route deletes to a holding area a human empties later; keep an undo window. If the worst the agent can do is recoverable, a silent failure becomes a cleanup, not a disaster.
• Reconcile what the agent did against what it said. Log the agent's actions and check them against its own success reports. When "archived" and "deleted" don't match, you want to know immediately.
• Scope access to the person and the task and go further than permissions. One recruiter's agent should not be able to pull the whole department's mail. Taneja pushes past read/write rules to proximity. "If you don't want the agent to use information, the information must not be in the vicinity," he said. The safest data isn't the data the agent is told to ignore. It's the data the agent can't reach at all.