Is the AI Model the Risk, or Is It the Harness Around It?

The debate over artificial intelligence safety has moved well beyond hallucinations and bad chatbot answers.

Anthropic CEO Dario Amodei is arguing that frontier AI development needs to be paced so safety work has time to catch up. In his essay, “We Must Pace the Frontier,” Amodei warns about increasingly capable AI systems outrunning the industry's ability to understand and control them.

Warnings are also coming from inside Anthropic. As VKTR reported, Anthropic researcher Jacob Coxon resigned amid concerns about the industry's trajectory, while Anthropic alignment researcher Evan Hubinger put his personal estimate of AI wiping out humanity within the next decade at greater than 10%.

The customer experience impact, you ask?

For customer experience organizations already connecting AI agents to customers, data and enterprise systems, know this: the model itself is only part of the risk. What surrounds the model — and what organizations permit it to do — can determine the consequences when something goes wrong.

That was the hot take from NiCE Chief AI Officer Philipp Heltewig on LinkedIn Sept. 13.

"AI won't kill us (on its own). Its harness could. ... The models themselves only generate text and propose actions," Heltwewig wrote. "I feel we should spend more time asking what we’re allowing AI models to do and how. The harness — the tools, connectivity and permissions around a model — turns its suggestions into actions and it’s the harness we should look at more." 

Why the AI Model Isn't the Real Risk

In the LinkedIn post, Heltewig, former CEO of Cognigy, compared the model to a human brain: It reasons and makes plans. He described alignment as an attempt to shape its behavior, while describing the harness as the tools available to the model and the rules governing their use.

In Heltewig's framing, that distinction becomes especially important once an AI model is connected to systems that allow it to act rather than simply generate an answer.

Heltewig pointed to tools that might allow a CX agent to look up a customer, update a case or send a confirmation. But he argued in the post that simply defining a tool does not make its use inherently safe. Organizations still need to determine whose data the agent can access, what it can change and where it can send information.

He drew a parallel between how AI Agents and how humans behave:

• The model itself is like the brain: it reasons and makes plans.
• Alignment aims to shape its behavior, much as values shape ours.
• The harness is the tools available to the model and the rules for using them, just like we have tools to perform actions in the physical world. Our laws determine whether we can use a tool for a given purpose or not.

Heltewig said NiCE Cognigy's approach is to enforce permissions independently of the model and make tool use auditable at scale.

His argument brings a CX deployment perspective to the much broader frontier-AI debate: The intelligence of a model matters, but so does the infrastructure that determines what that intelligence is empowered to do.

What Does NiCE's Heltewig Say Separates Safe AI Agents From Risky Ones?

NiCE's Philipp Heltewig argues that the harness — the tools, connectivity and permissions surrounding a model — determines whether an agent's mistakes stay contained or cause real damage.

Related Article: NiCE Makes Its Move at NiCE World 2026: Agentic AI Is Now the Architecture

Why Customer Experience Needs an AI Harness

Heltewig's use of the “harness” concept predates his current LinkedIn response to the Anthropic safety debate.

During an exclusive CMSWire interview at NiCE Interactions in Orlando in June, Heltewig discussed whether enterprises could bypass established CX platforms and build customer-facing AI applications directly with models and coding tools such as Claude.

Then, Heltewig drew a sharp distinction between quickly creating an impressive AI demonstration and operating an enterprise application at production scale:

"You can give me JavaScript and Claude, and I'm going to build a cool voice demo without Cognigy, but does that demo hold up when I'm getting bombarded with 20,000 concurrent calls? Does it hold up under the security and compliance reviews? Does it have things like auditing, logging and alerting, and all this kind of stuff? Of course not.”

Heltewig said enterprises cannot assume that connecting a powerful general-purpose model to a contact center provides the infrastructure required for production CX.

"You can't plug ChatGPT into a contact center and it does everything for you. You need a harness around that of a certain robustness, which is exactly what we're delivering," he added.

Heltewig's harness argument was primarily about enterprise readiness rather than the existential AI safety discussion he addressed on LinkedIn in September. His June comments focused on scale, performance, security, compliance, auditing, logging and alerting. It all kind of ties in together though, no?

Beyond Security: The Platform Behind the Agent

Heltewig also told CMSWire in June that the infrastructure surrounding AI agents matters beyond security and compliance. While discussing point AI solutions, he argued that agents ultimately depend on the broader CX platform supporting them.

"AI agents are really only as good as a platform that they stand on," he said.

Heltewig connected that platform argument to practical CX issues such as transferring calls and chats, combining analytics data, learning from conversations and feeding those learnings back into AI agents.

Point solutions can demonstrate agentic capabilities and produce strong standalone applications, but organizations pursuing CX holistically can face integration challenges when those AI systems sit outside their core CX platform. 

Why Can't Enterprises Just Plug ChatGPT Into a Contact Center?

NiCE's Heltewig told CMSWire in June that a general-purpose model lacks the security, compliance, auditing, logging and alerting a production contact center demands — infrastructure only a dedicated platform can provide at scale.

How Anthropic Frames the Same Question at Scale

Amodei's concerns operate at the frontier-model level rather than the contact center. But there is an overlap between the issues raised in Amodei's essay and Heltewig's two sets of comments.

In “We Must Pace the Frontier,” Amodei argues that AI capabilities are advancing quickly enough to warrant measures that create additional time for safety work, including alignment research, interpretability, testing and operational safeguards. 

Frontier AI Concern	CX Version of the Question
Can increasingly capable AI remain under control?	Exactly what is a customer-facing agent authorized to do?
Can an AI system act beyond its intended task?	Can an agent access data, tools or workflows outside its assigned scope?
Can unexpected behavior be detected?	Can CX teams monitor agent actions and intervene when necessary?
Can failures be reconstructed?	Are agent decisions, tool calls and data access logged and auditable?
How much autonomy is safe?	Which customer actions require human review or escalation?

How Does Anthropic's Pacing Argument Apply to Customer Experience?

Dario Amodei's call to pace frontier AI development for safety research mirrors a narrower enterprise question: whether an AI agent's granted capabilities are outrunning an organization's ability to govern them.

What McKinsey and Microsoft Say About Governing AI Agents

Broader responsible AI research increasingly distinguishes between an AI system generating content and an AI system taking actions.

McKinsey's 2026 AI Trust Maturity Survey, based on responses from approximately 500 organizations, examines how agentic AI changes enterprise risk as systems gain greater autonomy and access to tools. The research emphasizes governance and controls as organizations move from generative AI toward agents capable of executing tasks.

That research provides outside context for the concern Heltewig raised on LinkedIn about what happens when models can act through external tools rather than merely propose an action.

Microsoft's 2026 Responsible AI Transparency Report similarly addresses the expanded risk landscape created when AI applications connect models to tools, services, data and other applications. Microsoft's responsible AI work emphasizes controls around areas including access, permissions and agent behavior. Microsoft's research addresses the same broader enterprise challenge Heltewig identified on LinkedIn: AI risk increasingly depends on the systems and permissions surrounding the model.

How Does Anthropic's Pacing Argument Apply to Customer Experience?

Dario Amodei's call to pace frontier AI development for safety research mirrors a narrower enterprise question: whether an AI agent's granted capabilities are outrunning an organization's ability to govern them.

Govern What's Already Running

Michael Weis, CRO at Synergy Group AI and a longtime CX and BPO executive, is approaching the current AI safety discussion from an operational governance perspective.

In a LinkedIn post on Sept. 14, Weis acknowledged warnings from Amodei, Sam Altman and Elon Musk but questioned whether slowing AI development addresses the most immediate enterprise problem.

"Why aren't we governing what's already running in our environments?”

Weis' LinkedIn post focuses on visibility across multiple models and agents. He asks whether organizations know which agent accessed particular data, which LLM it called, which user initiated the activity, what the interaction cost and whether the organization can reconstruct what happened when an agent goes off course.

Weis summarized his LinkedIn position this way: "Maybe the question isn't whether we need to slow AI down. Maybe it's whether we've put the controls in place to safely keep moving forward.”

Weis' LinkedIn argument overlaps most closely with the position Heltewig expressed in his LinkedIn post. Heltewig's post focuses on limiting and governing the tools and permissions available to the model. Weis' post focuses on maintaining visibility and control across the environment in which agents operate.

Related Article: Dear CMOs: Your Problem Isn't Your AI. It's Your Operating Model.

What Do McKinsey and Microsoft Say About Governing AI Agents?

McKinsey's 2026 AI Trust Maturity Survey and Microsoft's 2026 Responsible AI Transparency Report both emphasize governance and permission controls as AI systems shift from generating content to taking actions through connected tools.

Does Cutting Staff for AI Actually Pay Off?

Christian Macaro, an enterprise technology and analytics leader, brings the workforce side of the AI debate into CX.

In a LinkedIn post Sept. 14 prompted by Coxon's resignation from Anthropic and broader concerns about AI-driven job displacement, Macaro questioned whether using AI primarily to eliminate skilled employees produces a sustainable competitive advantage.

Macaro's LinkedIn thought experiment starts with a company replacing half of its workforce with AI and initially benefiting from lower costs. But if competitors gain access to comparable AI capabilities, he argues, that technological advantage eventually compresses.

Macaro's LinkedIn post presents a different long-term strategy: combine AI with skilled employees to create more valuable products, services and customer experiences.

He points in the LinkedIn post to IKEA's approach to automating routine customer service while moving employees into higher-value customer-facing work as an example of augmentation rather than straightforward substitution.

Macaro summarized the question in his LinkedIn post this way: "How much more value can our team create when empowered by AI?”

Macaro's LinkedIn argument differs from both of Heltewig's positions. Heltewig's September LinkedIn post focuses on the potential consequences of giving AI systems tools and permissions. His June CMSWire interview focuses on the enterprise infrastructure needed to operate those systems at scale. Macaro focuses instead on what businesses are trying to accomplish with AI in the first place — cost reduction or greater human and customer value.

"Look at IKEA: when AI took over routine customer call center queries, rather than initiating mass layoffs, they reskilled 8,500 workers into remote interior design consultants—turning a customer support cost center into a new revenue generator," he wrote.

Should Enterprises Slow AI Down or Govern What's Already Deployed?

Synergy Group AI's Michael Weis argues the more urgent question isn't pacing frontier AI but whether organizations can already trace which agent accessed which data, which model it called and what it cost.

How Much AI Will Customers Actually Trust?

Customer experience offers one of the clearest environments for seeing the gap between AI adoption and AI trust.

Five9's 2026 customer experience research, based on 3,000 consumers and 600 CX and contact center decision-makers in the United States, United Kingdom and Germany, found that 92% of organizations surveyed had implemented or piloted AI use cases in customer service. Five9 also found that 80% of consumers were willing to use AI-powered customer service, while two-thirds still preferred speaking with a human. The research highlights transparency, choice and effective escalation to humans as important parts of customer trust.

Qualtrics research involving more than 7,000 consumers provides another warning about customer-facing AI. Its research found higher failure rates for AI-powered customer service than for other AI applications studied and identified understanding as a critical weakness for AI agents.

Those findings provide real-world context for the issue Heltewig raised in his September LinkedIn post. A customer-facing AI system does not need to approach superintelligence for its permissions to matter. An incorrect interpretation becomes more consequential when an agent also possesses the ability to act on customer information or enterprise systems.

"I think we should get a lot more serious about that: who wires the agent up, and to what end," Heltewig said. "We as the international community need to decide what actions are acceptable and which aren’t. Deliberate misuse and negligent deployment should carry consequences, whether the responsible operator is a company or a nation state. Keeping a model away from a gun means more than withholding the gun. It means controlling who can hand it one — and securing the seemingly harmless systems it could use to build one."