When Agents Go Rogue

Treat agents as unvetted workers, for their whole lifecycle.

Give every agent its own identity, register it, give it an owner, give it time-bound permissions, and make its behaviour observable. Expire permissions with an alert, and retire the agent and its credentials when its job ends. A forgotten, over-permissioned agent is an unmanaged access path for an attacker or a rogue instance. You would not let an unknown contractor act on your systems without a badge, a log and a leaving date, and an agent is no different. Time-bound permissions with expiry and a renewal alert are the right antidote to the permissions-review fatigue that can leave agents running after their task has ended or their owner has departed.

Understand the risk before you deploy.

Not every agent is equally dangerous when it goes wrong. An agent that drafts slides or summarises documents has a small potential blast radius; an agent with production credentials, code execution or offensive-security tooling has a larger one. In your threat modelling and risk assessment, tier your agents by what they could do if compromised, manipulated, or over-optimising for their objective, and by what they can actually reach. That tier informs, and should make explicit, how much isolation, monitoring and enforcement each agent needs.

Shrink the blast radius before the agent ships.

Vulnerability exploitation was a trigger here, when the models broke out through a zero-day in the package proxy, so a strong security baseline with vulnerability scanning and patching remains essential; this is the fundamentals-first posture the NCSC continues to urge, and the UK’s AI-assisted vulnerability-discovery work shows how far that baseline can now be pushed [6] [7]. But given the capabilities now emerging, including the rapid evolution of open-weight models, we should not treat it as foolproof prevention. Vulnerabilities will be found and exploited; that is precisely why reducing the blast radius matters so much.

The most effective control is the one that leaves a rogue agent with less to reach. Apply least privilege, time-bound and just-in-time permissions, isolation and segmentation, and remove the ambient credentials an agent can scavenge, so that a compromised or over-optimising agent inherits as little reach as possible.

Do this before deployment, not after: threat-model the agent and scan the environment as a condition of going live. The risk is not hypothetical: controlled research by Unit 42 shows how a malicious agent could abuse a Vertex AI service agent’s excessive default permissions [13], and the PocketOS coding agent deleted a production database using an over-scoped token it found in an unrelated file [11].

Rethink guardrails.

Content guardrails and model refusals still matter, but they address a different problem from rogue agency: the input and output surface, where someone feeds the model malicious input in the hope of harmful output. A rogue agent’s harm is in its actions, the tool calls that read, write, delete and move laterally, and it may be pursuing a goal it treats as legitimate.

So guardrails are far from redundant here. ASI01, ASI02, ASI05 and ASI06 treat natural-language inputs and outputs, retrieved content, tool results, memory, context, and model-generated code and commands as untrusted; those surfaces still need validation, sanitisation, isolation and monitoring. Anthropic notes that the safeguards shipped with its generally available models would have blocked the behaviour it found, which reinforces their value [3].

But guardrails are not a complete authority system. They may interrupt a harmful trajectory; they do not decide whether this agent may perform this action on this resource. That control belongs at the action layer, through identity, permissions, runtime policy and containment.

Enforce runtime security at machine speed.

Runtime security is the right complement, but how you roll it out matters. You cannot meet machine-speed activity with human-speed approval. Shrinking the blast radius decides what an agent can reach; this decides what it may do with that reach.

Policy-as-code, using an engine such as Cedar or Open Policy Agent, becomes an essential defence when it is enforced by agent-action interceptors, so that a call is checked against policy and against the surrounding detection context, before it takes effect.

At a minimum, check the consequential actions: any write, update, execute or delete. Implementations are emerging at different levels of maturity. You can start with platform primitives: Amazon Bedrock AgentCore Gateway interceptors [14], the Microsoft Foundry Control Plane [15], or Google’s before_tool_callback in the Agent Development Kit (ADK) [16]. These vary in granularity and depth.

Dedicated offerings go further: Microsoft’s excellent open-source, multi-platform Agent Governance Toolkit is a full reference implementation of what can be done [17], and CSA AARM-aligned projects, both commercial and open source, offer turnkey products to evaluate [18]. The Agent Control Standard (ACS), recently accepted as a workstream within the OWASP GenAI Security Project’s Agentic Security Initiative, is an emerging standard poised to standardise this area [19]. Start with platform capabilities for an immediate baseline while evaluating dedicated offerings.

Keep a human in the loop for high-consequence actions.

Runtime policy can enforce repeatable rules, but a narrow set of irreversible or high-value actions warrants human authorisation, so machine speed never carries an unrecoverable action on its own. Choose that set deliberately and keep it small: if approval fatigue sets in, people will click through it and it stops being a real control.

Extend monitoring, SOC and incident response to insider-style threat at scale.

Treat this as an insider problem, because a compromised or over-optimising agent behaves like one. Many organisations forward telemetry to the SIEM only for public-facing systems; extend that to internal agent activity, and enrich it with the denials and escalations that runtime enforcement produces.

Consider canary tokens on sensitive paths as high-fidelity tripwires. Wiring them to automatic containment can stop a rogue agent early, but automated revocation carries its own availability risk, since a false positive could disrupt a legitimate service, so test any such automation and make its use risk-based rather than universal. And prepare an incident-response playbook for agentic incidents specifically, rather than adapting a human-attacker template.

Maintain a dependable but controlled AI cyber capability.

Defensive analysis of malware and exploit code can trigger commercial models’ cyber-safety refusals when they lack enough context to distinguish a responder from an attacker; Hugging Face hit this and fell back to an open-weight model on its own infrastructure [1]. So decide your dependable, pre-approved defensive-analysis path in advance.

Vendor trusted-access programmes, such as OpenAI’s Trusted Access for Cyber [20] and Anthropic’s Cyber Verification Program [21], can relax the refusals for verified defenders, but vary in their retention, approval and scope terms, so confirm the current terms for the exact service. A gated model such as Anthropic’s Mythos [22] is a separate option available to vetted partners under trusted-access arrangements. The alternative is a model under your own operational control, which keeps both the decision and the data with you but comes with considerable investment and risk of its own.

A managed platform such as Amazon Bedrock [23] or Microsoft Foundry [24] sits between the two, so verify the specific data boundary of the exact service rather than assuming nothing leaves. Fireworks currently offers GLM 5.2 through a serverless API [25], although it remains a third-party service rather than your own tenancy. Validate whichever you choose before relying on it, confirming it will parse the hostile artefacts you expect to handle; the UK AISI’s open-source Inspect framework is a sound basis for that check [26].

Handle provenance separately, as a supply-chain control, verifying origin and integrity with signatures, hashes and artefact scanning, so your incident-response tool is not itself a supply-chain risk. And govern the model itself as an access-controlled, logged, dual-use asset.

Use this capability for incident response, red teaming and blast-radius exercises. We built AIRt-Eval on the UK AI Security Institute’s Inspect framework to evaluate models as attackers and scorers in AI red teaming. It is designed to replace reliance on defaults with task-specific evidence.

Validate your capability using the way agents actually behave.

Run user-journey-style reviews and offensive-security incident drills built around agentic behaviour to check whether your monitoring and detection actually fire. This matters most where SOC and monitoring coverage is fragmented, which is exactly where a rogue agent can pass unnoticed. Include a check that you can generate chain-of-custody evidence during the drill. The UK’s applied vulnerability-discovery pilot points the same way: GC3 reported that architecture mattered most, model choice mattered less, and triage and human review remained essential [7]. Validate your full security model, not just AI model and CI/CD world but also monitoring, containment and incident-response readiness for rogue agency.

Rethink red teaming for agentic systems.

Traditional AI red teaming, built on black-box assumptions, struggles here. Grey-box testing, working with access to telemetry and knowledge of the agent’s tools, permissions and policies rather than probing blind, exposes far more, as a recent engagement with a regulated financial-services client proved. Build rogue-agent behaviour into your test cases, and run them in a production-like sandbox rather than against live systems to find the gaps.