Earlier this year I built a small LLM-driven penetration-testing assistant: a model wired to standard offensive tooling over MCP, with a guided mode (the model proposes, a human runs) and an experimental autonomous mode where the agent loops toward an objective on its own.
Autonomous mode needed guardrails, so I wrote the obvious thing: an action classifier that puts every proposed tool call into one of three buckets.
| Class | Behaviour | Examples |
|---|---|---|
SAFE |
Runs automatically | In-scope scanning, enumeration |
NEEDS_APPROVAL |
Pauses for a human | Exploitation, brute force, DB writes |
FORBIDDEN |
Blocked | Out-of-scope targets, local destruction |
It looks like a sensible rules-of-engagement layer. Read with an attacker’s eye, it shows how not to contain an agent.
What the classifier actually checks
- Scope is a substring match. A call is in scope if any scope entry appears anywhere in the command string. A command naming an in-scope host and an out-of-scope one passes.
- No scope means everything is in scope. The safe default for an unconfigured run should be “nothing runs”, not “everything does”.
- Danger is a deny-list. Destructive actions are recognised by regex (
rm -rf,mkfs,shutdown…) and risky ones by keywords (exploit,payload,reverse shell). The same effect can be had with an interpreter one-liner, an encoded string, or a tool the list has never heard of. - It inspects text, not effects. The classifier sees what the model says it will run, not what the process does. DNS lookups, redirects, second-stage downloads and child processes are all invisible to it.
None of these are exotic. They are the first things a red teamer tries against any allow/deny filter, and a capable model can stumble into them while pursuing an objective, with no malice involved.
The underlying mistake
A deny-list assumes the thing being filtered is cooperative and only occasionally wrong. An autonomous agent is better modelled as a credentialed insider with an objective: it will route around obstacles because routing around obstacles is what solving the task looks like.
Pattern-matching on the text of a command is the wrong layer. Here is what I would build now, roughly in order:
- Enforce scope at the network, not the prompt. Put the agent in a sandbox whose egress is physically limited to the authorised range. Then scope violations are impossible rather than merely detected.
- Default deny. No scope configured, no execution.
- Allow-list capabilities, not deny-list strings. Expose narrow tools (“scan host X with profile Y”) instead of a raw shell wherever the task allows.
- Monitor trajectories, not single calls. Many harmful outcomes are a sequence of individually innocuous steps. Catching those needs a monitor that reasons over the whole run, which is where AI-control work comes in.
Why this matters beyond one tool
Capability evals put models inside ranges and ask them to break things. The containment around those ranges is the same problem at higher stakes, and models have already found gaps in it. In one case reported in the o1 system card, a CTF challenge container failed to start. The model scanned the network, found the evaluation host’s Docker API exposed, and started a new copy of the challenge container with a command that printed the flag. OpenAI notes that its wider infrastructure stayed secure. The model still reached its goal by a route nobody intended.
A follow-up note covers the monitoring side: what happens when you point a weak trusted model at a motivated attacker.