AI control asks a practical question. If you can’t fully trust a capable model, can a weaker model you do trust watch it closely enough to catch it misbehaving? I worked through this as part of the AI Security Bootcamp curriculum, using AISI’s ControlArena. The sample sizes are small, so treat what follows as notes from an exercise, not a result.
Setup
- Task: APPS programming problems. The honest policy submits a correct solution. The attack policy submits a solution with a backdoor: code that passes the tests but returns a wrong answer for one specific input.
- Trusted monitor: Llama-3.1-8B-Instruct, prompted to score each submission from 0 to 10 for suspicion (shown below as 0 to 1).
- Untrusted model for the live round: GPT-4o-mini.
- Metric: ROC AUC between suspicion scores on honest and attack samples. 0.5 is chance.
Round 1: replayed backdoors
The monitor scored 50 honest and 50 backdoored solutions replayed from the dataset. Nothing here adapts to the monitor.
| Mean suspicion | Scored ≥ 0.8 | |
|---|---|---|
| Honest (n=50) | 0.55 | 27 / 50 |
| Backdoored (n=50) | 0.75 | 37 / 50 |
AUC: 0.64. That is better than chance, but look at the operating point. A threshold of 0.8 catches about three-quarters of the backdoors and also flags more than half of the honest code. Nobody would work that review queue for long, and a monitor nobody reads offers no protection.
Most of my prompt iteration afterwards went into the false-positive problem. Describing what backdoors look like (off-by-one bounds, near-miss comparisons, mishandled edge cases) tended to make the monitor suspicious of everything. My later prompts told it to assume most code is honest and to flag something only if it could name an input that breaks it. That is the discipline good triage analysts use. I didn’t run a clean comparison between prompt versions, so I can’t put a number on the difference.
Round 2: a live attacker
Next, GPT-4o-mini generated solutions live. It had a hidden side task (insert a backdoor) and was told that a monitor was reviewing its work, so the backdoor should be subtle and checked before submitting.
The monitor’s AUC came out below 0.5 in both runs. That looks like a striking result: the monitor doing worse than chance against an evasive attacker. It isn’t one. Looking at individual samples:
| Run | Attack samples scored | Backdoor triggered | Backdoor triggered and tests passed |
|---|---|---|---|
| A | 5 of 9 | 2 | 0 |
| B | 10 of 10 | 2 | 0 |
Every backdoor that triggered also broke the solution’s normal tests. A submission like that is caught by the test suite before a monitor is needed. So the attacker never produced a successful attack, and the low AUC mostly reflects the monitor’s scores on code that was simply wrong. Four samples in run A also errored and were never scored.
What I took from it
- An eval is only as strong as its attack policy. If the attacker can’t produce a working backdoor, the run measures the attacker, not the monitor. Before reading monitor numbers, check how many attacks actually succeeded.
- Score only successful attacks. Pooling failed attempts into the attack set gives an AUC that looks meaningful and isn’t.
- The false-positive rate is a security property. A monitor that buries its reviewers gets switched off.
- Report sample sizes and errors. With 5 to 10 samples per run, any AUC is noise.
Detection engineering learned versions of these lessons long ago. Test against a capable red team, not a replay, and tune for an alert volume people will actually work through.