A probe spotted agents about to leak a credential—but stopping them came at a cost

A useful warning is not yet a safe shutdown policy

The result weakens when the alarm has to govern a whole agent. Qwen episodes that never leaked contained a median 23 scored calls, so small per-call false-positive rates accumulated: 30 of 51 refusing episodes received at least one alarm. A stricter threshold limited alarms to two refusing episodes but caught only 10 of 38 leaking episodes. The warning also arrived mainly on the leaking call itself; prediction one call earlier was substantially weaker.

The probe largely detected an imminent disclosure rather than whether disclosure violated the rule. Permitted sharing could also trigger it, and a trained bag-of-words classifier came close to the activation probe, suggesting much of the advantage came from labelled training rather than privileged access to model internals. For security teams, the evidence supports further testing of pre-output review—not unattended deployment. Confidence would require independent replication, more models, realistic tools and credentials, adversarially reworded leaks, and alarm policies validated on previously unseen workplaces.[1]