Can selective audits cut human review without letting AI errors through?

Benchmarks support the mechanism, not deployment readiness

Tests covered 7,040 SORRY-Bench instruction-response pairs and 1,361 StrongREJECT jailbreak triples with human safety labels. They also used eight interview transcripts and four established text or image labelling tasks. Error rates generally tracked the chosen limit and the calibrated or optimised policies saved more audits than simpler baselines. However, two methods ran slightly above the target on StrongREJECT, and the optimised method was somewhat anti-conservative on ImageNet.

For organisations, selective auditing could make human review more affordable than checking every output. It is not a licence for unattended high-stakes automation: the experiments reuse fixed benchmarks, one qualitative task substitutes a stronger model for a human auditor, and the framework assumes the auditor is always correct. The assessment would change with prospective deployments using fallible reviewers, pre-specified costs and patient, worker or public-service outcomes. The work is an unreviewed preprint; no dedicated funding or competing-interest statement appears in the HTML version.[1]