A small DistilBERT classifier sits between an AI agent and the web. Every search passes through it; dangerous queries are blocked and reported back. We expanded the data, built a real evaluation harness, and ran an autonomous train–eval–fix loop on a single NVIDIA GPU — then proved the gains held on a second held-out set the model had never seen. The model runs on CPU in production, so it deploys anywhere.
Software agents have moved from answering questions to taking actions — and among the first tools they are given is a web search.
A chatbot returns text. An agent with a search tool reaches the open internet, and the query it generates is the action: run as-is, with no judgement, by the underlying service. Content moderation traditionally works after retrieval, filtering what comes back. With agents, the harm can begin at the query — in the search itself, and in whatever the agent then does with the results.
Three forces make this surface grow rather than shrink:
A classifier at the query layer is the guardrail that fits how agents actually work. It sits between the agent and the web, decides in tens of milliseconds, and turns a would-be harmful search into an error the agent can reason about — before any result is fetched.
An agent with a search tool will run any query it is given. A web_search returns results for "where to buy stolen oauth tokens" the same as any other query. The guard classifies each one in roughly thirty milliseconds, then allows or blocks it.
A classifier's quality isn't one number — and "95% accurate" can still mean CSAM reaches the web. Here is the vocabulary, and the metric that mattered.
Simple but misleading. A model that says "SAFE" to everything can still score 60% — and be entirely broken. We don't lead with this.
Of everything we blocked, how many were truly dangerous? High precision means we are not blocking queries that should run.
Of everything truly dangerous, how many did we catch? This is the safety score. Low recall means leaks. We optimized hardest here.
The harmonic mean of precision and recall — unforgiving by design. You can't inflate it by trading one for the other. The number we tracked across iterations.
"How to kill time" blocked because of the word kill. Agent friction — and for a defense agent, blocking research queries would cripple it.
A dangerous query the guard let through — for example, one seeking to buy illicit drugs. We drove this from 25 to 0.
The original training script graded itself on a test split taken from the training file — in-distribution — and reported F1 0.98. Real performance on held-out data was 0.80. That eighteen-point gap separates a notebook result from a production-ready model.
Six iterations, each targeting the misclassifications the previous one left behind. The table gives the exact figures; the chart shows the trend.
| Stage | DANGER F1 | Leaks (FN) | Safe blocked (FP) |
|---|---|---|---|
| Deployed (live hub model) | 0.800 | 25 | 2 |
| v0 — local retrain, expanded data | 0.919 | 5 | 8 |
| iter1 — reporting, idiom, cyber | 0.956 | 3 | 4 |
| iter2 — idiom, oblique violence | 0.968 | 3 | 2 |
| iter3 — self-defense cover, hate | 0.981 | 0 | 3 |
| iter4 — cyber-scanning fix | 0.988 | 0 | 2 |
| iter5 — final | 0.994 | 0 | 1 |
This guard also protects a cyber-defense agent, so the boundary is intent, not keywords. Researching how an attack works is safe. Stealing, buying, or selling credentials is dangerous. We planted these as contrastive pairs so the model learns the difference.
Final model on the primary held-out set. Fifteen of sixteen categories are perfect; the only residual error is a single help-seeking false positive we chose not to overfit (see §06).
| Metric | Primary held-out set | Fresh held-out set |
|---|---|---|
| Accuracy | 99.27% | 100.00% |
| DANGER precision | 0.9875 | 1.0000 |
| DANGER recall | 1.0000 | 1.0000 |
| DANGER F1 | 0.9937 | 1.0000 |
| Dangerous leaks (FN) | 0 | 0 |
| Safe blocked (FP) | 1 | 0 |
Fair before/after on the fresh set neither model was tuned against: v0 scored 90.7% / F1 0.882 with 3 leaks; the final model scores 100% / 1.000 with 0.
We then attempted an iter6 to fix the remaining false positive — a domestic-violence "safely exit a threatening relationship" help-seeking query. The addition looked safe and targeted. It regressed both eval sets and let a dangerous query through again.
| Metric | iter5 (shipped) | iter6 (reverted) |
|---|---|---|
| DANGER F1 | 0.994 | 0.975 |
| F1 on fresh set | 1.000 | 0.971 |
| dangerous leaks | 0 | > 0 |
A single eval set would have let us ship a regression as an improvement. The second set is what caught it.
Real outputs from the final model — including the boundary cases that took five iterations to resolve: the idioms, the help-seeking reports, and the narrow cyber boundary.
The model is available on Hugging Face under the MIT license.
I design and deploy AI-powered systems for on-premise training and inference — from custom guardrails and classifiers to retrieval pipelines and tool-using agents. If this kind of work fits what you're building, let's talk.