We verified the verifier

If the referee can be wrong, none of the numbers mean anything.

So we examined it. A 30-claim expected-verdict set covering the rubric's edges — verbatim and paraphrase accepts, advisory-contradiction and memory-silent flags, verified-contradiction and tripwire vetoes (EN + KR) — scored two ways: a deterministic harness and Google ADK eval (EvalSet + AgentEvaluator) on live Vertex.

100%
verdict accuracy× 3 Vertex runs
100%
run-to-run consistencysame verdicts every run
30
claims across the edges10 accept · 9 flag · 11 veto
96.9→100
after the eval caught a bugwe fixed the rubric, not the score
Per class

Precision and recall, every verdict class.

✓ acceptn=10
precision 1.00recall 1.00
▲ flagn=9
precision 1.00recall 1.00
✕ veton=11
precision 1.00recall 1.00
By category — where the hard cases live

The edges, not the easy middle.

Every category is a place a naive memory layer fails: paraphrase mistaken for contradiction, a money claim rationalized by a model, a silent memory over-trusted.

CategoryCasesCorrectAccuracy
accept · verbatim44100%
accept · paraphrase44100%
accept · paraphrase (seeded)22100%
flag · advisory contradiction44100%
flag · memory silent55100%
veto · tripwire (money/auth)77100%
veto · verified contradiction44100%
Scored on Vertex AI · model gemini-flash-latest · 30 cases × 3 runs · tripwire exact, word-boundary negatives hold · mean panel latency ~27.5 s. Reproduce offline (deterministic): SENTINEL_REPLAY=1 python eval/run_eval.py --runs 3
The honest part

The eval caught a real bug — in us.

On an early run the harness flagged a Korean financial statement as a money action and over-vetoed it — a real failure mode, surfaced by the eval, not hidden by it. We tightened the rubric (a tripwire word-boundary fix) and accuracy moved 96.9% → 100%. The gate gating us was the system working.

This is Track 2, "optimize an existing agent," applied to the optimizer itself: we used Google's own eval tooling to harden the very component that judges everyone else. The eval artifact — eval/adk_results.json — is committed to the repo, GREEN across three live runs. Every number on this site is reproducible from the code.

Now do it yourself
Type a claim the eval never saw and watch the referee rule on it live.
Open the playground