faultgate 0.3.1 · fixes from the first outside test, 180 real tau2-bench runsRead the changelog →
faultgate docs

Trust & Calibration

How much to trust a verdict

Every check and judge here is scored against human failure-mode labels on 200 real agent runs (FAULTLINE P4). The table reports results on all 200 and on P5’s frozen test split (n = 60). Regenerate the numbers with scripts/calibrate.py.

Check Kind Positives TPR TNR κ (n=200) κ (P5 test, n=60)
search_loop rule 36 1.00 0.99 0.98 1.00
abstention rule 1 1.00 1.00 1.00 — (0 positives)
policy rule synthetic 4/4 P9 categories 0 FP on 290 real traces — —
context_overflow rule synthetic finish-reason and provider-error forms in tests/test_checks.py 0 FP on the 125 shipped real traces — —
wrong_answer golden 4 1.00 1.00 1.00 1.00
truncation golden 1 1.00 1.00 1.00 — (0 positives)
wrong_direction Laya judge 1 1.00 0.61 0.02 0.00

For comparison, the same search_loop question put to judges instead of a rule: Laya κ 0.09, GLM-5.3 (API) κ −0.03. Where a failure is structural, code beats every model we measured, which is why the default checks are rules.

Read the table with these caveats:

  • search_loop is solid on this task family. The human coder saw the same structural signals, so κ measures agreement with that coding, not ground truth from nowhere.
  • abstention and truncation have one positive each to learn from. abstention is a phrase list, so a paraphrase it does not know is a miss. It was tuned on question-answering runs and then narrowed after an outside test on tau2 retail support runs, where “cannot” and “unable to” fired on 21 of 180 final answers that were policy reminders, authentication refusals and explanations of what a tool cannot do. It now fires on 1 of those 180 (“couldn’t locate the … payment method”); the P4 numbers above are unchanged.
  • wrong_answer agrees with the human verdict on all 21 answered runs. It is case-insensitive substring containment. That is right for short factual answers and wrong for free-text ones. Say what “expected” means for your task before you trust it.
  • policy has no human-labelled positives. FAULTLINE’s agent rejects hostile calls before they become steps, so its traces never contain an executed attack. Recall is by construction on the P9 attack shapes. The number that matters is zero false positives on real runs.
  • wrong_direction is the only check that needs reading. The only local judge we have is noise on it: it fires on 39% of clean runs. It ships because the contract is the product. The day a local model scores κ ≥ 0.7 on it, that is a one-line adapter, and this harness is what proves it.
  • Laya’s context is 512 tokens. faultgate compacts the trace to fit (answer and termination first, middle steps elided) and counts every call that still overflows. The count is printed.

Full confusion matrices and Wilson CIs for the original P5 judges are in FAULTLINE projects/p05_judge/DECISIONS.md.

Measured on real agent traces

The two rule checks added in 0.2.0 were calibrated against FAULTLINE P18 on Exgentic real benchmark traces. A blind second rater (Claude Opus 5.5) scored them:

Check Kind Measured trust Source
repeated_call rule κ 0.95 vs a blind second rater (Claude Opus 5.5) on 97 real runs FAULTLINE P18
unrecovered_tool_error rule Caught 13 of 23 unrecovered errors confirmed by the blind second rater (Claude Opus 5.5) (recall 0.57, κ 0.54); misses plain-text errors and run-ending API failures (faultgate has no check for those yet; read flagged traces manually) FAULTLINE P18

Span error = error.type or OTLP status code 2; P18 measured error.type only.

The Checks Reference has the full section, including the 30-trace demo output.