faultgate 0.3.1 · fixes from the first outside test, 180 real tau2-bench runsRead the changelog →
faultgate docs

CLI Reference

Usage

faultgate check traces.json                          # built-in rule checks
faultgate check traces.json --json report.json       # full per-trace verdicts
faultgate check traces.json --golden golden.json     # expected answers: correctness + truncation
faultgate check traces.json --policy policy.json     # what the agent was allowed to do
faultgate check traces.json --judge laya             # also run judge checks, locally
faultgate check traces.json --judge litellm:openai/gpt-5.6-luna   # any API model
faultgate check traces.json --markdown pr.md --audit audit/   # PR comment + audit files
faultgate precheck traces.json                       # validate an export before gating
faultgate checks                                     # list checks and their kind
faultgate judges                                     # list judge specs

faultgate baseline run1.json run2.json run3.json --out band.json   # measure your noise (3+ runs)
faultgate check candidate.json --band band.json          # regression, or just noise?

Exit codes: 0 clean · 1 at least one check fired (or, with --band, gate FAIL) · 2 bad input, or an output file (--json, --markdown, --audit, --out) that cannot be written. Missing parent directories of output files are created.

Commands

  • check: runs the checks over a trace export. Add --band to gate against a baseline band.
  • precheck: validates an export before you gate on it.
  • baseline: builds a noise band from 3 or more runs, written with --out.
  • checks: lists the built-in checks and their kind.
  • judges: lists the judge specs.

Flags

Flag Applies to Purpose
--json <path> check Full per-trace verdicts. The report carries the redaction mode and, with --band, the gate stats.
--golden <path> check, baseline Expected answers, keyed by scenario or by prompt text. Enables wrong_answer and truncation.
--policy <path> check Policy file: allowlist, denied patterns, call budget, business rules.
--judge <spec> check Runs judge checks. laya runs locally. litellm:<model> calls an API model.
--markdown <path> check Writes the PR comment as Markdown.
--audit <dir> check Writes verdict.json, failing_traces.json and repro.sh.
--band <path> check Gates the candidate against a band from baseline.
--out <path> baseline Where the band is written.
--format <format> check, baseline auto (default), otlp, openinference, langfuse, langsmith or braintrust.
--redact <mode> check builtin (default), presidio or off. presidio needs pip install 'faultgate[pii]'.
--sample <N> precheck Number of traces to inspect. Default 5. 0 checks all of them.

Precheck

faultgate precheck inspects the first 5 traces by default. Use --sample N to change that, or 0 to check every trace. See Trace Formats & Ingestion for what it reports.

$ faultgate precheck examples/formats/langfuse.json
precheck: OK · format langfuse · checked 1 of 1 traces · 2 tool steps read · 0 error(s), 0 warning(s)

Redaction

--redact builtin|presidio|off controls what is masked in API judge prompts and in every verdict reason. The default is builtin. See Data Privacy & Redaction for what each mode masks.

CI parity

The same reports come from the CLI with --markdown summary.md --audit audit/. The GitHub Action wraps these commands. See CI & GitHub Action Integration.