AI RELIABILITY & EVALS PLATFORM

Non-deterministic models.
Deterministic decisions.

FAULTLINE measures how AI agents behave across repeated runs, turns that into rule-checked release decisions, and guards tool calls at runtime. Open-source tools, and services when you want us to run it with you.

Explore the platform Talk to the founder

Read the docs

OPEN SOURCE · MIT · FAULTLINE EVALS · FAULTGATE 0.3.1 · MCP-GUARD

ONE CONTINUOUS LOOP · EVALUATE → GATE → GUARD

LOOP

Evaluate01 / WORKBENCH

Measure the agent over repeated runs with a deterministic oracle

1,350 graded runs · deterministic oracle, no LLM judge

Gate02 / RELEASE GATE

Turn test runs into rule-checked release decisions in CI

PASS / FAIL / INCONCLUSIVE in CI

Guard03 / RUNTIME FIREWALL

Check tool calls at runtime against pinned tool definitions

Flags poisoned tool descriptions · blocks unknown tools and rug-pulls in enforce mode

One continuous loop: FAULTLINE Evals in the lab, faultgate in CI, and mcp-guard at runtime.

RESEARCH PROJECTS

684

TESTS

PAPERS

MODELS TESTED

01 / WHY NOW

Why now: empirical proof of the agent release problem.

Same model, same inputs, different results.

27 pts

Same model, same 30 inputs, five runs: pass rate moved from 0.60 to 0.87.

1 change

A new model, prompt or tool can break an agent that passed yesterday. Every one is a release.

0 decisions

Tracing tools show what happened. A person still has to decide whether to ship.

κ 0.98

Our code check for search loops agrees with human labels (κ 0.98, n = 200). Model judges on the same question: κ 0.09 or below.

02 / THE PLATFORM

Three open-source tools in one continuous loop.

One loop: evaluate, gate, guard.

01EVALUATE

Open source · MIT

FAULTLINE Evals

A reproducible research workbench for evaluating autonomous AI agents under repeated execution. Measures run-to-run variance with a deterministic oracle.

1,350 graded runs across 150 scenarios at temperature 0.0
Deterministic dual-condition oracle, no model judges in the loop
pass^k independence holds within 2.5 percentage points

Explore FAULTLINE Evals

02GATE

Open source · MIT

faultgate 0.3.1

A local-first release gate that checks traces for known failure modes, measures noise bands, and returns PASS, FAIL or INCONCLUSIVE in CI.

PASS / FAIL / INCONCLUSIVE with required sample size in CI
Business rules in policy.json as a hard block
Reads OTLP, Langfuse, LangSmith, Braintrust, Phoenix

Explore faultgate

03GUARD

Open source · MIT

mcp-guard

A local firewall for MCP tool calls. Sits between your AI client and MCP servers: logs every call, flags poisoned descriptions, and blocks rule violations.

Pins tool definitions on first run to detect rug-pulls
Flags poisoned descriptions and prompt injection heuristics
Blocks unknown tools and extra arguments in enforce mode

Explore mcp-guard

How the platform fits together

Models will stay non-deterministic. Our job is to make every decision about them deterministic.

Samir Sawarkar, Founder

03 / HOW IT WORKS

Three tools in one closed loop.

Block regressions before they merge, in three commands.

01Evaluate

Measure the agent over repeated runs with a deterministic oracle; get a noise band, not a single score.

02Gate

On every change, faultgate returns PASS, FAIL or INCONCLUSIVE in CI; business rules are a hard block.

03Guard

At runtime, mcp-guard checks tool calls against pinned tool definitions.

GITHUB ACTIONS · RUN IN CI

- uses: Faultline-lab/faultgate@main
  with:
    traces: traces/candidate.json
    band: traces/band.json
    policy: traces/policy.json
    comment: "true"

04 / SEE A RELEASE DECISION

The faultgate GitHub Action posts the verdict, failing traces, latency change and a local repro command on your pull request.

See exactly why a release failed, on the PR.

faultgate · comment on PR #214faultgate: FAIL

Gate: clean rate 0.30 vs band [0.80, 0.87] from 2 baseline runs · fail below 0.70 · p=0.000

Regressions (clean in every baseline run, not now): r-0214, r-0216, r-0226, r-0258, r-0279, r-0286, r-0292, r-0317, r-0320, r-0322, r-0324, r-0330, r-0332, r-0344, r-0350

21/30 traces failed · judge none · redaction builtin

CHECKS

check	fired
`search_loop`	18
`abstention`	0
`policy`	0
`repeated_call`	0
`unrecovered_tool_error`	0
`context_overflow`	0
`wrong_answer`	3
`truncation`	3

Failing traces (first 20 of 21)

scenario	trace	check	reason
`r-0336`	`4dc5e933abe8`	wrong_answer	expected 'Alto-5275' not found in answer
`r-0336`	`4dc5e933abe8`	truncation	answer has 'Alto' but not 'Alto-5275'
`r-0221`	`51c15300072b`	search_loop	no answer, 24 steps (cap), 62% empty results
`r-0286`	`b6f824a010a7`	search_loop	no answer, 24 steps (cap), 79% empty results
`r-0232`	`7e74e68d7296`	search_loop	no answer, 24 steps (cap), 58% empty results
`r-0216`	`10fa2370723e`	search_loop	no answer, 24 steps (cap), 50% empty results
`r-0292`	`10a63caed2d6`	search_loop	no answer, 24 steps (cap), 62% empty results
`r-0214`	`04a16662742e`	search_loop	no answer, 24 steps (cap), 67% empty results
`r-0263`	`b80e27d064fe`	search_loop	no answer, 24 steps (cap), 75% empty results
`r-0344`	`ab3503d28cda`	search_loop	no answer, 24 steps (cap), 62% empty results
`r-0320`	`4d6f5e788fe1`	search_loop	no answer, 24 steps (cap), 67% empty results
`r-0350`	`316bcde0d931`	search_loop	no answer, 24 steps (cap), 79% empty results
`r-0224`	`8fd42bacf992`	search_loop	no answer, 24 steps (cap), 67% empty results
`r-0332`	`09a946f2c89d`	wrong_answer	expected 'Gossamer-5255' not found in answer
`r-0332`	`09a946f2c89d`	truncation	answer has 'Gossamer' but not 'Gossamer-5255'
`r-0279`	`54f099f8147e`	search_loop	no answer, 24 steps (cap), 67% empty results
`r-0330`	`ac2a7d9eaf68`	search_loop	no answer, 24 steps (cap), 67% empty results
`r-0226`	`6f508650cff0`	search_loop	no answer, 24 steps (cap), 62% empty results
`r-0258`	`25465761c097`	search_loop	no answer, 24 steps (cap), 83% empty results
`r-0324`	`4c78f65e849d`	wrong_answer	expected 'Talc-5218' not found in answer
`r-0324`	`4c78f65e849d`	truncation	answer has 'Talc' but not 'Talc-5218'
`r-0322`	`8e4ab8196357`	search_loop	no answer, 24 steps (cap), 75% empty results
`r-0317`	`a3131ab0f714`	search_loop	no answer, 24 steps (cap), 79% empty results

LATENCY AND COST VS BASELINE

metric	baseline	this run	Δ
latency / run	16.5 s	21.9 s	+32.9%

REPRODUCE LOCALLY

pip install 'faultgate==0.3.0'
faultgate check examples/gate/candidate_gpt.json --band examples/gate/band.json --golden examples/gate/golden.json

faultgate · comment on PR #216faultgate: PASS

Gate: clean rate 0.87 vs band [0.80, 0.87] from 2 baseline runs · fail below 0.70 · p=0.642

4/30 traces failed · judge none · redaction builtin

CHECKS

check	fired
`search_loop`	2
`abstention`	0
`policy`	0
`repeated_call`	0
`unrecovered_tool_error`	0
`context_overflow`	0
`wrong_answer`	2
`truncation`	2

Failing traces (4/30)

scenario	trace	check	reason
`r-0221`	`30e9a0bd0f27`	search_loop	no answer, 24 steps (cap), 58% empty results
`r-0263`	`b314e7088369`	wrong_answer	expected 'Prism-4914' not found in answer
`r-0263`	`b314e7088369`	truncation	answer has 'Prism' but not 'Prism-4914'
`r-0274`	`3f7adb56dfc8`	wrong_answer	expected 'Sable-4967' not found in answer
`r-0274`	`3f7adb56dfc8`	truncation	answer has 'Sable' but not 'Sable-4967'
`r-0277`	`93b9ae6c03a6`	search_loop	no answer, 24 steps (cap), 58% empty results

LATENCY AND COST VS BASELINE

metric	baseline	this run	Δ
latency / run	16.5 s	16.0 s	-2.7%

REPRODUCE LOCALLY

pip install 'faultgate==0.3.0'
faultgate check examples/gate/baseline_glm_run2.json --band examples/gate/band.json --golden examples/gate/golden.json

PASS example: a baseline run checked against its own band.

verdict.jsonTRIMMED PREVIEW

{
  "faultgate_version": "0.3.0",
  "created_utc": "2026-10-13T13:49:21Z",
  "verdict": "FAIL",
  "exit_code": 1,
  "command": "faultgate check examples/gate/candidate_gpt.json --band examples/gate/band.json --golden examples/gate/golden.json",
  "inputs": [
    {
      "role": "traces",
      "path": "examples/gate/candidate_gpt.json",
      "sha256": "0f23010a2da86e7695e7c99b6748246aac13a36d28748ea1d76128eff9144b29"
    },
    {
      "role": "band",
      "path": "examples/gate/band.json",
      "sha256": "5cd572abf5efce5cdbc9d53acd4de7b5c7abafa3388b65813c32affa51e62a12"
    },
    {
      "role": "golden",
      "path": "examples/gate/golden.json",
      "sha256": "b70463146559496637f01e004cb35d485a1f33446f282964b8e0e59f104ddc0b"
    }
  ],
  "judge": "none",
  "checks": [
    "search_loop",
    "abstention",
    "policy",
    "repeated_call",
    "unrecovered_tool_error",
    "context_overflow",
    "wrong_answer",
    "truncation"
  ],
  "redaction": "builtin",
  "traces": 30,
  "failed_traces": 21,
  "fired": {
    "search_loop": 18,
    "abstention": 0,
… (57 more lines)

failing_traces.jsonTRIMMED PREVIEW

[
  {
    "trace_id": "4dc5e933abe820a352c63601a0319ce9",
    "scenario": "r-0336",
    "checks": [
      {
        "check": "wrong_answer",
        "reason": "expected 'Alto-5275' not found in answer"
      },
      {
        "check": "truncation",
        "reason": "answer has 'Alto' but not 'Alto-5275'"
      }
    ]
  },
  {
    "trace_id": "51c15300072be9563b8dcdae613d3f7f",
    "scenario": "r-0221",
    "checks": [
      {
        "check": "search_loop",
        "reason": "no answer, 24 steps (cap), 62% empty results"
      }
    ]
  },
  {
    "trace_id": "b6f824a010a79968e6d3403a3eb240f0",
    "scenario": "r-0286",
    "checks": [
      {
        "check": "search_loop",
        "reason": "no answer, 24 steps (cap), 79% empty results"
      }
    ]
  },
  {
    "trace_id": "7e74e68d7296e93d8e5e679451c99795",
    "scenario": "r-0232",
    "checks": [
      {
… (184 more lines)

repro.shVERBATIM

#!/bin/sh
# faultgate 0.3.0 · verdict FAIL · exit 1
pip install 'faultgate==0.3.0'
faultgate check examples/gate/candidate_gpt.json --band examples/gate/band.json --golden examples/gate/golden.json

faultgate check — terminal

$ faultgate check examples/gate/candidate_gpt.json --band examples/gate/band.json --golden examples/gate/golden.json --markdown pr-fail.md --audit audit-fail

…

21/30 traces failed · search_loop=18, abstention=0, policy=0, repeated_call=0, unrecovered_tool_error=0, context_overflow=0, wrong_answer=3, truncation=3 · judge none · 0.0s

gate: FAIL · clean rate 0.30 vs band [0.80, 0.87] from 2 baseline runs · fail below 0.70 · p=0.000

regressions (clean in every baseline, not now): r-0214, r-0216, r-0226, r-0258, r-0279, r-0286, r-0292, r-0317, r-0320, r-0322, r-0324, r-0330, r-0332, r-0344, r-0350

markdown written to out/pr-fail.md

audit written to out/audit-fail

Real output of faultgate 0.3.0 on the examples in its GitHub repo. PR number is illustrative.

Reproduce it

05 / SERVICES

Use the tools for free, or have us run the work with you.

Want us to run it with you?

Readiness Review

$7,5005 business days

We write your policy.json rules, import your traces, run 3+ baselines, and deliver your verdict and audit pack.

Talk to the founder →

Custom eval suite build

Contact usScope and price agreed per engagement

Scenarios built from your agent's real tasks with deterministic checks and CI integration in your repo.

Talk to the founder →

Tool-poisoning red-team

Contact usScope and price agreed per engagement

Attack battery across MCP tool-description injection, function hijacking, and parameter injection with paired reports.

Talk to the founder →

Ongoing reliability retainer

Contact usScope and price agreed per engagement

Periodic re-baselining when models change, release verdict reviews, and rule-pack maintenance.

Talk to the founder →

All services

06 / WORKS WITH YOUR STACK

Point it at an export from the tool you already use.

Reads the traces you already record.

OpenTelemetry--format otlpLangfuse--format langfuseLangSmith--format langsmithBraintrust--format braintrustArize Phoenix--format openinferenceGitHub Actionsuses: Faultline-lab/faultgate@main

07 / FINDINGS

What the evidence says. Four results, each bound to a frozen protocol.

What 1,800 paired runs taught us about agent failure.

−21.9 pp

POOLED ATTACK SUCCESS, ARM A → B

A client-side provenance contract on tool-call arguments removed 421 attack successes and induced 26. McNemar exact p = 5.6e-93, six models, 1,800 paired runs.

2.5 pp

PASS^K VS NAIVE INDEPENDENCE

Repeated trials compound almost exactly as (pass@1)^k at k = 3. 150 held-out scenarios, 1,350 runs, deterministic oracle, no model judge.

94 %

FAILURES OWNED BY THE RETRIEVER

32 of 34 R2 failures never surfaced a required chain document. A stronger model does not fix that; oracle retrieval rescues all 32 at 3.1× fewer tokens.

0.60 – 0.87

SAME MODEL, FIVE RUNS, PASS RATE

Serving variance moves a fixed model across 27 points. Release gates must be tolerance bands graded on a pinned golden set, not thresholds.

Paper 01 — Multi-trial reliability ↗Paper 02 — Runtime provenance contracts ↗

08 / SECURITY

faultgate runs where your CI runs.

Your traces stay in your CI.

Turn on an API judge and only a redacted copy goes to your model provider.

Local-first

faultgate reads traces in memory and writes no trace payloads anywhere. Rules, policy, bands and the local judge never send anything off the machine.

Redacted by default

anything that leaves (API-judge input, reasons in CI logs and PR comments) is masked: emails, Luhn-checked cards, SSNs, phones, IPs, API keys, bearer tokens, JWTs. Presidio optional.