A reproducible research workbench for evaluating autonomous AI agent reliability under repeated execution and defending against Model Context Protocol (MCP) tool poisoning.
01 / KEY FINDINGS
On 300 MCPTox instances across six models, a client-side runtime provenance contract cut attack success from 30.32% to 7.41% (McNemar exact p = 5.61 × 10⁻⁹³).
Empirical test of naive Bernoulli compounding pass^k = (pass@1)^k matches joint reliability within 2.5 percentage points at k=3 across held-out hard scenarios.
150 held-out hard Tier-3 scenarios evaluated across k=3 repeated trials at pinned temperature 0.0 with a deterministic dual-condition oracle.
02 / SCIENTIFIC METHOD
Every experimental result is bound to immutable contracts, cryptographic hashes, and reproducible seeds.
Evaluation protocol frozen in MEC.md; all primary hypotheses registered prior to execution in HYPOTHESES.md; protocol adjustments logged in AMENDMENTS.md.
Grounded answer verification uses deterministic normalization and verified citation hashes against a frozen corpus (SHA256 84e6ff590704aa94d10912f8002716b14c7436080e1a3270b53f9385dc728efc, master seed 42).
Pinned model pricing, per-project budget caps, and append-only ledgers (ledger.jsonl) recording timestamp, model, token counts, and USD cost per call.
Both publications passed independent 7-gate red-team audits before publication: Publication 1 Checklist and Publication 2 Checklist.
03 / PUBLICATIONS
Grounding Collapse, Failure Taxonomy, and Evaluator Calibration
Controlled empirical test of the independence assumption pass^k = (pass@1)^k on a contamination-resistant 5-hop knowledge graph benchmark. Evaluated across 150 hard scenarios with k=3 repetitions (1,350 total runs) graded by a deterministic dual-condition oracle without subjective model judges in the loop.
A Paired Evaluation on MCPTox Across Six Models
Evaluated an independent defensive second arm on the Model Context Protocol Poisoning benchmark (MCPTox), testing whether a client-side runtime provenance contract on tool-call arguments with call-only retry gating mitigates tool poisoning across six model rungs. Cut pooled valid attack success from 30.32% to 7.41%.
04 / PROJECT BOARD
| ID | Project | Focus | Status |
|---|---|---|---|
P01 | Baseline | Grounded multi-hop agent evaluation baseline | Complete |
P02 | OTel Exporter | OpenTelemetry GenAI semantic conventions & spans | Complete |
P03 | Grounding Zero-Point | Multi-hop depth grounding decay across T1–T3 | Complete |
P04 | Failure Taxonomy | Human-in-the-loop trace coding (8 failure modes) | Complete |
P05 | Evaluator Calibration | LLM judge calibration & Rogan-Gladen correction | Complete |
P06 | Multi-Trial Reliability | pass^k independence testing & failure clustering (Pub 01) | Complete |
P07 | Serving Variance | Temperature-0 serving nondeterminism across providers | Complete |
P08 | MCP Defense | Client-side runtime provenance contract on MCPTox (Pub 02) | Complete |
P09 | Red-Team Regression | Document-text injection suite, 10 categories, two arms; regressions as strict xfails | Complete |
P10 | Calibrated Cascade | Cheap→frontier router on deterministic signals; Pareto frontier | Complete |
P11 | Release Gate | Tolerance-band gate: sha256 golden set, pinned oracle judge, live upgrade drill | Complete |
P12 | Failure Attribution | Retriever-vs-generator ablation with oracle retrieval (H8) | Complete |
P13 | SLO Incident | Multi-window burn-rate SLO alerting & incident triage | Complete |
P15 | Resilience | Adaptive circuit breakers & jittered backoff policies | Complete |
P16 | Runtime Policy | Deterministic call allowlists, path containment, token caps | Complete |
05 / REPRODUCE
All test suites and benchmark dry-runs execute locally without network access or paid API credentials:
make venv # Create virtualenv and install pinned requirements
make test # Run 528 Phase 1 unit/integration tests
make phase2-test # Run 156 Phase 2 tests (P00-P08, P09, P10, P11, P12, P13, P15, P16; 5 strict xfails)
make p06 ARGS=--dry-run # Dry-run P06 multi-trial sweep
make p07 ARGS=--dry-run # Dry-run P07 variance analysis
make p08 ARGS=--dry-run # Dry-run P08 MCPTox evaluation
make p08-replay # Replay cached MCPTox execution traces
make p09 ARGS=--dry-run # Dry-run P09 red-team regression suite
make p09-rescore # Re-score P09 results from disk without model calls
make p10-simulate # Simulate P10 cheap-to-frontier cascade
make p11 ARGS=--dry-run # Dry-run P11 release gate
make p11-drill # Run P11 CI release-gate stub drills (both pass & fail)
make p12 ARGS=--dry-run # Dry-run P12 failure attribution06 / LIMITATIONS
This repository reflects the experimental findings of a single author across a controlled set of commercial model endpoints accessed through a single unified API gateway. Evaluator calibrations for qualitative classifications were performed on held-out splits without multi-annotator human consensus panels. Model responses and serving latency are subject to cloud provider infrastructure drift and nondeterministic GPU reduction scheduling. Findings should be validated across diverse gateways and multi-turn human task distributions.
Every number, hypothesis and chart reproduces from open source code on GitHub.