A Passing Agent Contract Check Does Not Prove Safe Production Operation: Turning Evidence into Sales Claims
Tomohiro Iida ·
A technical team wants to put a successful AI-agent test in a proposal. It is tempting to write, “Prevents agents from exceeding their permissions.” But if the test only detected a contradiction between a declared task and a declared permission, that sentence silently promises runtime enforcement that has not been established.
Netsujo's public Agent Role Contracts project offers a concrete example. This article builds a hypothetical proposal-copy revision from public source, tests and historical CI records. It does not allege that an actual customer proposal contained the exaggerated wording.
Begin with one bounded test
The October 15, 2026 source snapshot `966e3c2` includes a quick-start demo and starter input files. The implementer's declared write scope is src/**; the reviewer is declared read-only.
| Input or change | Assertion in the published source | What it establishes |
|---|---|---|
Task points at src/example.mjs | Requires PASS | The task agrees with the declared write scope |
Task points at secrets/production.txt | Requires TASK_WRITE_SCOPE_OUTSIDE_AUTHORITY | That declaration is rejected as outside the allowed scope |
Restore task scope to src/example.mjs | Requires PASS again | The declared contradiction has been removed |
The repository README shows PASS → FAIL (expected) → PASS as expected demo output. We did not rerun the current source on a host for this article. There is separate historical execution evidence: the public September 30, 2026 CI run logged a passing starter out-of-scope rejection test; the Ubuntu/Node 24 job recorded 198 passing tests and zero failures for that earlier commit. It must not be relabelled as a CI result for the October 15 source snapshot.
That provenance distinction determines how much the proposal may say.
Why use declaration checks, and why stop short of a safety guarantee?
A prompt instruction such as “only change files in src” does not itself provide a machine-readable diagnostic when the next task asks for something else. Agent Role Contracts checks the task against explicit role and authority declarations before an executor is asked to act.
But the demonstration does not open the requested files or start an agent. Its Quick Start documentation separates declaration consistency from execution permission, identity, and filesystem enforcement. Different role names do not prove that two independent real-world actors performed implementation and review.
For a customer asking whether protected files cannot be modified in production, a declaration PASS is only one piece of evidence. The actual runtime denial, its log, identity checks, failure recovery, and independent review still require their own tests.
A before-and-after example for proposal language
The following is illustrative editorial copy, not a quote from a real submitted proposal.
| Too broad | Bounded by the evidence | Not yet established |
|---|---|---|
| “Prevents agent permission violations” | “In the public starter, a preflight check rejects a task outside the declared src/** write scope with a specific diagnostic” | OS-level refusal of actual file writes |
| “Guarantees independent review” | “The declaration checker validates separation between a write-scoped implementer role and a read-only reviewer role” | Verified reviewer identity and independent judgement |
| “Production safety proven” | “The declared roles and task scope can be checked with a versioned starter fixture” | Reduced customer incidents, saved hours or commercial outcomes |
A sales slide need not contain every line of a test log. It should state the bounded result, and link to a fixed source revision, input JSON, check implementation, and an actual timestamped run where available. Expected output in source is not a substitute for a measured run of that revision.
Turn the remaining gap into the next proof of concept
If runtime protection is the customer's requirement, agree on the executor, forbidden operation, identity and permissions, observable OS or sandbox denial, independent review and termination evidence before the PoC begins. Record failed cases as well as passing ones.
The value of disciplined language is not merely sounding cautious. A precisely bounded claim tells both sides what evidence is still missing and what test would change the decision.
For technical details about the checker's PASS boundary, see Netsujo's existing Agent Role Contracts article. This article focuses on translating that evidence into a B2B claim and an acceptance plan. For help designing an AI implementation or proof of concept, contact Netsujo.