Skip to main content

How We Turned External Developers’ Feedback into Executable Failure Tests

Netsujo Inc. ·

How We Turned External Developers’ Feedback into Executable Failure Tests

At Netsujo, we have been turning external developers’ questions about AI-agent verification into small failure tests that others can run. In November 2026, we published a four-case example about the producer of review evidence and a five-case example about binding a signed receipt to a candidate commit. This article explains how we moved from a concern to specific inputs, changes, and observations that could answer it.

Agent Role Contracts (ARC) is an open-source declaration checker developed by Netsujo. It checks supplied declarations about agent roles, task scopes, review relationships, and handoffs offline. The surrounding execution system remains responsible for authenticating actors, managing private keys, and enforcing actual write restrictions. The signature test discussed here is a companion experiment for examining questions at that surrounding system’s boundary.

Turn feedback into a question another person can test

When an AI agent returns a review result, what evidence lets us proceed? A report may name a reviewer, identify code, state a verdict, and reference a policy. Even with those fields present, we still need to ask who could have produced the record.

On October 27, Netsujo mapped seven pieces of public feedback to design decisions and incorporated them into four design documents. The topics included authority provenance, the subject to which evidence belongs, and retries after an uncertain result. This was a documented design update; it did not ship a new runtime enforcement feature. Our earlier account of the public discussions separates those scopes.

In November, the discussion moved toward the producer of the review evidence itself. Suppose a JSON record contains a name representing an independent reviewer. If the implementer can also write that JSON, we need to examine whether matching the name is sufficient.

We wanted to express the concern as a relationship between an input and a decision. “We are worried about independence” becomes a testable question when we ask: “If an implementer submits a record claiming to be the reviewer, what evidence makes the verifier reject it?” Another person can then run the same conditions.

To make that conversion, we specified a valid control alongside the change under test. Defining exactly what changes also makes the question an implementation has to answer narrower.

Establish a valid control, then separate the reasons for rejection

The example published on November 11 used three kinds of records: declaration, evaluation, and acceptance. It also used a fixed candidate identity. We defined four cases:

  • All required records are present, with matching subject and authorized provenance: expect PASS.
  • The evaluation record is removed: expect PENDING.
  • The evaluation refers to a different candidate: expect DENY.
  • The implementer emits an evaluation while claiming to be the independent reviewer: expect DENY.

The order matters. If the evaluation record is already missing, rejecting the final case would not tell us whether the verifier distinguished its producer or simply found incomplete evidence. Establishing the valid control and checking missing records and mismatched subjects separately made the producer condition easier to examine. The published four-case example describes the inputs and verification approach.

In that example, a mock host journal supplies the trusted provenance. It does not establish real authentication or isolation. The question was whether a verifier could distinguish a producer’s own claim from a check based on a separate trusted input.

The same control design can help beyond code review. Separating a change to the subject from a change to the presence of evidence makes the reason for rejection explainable. An example that breaks several conditions at once makes it harder to determine which required check actually ran.

Bind a signature to the subject being checked now

The next example addressed a valid signed receipt that remains available after the code has advanced. A review covers candidate A, but the gate is now considering candidate B. We made explicit what the verifier must compare in addition to checking the signature.

The November 17 example uses temporary Ed25519 keys to sign a receipt containing the candidate identity, policy, verdict, and other context. The verifier receives the trusted reviewer’s public key and the candidate and policy currently being checked. We vary the candidate, policy, signer, or presence of the receipt and inspect the resulting decision.

Local fixture executed on November 17, 2026

A fixture is a small executable example with repeatable input conditions. We ran this one using Node.js v22.16.0, and all five expected decisions matched. The code and execution record are public.

Local fixture executed on November 17, 2026
InputObserved decisionWhat this establishes
Candidate A, policy v1, and a PASS signed by the trusted keyEVIDENCE_MATCHED_ONLYThe supplied comparison conditions matched
A’s receipt is presented for candidate BDENY_STALE_CANDIDATEA candidate mismatch was rejected
A v1 receipt is checked under policy v2DENY_STALE_POLICYA policy mismatch was rejected
Another key signs the same fieldsDENY_UNTRUSTED_SIGNERA signature that does not match the pinned trusted key was rejected
No receiptPENDING_MISSING_RECEIPTMissing evidence remained unresolved

This is an educational fixture that generates two temporary key pairs inside one process. Its scope is signature verification against a pinned public key and comparison with the supplied candidate and policy. It does not test operational private-key custody, whether a review actually occurred, operational independence between implementer and reviewer, or permission to merge. It also adds no signature-verification or key-management feature to ARC itself.

We called the matching decision EVIDENCE_MATCHED_ONLY to make that scope visible. The system deciding whether to permit the next action still needs to establish such things as complete review and test coverage and the applicability of its current policy.

Six fields for turning a concern into a test

Looking back at both examples, we find these six fields useful when converting external feedback into a test. This is a proposed working method based on the experience; the table is not the result of a new experiment.

Six fields for turning a concern into a test
FieldReview-evidence example
Claim to examineDoes this PASS belong to the current candidate and an authorized reviewer?
Valid controlSupply a receipt with matching candidate, policy, and signer
One change at a timeChange only the candidate, policy, signing key, or presence of the receipt
ObservationThe verifier’s decision and its candidate, policy, and signer comparisons
Expected decisionDistinguish a match, stale candidate, stale policy, untrusted signer, and missing evidence
Remaining questionWho can use the private key, did the review occur, and is the accepting gate’s policy effective?

The first field should identify one claim. A broad question such as whether an agent is safe leaves the conditions for answering it undefined. Narrowing it to rejection of candidate A’s receipt when evaluating candidate B gives us an input and an observable result to compare.

We also separate observations from expected decisions. An expectation is specified during test design. An observation comes from execution. Leaving an unexecuted item marked as such lets the next discussion focus on what is missing: an implementation, an environment, inputs, or authorization.

The final field describes the boundary of the test. Making that boundary explicit helps keep a successful small experiment from being interpreted as a broader operational conclusion. It also helps a reader decide where the check belongs in their own system.

Preserve what ran and the questions that remain

The public discussion has raised further questions: which policy the accepting gate considered effective, how revoked keys should be handled, who may write policy history, and what records should remain when an earlier judgment is corrected or retracted. These are design proposals in the public discussion; they are not additional executed cases in the five-case fixture.

We view the progression from the four-case example to the five-case example as a refinement of the subject being observed. The first examined required records and their provenance. The second compared a signed record with the current candidate and policy. Preserving each example’s assumptions lets another developer reproduce its conditions and propose a different failure case.

When publishing these experiments, we aim to keep the inputs and outputs together. A reader who can see the inputs, execution conditions, observed decisions, and unanswered questions can discuss precisely which part they reproduced.

Bring a concrete concern into joint evaluation

Choose one concern from an AI-agent workflow and fill in the six fields. Include what existing diff review or CI already detects and where uncertainty remains. That makes the role of an additional check clearer.

At Netsujo, the ARC real-workflow evaluation thread accepts descriptions of actual task scope, existing safeguards, observed results, and remaining questions. Setup and recheck effort, including cases where existing safeguards were sufficient, are useful evidence about practical value. An idea that has not yet been executed can be shared with that status intact.