What 24 PASS Results Show: Receiver Provenance in an AAE Cross-run
Netsujo Inc. ·
“24 pass, 0 fail, 0 skipped” sounds reassuring. We saw those exact figures when checking a cross-run report for an external agent-authorization profile. But the number does not reveal who built the receiver, what actually executed that day, or what was independently verified. A public discussion about those missing distinctions led to a concrete improvement in the other project's reporting format, and to a follow-up task in Agent Role Contracts (ARC).
Two records with different purposes
The A2A extension discussion #2391 concerns AAE, an Agent Authorization Envelope presented across agent-to-agent calls. An AAE receiver verifies the mandate and the request it receives, and records an authorization result with a digest of the request. ARC checks supplied declarations and task/action bindings before execution. A PASS from one component cannot be translated directly into a result from the other.
The author of the AAE proposal invited the ARC maintainer to run the pinned cases, as recorded in ARC #105. The invitation was useful because it forced us to distinguish an offline declaration checker from a receiver that applies an authorization profile and from the eventual skill or external effect.
What actually happened on November 6
On November 6, 2026, Netsujo invoked the project's official python -m aae_a2a.crossrun check at pinned commit 08f41146… on an existing offline receiver report. It returned valid (24 pass, 0 fail, 0 skipped), exit code 0. The report covered 24 cases and 26 presentations. The November 6 action checked recorded results against the published cases. It did not rerun the receiver or connect to an external A2A implementation. The original ARC #105 receipt records the fixed evaluation time, audience, synthetic scope and report hash.
The locally authored receiver used knowledge of reference-source behavior. We therefore did not call it a clean-room implementation. Formal independent code QC, live remote interoperability, and official third-party cross-run submission remained unperformed or unsubmitted. Those labels are as informative as the PASS count, because they identify the next evidence we actually need.
An external review changed the reporting contract
The proposal's author observed a gap in the cross-run results table: readers could see “24 pass” but could not tell whether the receiver was written solely from profile text or informed by the reference source. We responded with the precise provenance of our own run, rather than presenting it as independent interoperability.
The proposal's author subsequently added a required receiver_provenance field to results and the public cross-run table. The recognized values are clean_room, reference_informed, and unknown; the checker now refuses a result file without the field. A separate third-party run was listed as the first third-party run, while Netsujo's unsubmitted reference-informed report stayed out of the table at our request.
This is a tangible result of public technical dialogue: the report makes implementation provenance visible beside outcome counts. It is an improvement to the AAE-side reporting contract, not a demonstrated ARC performance advantage or a new customer adoption.
The reporting shortcut we want to prevent in ARC
We opened ARC #119 to make detailed evaluation summaries self-contained. A copied table should show implementation provenance, offline or live scope, the executed stage, independent review status, links to the original records and any NOT_RUN items. On November 8 this remained an open task with no implementation started, not a merged feature.
An evaluator can then answer a useful question: does “PASS” describe a declaration check, a report checker, the receiver, or an externally observed operation? If it is only a report check, the next step may be independent receiver review or a live interoperability exercise. If it describes a claimed business effect, the operator needs durable receiver-side evidence and a link to the original authorization.
Five fields for OSS evaluation reports
| Field | What the reader should know |
|---|---|
| Implementation provenance | Reference implementation, reference-informed code, independently authored code, or unknown |
| Executed stage | Declaration checker, report checker, receiver/workflow, or external effect |
| Conditions | Pinned code revision, data, date, environment, offline or live |
| Independence | Who independently reran or reviewed which stage; unperformed work marked NOT_RUN |
| Continued value | A second distinct real task, human setup/review time, and overlap with existing controls |
The last item remains an open question for ARC itself. Issue #47 targets evaluations on ten real workflows, but we do not count synthetic demos or our own reproductions as external adoption. We still need evidence of human effort, incremental benefit over existing controls and a second genuinely different task.
Trustworthy evaluation depends less on impressive totals than on whether another engineer can determine exactly what those totals establish and what still needs testing. For ARC's own checker rules, see what an ARC PASS does and does not guarantee. In ARC #37, we welcome first-use feedback, failures and examples where an existing safeguard proved sufficient.
Primary sources: ARC #105 execution receipt, A2A #2391 reporting revision, ARC #119 follow-up, ARC #47 adoption evaluation. Netsujo develops ARC. AI-assisted editing was used; the source and scope of each claim are linked above. This article does not show independent third-party verification or commercial adoption of ARC.