Skip to main content

The Decision to Focus ARC Validation on Two Tracks for Measuring Practical Value

Netsujo Inc. ·

The Decision to Focus ARC Validation on Two Tracks for Measuring Practical Value

On November 13, 2026, Netsujo decided to focus Agent Role Contracts (ARC) validation on two tracks. Alongside maintenance of the stable version, we would concentrate on a joint test of changing tool definitions and a second real task with human effort measurement. This article explains the evidence available at that decision, what remained unknown, and what would change the next decision.

ARC is an open-source declaration checker developed by Netsujo. It checks supplied declarations about AI-agent roles, task scopes, review relationships, and handoffs offline. The surrounding execution system handles actual permission enforcement and verification of external effects. Our product evaluation asks what practical value people gain when this declaration check is added to an existing workflow.

What we decided on November 13

By then, the evaluation had produced external reports, reproducible synthetic tests, and feedback on the design. Each informed development. Whether the extra checking justified its burden in actual work still required separate evidence.

We used the records confirmed through November 12 to make the planned November 13 decision. We limited v1.0.x to maintenance, including security and correctness fixes, and kept feature and adapter expansion paused. We retained the T4 collaboration, whose concrete test conditions were being developed, and a second real task measured by a human. The product decision recorded that day is public.

This was a choice about directing limited time toward evidence needed for the next decision. We paused expansion of other comparisons until the counterpart executed its side. The decision concerned the scope of Netsujo’s next evaluation work.

Five questions behind the decision

We separated actual use, added value beyond existing controls, human effort, effects on legitimate work, and reuse on a subsequent task. The following summary comes from the evaluation status recorded at the time.

Records used for the November 13 decision, with counters confirmed on November 12

Records used for the November 13 decision, with counters confirmed on November 12
QuestionRecord at the timeEvidence needed next
Was ARC used in external work?Two self-reported real-workflow reports; zero independently verified external adoptionsAn external execution record linking version, inputs, task, and outputs
Did it add value beyond existing safeguards?Zero natural incremental findings in the bounded internal comparisonAn equal-information comparison and a finding a human judged useful
Was the human effort justified?Time saved was unmeasuredHuman time spent on setup, rechecks, and review
Did it obstruct legitimate work?No observations sufficient to estimate a field false-refusal rateRecords of refused authorized work and assessment of those refusals
Would someone use it for another task?An executed external second real task was unprovenActual second-task use and a human decision to keep, defer, or remove it

The two external reports were self-reported; they were not counted as independently verified adoption. Zero incremental findings refers to the recorded, bounded internal comparison. Unmeasured time savings does not mean zero hours. The second-task result lacked sufficient execution evidence. This table describes the November 13 decision inputs, not fresh usage measurements as of this article’s publication or detection rates across all environments.

Laying out the questions revealed that different gaps required different work. An external usage report needs records connecting the job to the check. Human effort needs measurement by the person performing setup and review. Added value needs a comparison under equivalent conditions.

One external self-report did mention approximately three minutes of setup. That helps us understand how the reporter experienced that task. By itself, it did not complete measurement of setup, recheck, and review burden or establish time saved across the whole job.

What one actual maintenance task showed

On October 27, we used ARC during maintenance work on an existing public project’s tests. The work revised a test description and added checks for receiver-side evidence. It involved two files, and we declared each scope before making changes. The public task record counts this as one real maintenance task.

We fixed the unit of work to distinguish additional checks from use on another job. Two file declarations still served one maintenance objective. Checking before and after the change did not turn that into two real tasks.

Both legitimate scope declarations passed, and the final diff contained only the two intended files. We also checked a deliberately injected out-of-scope declaration. That was a synthetic negative control for the checker’s behavior, so we kept it separate from naturally occurring problems found during the work.

The record contained zero natural contradictions that ARC found beyond existing controls. The misleading test description had already been identified by source inspection. Existing tests and diff review handled code correctness and the changed files. We retained those observations alongside ARC’s declaration checks, each with its own role.

That result cannot establish that ARC adds no value in every workflow. What we obtained was a record that this bounded task produced no incremental discovery. Human setup and review time, effort saved, and a decision about future use still needed measurement.

The same discipline is useful when evaluating your own product: define what counts as one unit of work before interpreting a number. Files, test cases, executions, and business tasks answer different questions.

The gaps addressed by the two remaining tracks

We mapped the next work to two gaps: technical observation and human continuation of use.

The gaps addressed by the two remaining tracks
Continuing trackQuestionRequired output
T4: a tool definition changes after approvalCan declaration checks, the decision immediately before forwarding, and persisted results be correlated to the same operation?Agreed inputs and conditions, authorized execution, correlated records, and independent effect readback
A second real task measured by a humanDoes initial interest lead to continued use at a justified level of effort?An actual task, existing controls, setup and recheck effort, assessment of findings and refusals, and a continuation decision

In the November 13 record, the joint T4 test contract was not frozen, lab execution was not authorized, and the ERP experiment had not run. The outputs in this table were requirements still to be met. They did not describe a successful joint experiment with persisted business-system effects and independent readback. The status recorded at the time is part of the interpretation.

T4 observes the same operation at several boundaries

T4 concerns a tool definition that changes after approval. Even when the declared request stays the same, a changed definition—for example, one affecting quantity conversion—could alter the resulting operation. The evaluation asks how to correlate the declaration, the decision before forwarding, and the persisted effect. Our background article describes the proposed mutation and observation points.

There had already been progress on publishing ARC’s synthetic fixture and incorporating it into the counterpart’s comparison, as well as reproducing the counterpart’s reference implementation. The remaining work was to connect those bounded results to observation of actual forwarding and persisted effects.

A second task examines human effort and the decision to continue

The other track asks the person doing the work to record setup, verification, and the effort needed to understand a refusal. Overlap with existing CI or diff review is also part of the evaluation.

A first attempt may be motivated by learning the tool or exploring its potential. The effort required to use it on the next real task is a separate question. We aimed to record that experience alongside the person’s decision to keep, defer, or remove the check. Two uses alone would not establish business viability.

Specify in advance what would change the decision

The November 13 decision identified three kinds of evidence that could support selective continuation: independently verified external adoption, an executed external second task, or a natural contradiction caught by ARC beyond existing controls and judged useful by a human.

It also specified a pause condition: if T4 remained unfrozen at the next review and there was still no external second task or human measurement, pausing would be the decision. The plan recorded that day set the next review for December 11, or earlier upon a T4 freeze or an external second-task report. These are the decision conditions recorded on November 13.

Writing the conditions beforehand reduces the opportunity to select a convenient criterion after a result arrives. It also helps distinguish work that produces new decision evidence from work that repeats an existing result. The practice we took from this decision was to manage the number of evaluation activities separately from evidence capable of changing our direction.

Evaluate an actual task with us

When evaluating your own product or internal tool, keep a table of what is known about practical value alongside the list of completed activities. Setup effort, differences from existing methods, effects on legitimate work, and willingness to use the tool again make the next gap to investigate visible.

Netsujo accepts real-workflow evaluations through the ARC evaluation thread. Share the task scope, existing safeguards, setup and recheck burden, observed result, and whether you would use it again. We want useful findings, extra effort, and cases where existing controls were sufficient recorded in the same form, so they can inform the next product decision.