What Agent Scope Checks and Code Scanners Catch: Four ARC × ShipRules-free Cases

Netsujo Inc. · Published: November 3, 2026

When an AI agent edits a codebase, two questions have to be answered: *Was this task declared within its allowed scope?* and *Is the code appropriate for the target environment?* One tool's passing result may answer only one question.

Agent Role Contracts (ARC) checks declared roles and requested task scopes before execution. The public ShipRules-free scanner looks for listed Godot 3 syntax in Godot 4 .gd files. If they address different problems, it should be possible to build cases where only one of them returns FAIL. In this article, "catch" means a check returns FAIL; it does not mean blocking a write at runtime.

Netsujo compared the two checks on four public, pinned cases on October 26, 2026. This was a maintainer-executed synthetic comparison. It was not a test of safety improvements in a customer's live environment.

Four cases under the same conditions

The experiment pinned ARC v1.0.0 at 9f6771056c47de971202360952af6d41941f7625 and ShipRules-free at 8b8302e9551fb8784d7cbe7fa61eeecf127bf247. The reproducible fixtures are public.

One clean demonstration would merely show two passing results. It would not answer the adoption question: what does each tool miss if it is used alone? The comparison therefore starts from a valid Godot 4 file with an in-scope declaration and changes one condition at a time.

Each result is listed as ARC check / syntax scanner.

Four cases under the same conditions
Case	Result	What it means
1. In scope, compatible syntax	PASS / PASS	Both accept the dimension they check.
2. Out-of-scope request, clean syntax	FAIL / PASS	A syntax scanner does not detect a declared scope violation.
3. In scope, old Godot syntax	PASS / FAIL	A scope check does not detect syntax incompatibility.
4. Scope and syntax repaired	PASS / PASS	Each side checks a different problem.

In case 2, ARC reports TASK_WRITE_SCOPE_OUTSIDE_AUTHORITY. Only the task's declared write target changed. This is not evidence that an unauthorized filesystem write happened and ARC stopped it; it records that the requested scope does not fit the declaration. The scanner column for this case reuses the case 1 scan result (No Godot 3 syntax found.); the scanner was not run separately for case 2.

In case 3, ARC passes because the target is in scope. The scanner finds export var speed = 120 on line 3 of player.gd and yield(get_tree().create_timer(0.1), "timeout") on line 6, and returns FAIL with exit code 1. The question is not which FAIL is right, but which input and contract each tool checked.

An ARC PASS grants no execution authority. The result JSON records execution_authorized: false and runtime_enforcement: false for every case.

Results, and the questions they cannot answer

The recorded run used Linux/x64 and Node.js v24.22.1. All four expected outcomes matched and the runner exited 0. The source PR #76 was merged after four hosted CI jobs succeeded and a maintainer-dispatched, read-only automated source review (an AI reviewer) passed.

Automated execution took about 421.8 milliseconds, and individual scanner runs about 29–33 milliseconds. Both figures are observations on this machine and fixture. They are not human setup time, and they are not a general performance benchmark.

These results apply to the pinned public fixtures and the maintainer's own runs. They say nothing about ShipRules' paid hooks or unpublished CI files. They do not establish ARC runtime enforcement, actor authentication, production execution authority or a customer's successful adoption. Issue #75 records that the ShipRules counterpart found the four-case split useful and intends to review the cases. As of November 2, 2026, an independent counterpart run and keep/remove decision had not been completed. Completed external hands-on evaluations added: zero.

So this comparison cannot answer:

Whether ARC would block a real agent's write to another file. Actual permission enforcement needs runtime IAM or a sandbox; ARC's declaration check does not replace them.
How many failures it finds that existing access controls or CI would miss.
Whether it saves reviewer time and pays for its setup cost.

A count of PASS results cannot estimate operating cost-effectiveness.

List the checks you already run before adopting another

If your team already has CI and permission controls, map which check runs when, and who receives its result, before adding ARC. For the two checks compared here:

Before execution: Are declared task, role and requested scope consistent?
Code checks: Does a changed file violate a specific syntax rule?
Execution boundary: Can the authorized actor actually perform the operation? IAM, sandboxes and runtimes own this.
After execution: Did only that scope change, and did the required tests and independent review pass? This needs actual diffs and CI evidence.

Diagram of four check layers: ARC declared scope and ShipRules-free syntax scanning were tested; runtime IAM or sandbox and post-run evidence were not — Figure: the layers this comparison tested, and those it did not.

Assuming one check covers everything hides the failures outside it. Conversely, if an existing control already catches the same contradiction well, adding ARC may add little.

To gather evidence for a decision, pick one real task and check whether your existing controls alone catch the same problem. Then record setup effort, false refusals, repeated reviews, and whether the same declaration could be reused for a second real task. Deciding not to adopt ARC, when no reason to keep it appears, is a valid outcome.

Reproduce it in your own environment

The experiment README pins both repository commits, because an update could silently change what the same command tests. It also records the public ShipRules scanner SHA-256, 58b650402c321d69cdeddc3b71abcb55ad882585a9bdad00929b1f545c61b8d7. The runner rejects a wrong commit, modified tracked files or a scanner-hash mismatch.

Use separate checkouts rather than a production repository:

git clone https://github.com/suirindo/agent-role-contracts.git arc-kit
git clone --branch v1.0.0 --depth 1 https://github.com/suirindo/agent-role-contracts.git arc-v1
git clone https://github.com/otctrading101/shiprules-free.git shiprules-free
git -C shiprules-free checkout 8b8302e9551fb8784d7cbe7fa61eeecf127bf247
node arc-kit/experiments/external-evaluation/shiprules-free/run-comparison.mjs \
  --arc-v1 arc-v1 --shiprules shiprules-free

The prerequisites are Git, Bash and Node.js 22.5 or later. The recorded observation was made by the maintainer on Linux/x64 and does not guarantee the same result in your environment.

Whether the comparison reproduces or not, please record it in ARC Issue #75, with your environment, the pinned commits, differences from your existing CI, setup effort and any false refusals.

Related: What ARC PASS guarantees and what it does not covers the single checker. This article is limited to comparing a different kind of check under the same conditions.