AI Review Should Not Become a Reassurance Ritual

Tomohiro Iida · Published August 25, 2026 · Updated August 25, 2026

Adding AI code review to a workflow makes every reviewed diff look safer. If the review scope, the strength of the review, the per-diff limit, the rerun condition, and the exit condition are undefined, review stops being an inspection and becomes a ritual for reducing anxiety. At Netsujo we ran repeated reviews on the same diff and a human re-read the same class of finding every time. This is the design we moved to: decide review scope, risk classification, review budget, rerun conditions, and exit conditions first, and keep review demand separate from execution authorization.

Incident card

What we observed

• We started running an AI review for every diff.
• Repeated reviews of the same diff returned the same class of finding.
• Findings survived as text but did not automatically apply to the next diff, so a person re-read them.
• Waiting time and cost per review accumulated.
• “It passed review” became reassurance, and we could not state what had actually been guaranteed.
• The reduction in review cost or effort is unmeasured; only the caps, scope, and exit conditions are confirmed as implementation.

The false assumption: more review is safer

Review is not free. It consumes time, money, waiting, and human attention, which are all finite. A review loop with no count limit effectively has no termination condition, and delegating termination to a feeling turns the condition into “until the anxiety is gone”. That is not a condition; the same diff can be reviewed indefinitely. The second error is reading review output as a guarantee. Review returns findings, not proof of correctness.

Root cause: review demand and execution authorization sat in the same layer

The original design called a review automatically whenever machine checks passed, which made “checks passed” equal to “execution permitted”. The current repository separates the two. The header of scripts/quality/run-quality-gate.mjs states that machine checks and risk classification calculate review demand only and never authorize a runtime execution. The same rule appears in .quality/codex-review-policy.json as classification_or_machine_check_pass_never_authorizes_runtime_execution, together with missingTelemetry set to unknown_not_invented.

Immediate fix: a per-diff review count limit

The eligibility decision lives in decideCodexEligibility inside scripts/quality/prepare-codex-review.mjs, and only three gates remain: there must be a reviewable change, machine checks must have been run and passed, and the automated review count for this diff must be below the limit. maxReviewsPerPullRequest is 3 in .quality/codex-review-policy.json. We deliberately do not stop on diff size, because a size cap creates the inversion where the largest diffs receive the least review. Instead largeDiffFiles (25 files) and largeDiffChangedLines (800 lines) are advisory: exceeding them produces a warning that splitting the change would improve finding precision.

System fix: split review into five layers

Review scope is decided by naming what is never read. The exclude list covers build output and binaries such as .next, node_modules, coverage, minified JavaScript, source maps, images, and fonts; excluded files are reported as a count only. package-lock.json is manifestOnly: its full text is not sent, but it still counts as a dependency-change signal. Only a diff-limited bundle is handed over. Letting a reviewer explore the whole repository makes the input volume independent of the diff, and the budget estimate stops working.

Risk classification: never treat “undecidable” as safe

• Test files stay at medium even when they match a high-risk pattern, because they do not change production behaviour directly.
• Files that match no rule are classified medium rather than low, so “we could not decide” is never read as “it is safe”.
• Classification selects reasoning strength; the high categories are auth, billing, db-schema, data-destruction, customer-data, env-deploy, signal-diagnosis, and infrastructure.

Review budget: cap the input and the output

Capping the output side is not only about cost. When dozens of findings come back, a person reads from the top and gets tired. The findings limit is a budget for the reader as well.

Exit conditions written in advance

• EXIT 1: there is no reviewable change.
• EXIT 2: the automated review count reached the limit of three, after which a human decides.
• EXIT 3: execution is not authorized by runtime status, budget, or owner approval.
• “Zero findings” is not an exit condition: zero means nothing was found from that viewpoint, not that the code is correct.

Move findings into tests, CI, and gates

What stops, and what stays, when the budget runs out

The common mistake is to treat a stop as a removal of capability. In .claude/agent-os/vendor/runtime-tools.json the Codex runtime has status emergency_stopped with reason_code CODEX_SPEND_EMERGENCY_STOP, while capability_revoked is false: the stop is a temporary runtime state and the reviewer and implementer capabilities remain. Blocked operations are start_session, exec, automated_review, rerun, and delegated_invocation; reading existing artifacts remains allowed. The resume contract defaults to deny. An exception holds only when owner_authorization_id, token_budget_snapshot, spend_budget_remaining, exact_scope, expected_sha, spend_cap_usd, and expires_at are all present and the expiry, SHA, and scope match. In the recorded scoped override the two budget fields are UNOBSERVABLE with invented set to false. Not being able to see the budget is different from having budget left.

Reusable checklist

• Have you listed the paths that are never sent for review?
• Are build output and binaries excluded from the review input?
• Does risk classification keep undecidable files out of the lowest tier?
• Is there a cap on automated review passes per diff?
• Have you decided what happens after the cap is reached?
• Is there a cap on output volume as well as input volume?
• Is the rerun condition fixed to “after machine checks pass”?
• Is the destination of each finding decided: test, CI, gate, or human?
• Are review executions and their input and output volumes recorded?
• Does execution authorization live in a different layer from review demand?

Evidence

Limitations

• The numbers here are Netsujo operating defaults, not universal answers.
• The effect on review cost and effort is unmeasured; this article covers only the existence of caps and exit conditions.
• Spend telemetry can be unobservable, and some fields are recorded as UNOBSERVABLE.
• Canonical decisions move: the 2026-08-11 review-everything policy was replaced by the 2026-08-21 runtime policy, so old decision documents should not be read as the current rule.

Remaining risk

• A review inside the cap is still not guaranteed to produce useful findings; count design is separate from correctness.
• Larger diffs lower finding precision, and not stopping on size leaves the split decision with a human.
• Moving findings into tests, CI, and gates costs effort, and over-mechanising low-value findings raises maintenance cost.
• An exit condition that is too strict stops the reviews you need, so the resume condition has to be written in advance.

Frequently asked questions

Is a limit of three review passes correct?

It is a Netsujo operating default, not a universal answer. When findings do not converge, splitting the diff improves precision more than adding passes. Record how often the cap hands work to a human and adjust on real data.

Can review end when there are zero findings?

Zero means nothing was found from that viewpoint. It is not proof of correctness, so termination is decided by count and state rather than by a zero result.

Can low-risk diffs skip review?

Using the classification to decide whether to review at all means nothing is detected when the classification itself is wrong. Classification selects reasoning strength, while every diff that passes machine checks stays in scope.

How is quality protected while AI review is stopped?

What stops is the execution, not the role. Machine checks keep running, and anything they cannot decide returns to human review.

When should a finding become a test?

A useful trigger is the second occurrence. The first is an individual lesson; from the second, treat it as a system defect rather than a lapse of human attention.

Previous: separating ChatGPT, Claude Code, Codex, and human judgementEpisode 00 of the incident record covers the role split before adding more agents.

Next: a second correction is an orchestration bugEpisode 02 moves repeated corrections into a Controller, Evidence Ledger, and Gate.

AI agent development and operations incident logThe series hub lists all twelve episodes and the order to read them in.