Designing AI Agent Operations
Tomohiro Iida ·
Netsujo Inc. is a small startup based in Kyoto. We run the company by combining AI agents (Claude Code) and automation with quality gates that enforce facts through mechanisms rather than relying on individual care. This article publishes the details of that operating system as we actually implement it: it is not a showcase of flashy AI use, but an account of how we run day-to-day work. Three principles anchor it: humans hold judgment, AI handles implementation and verification, and errors are stopped by mechanisms rather than caught by attentiveness alone. We describe the three quality layers, the audits registered on weekly schedules and how their execution is verified, and the decisions we deliberately keep in human hands: positioning, publication approval, first-hand verification of facts, and ethical boundaries. We also share, honestly, what has worked and what AI still cannot do.
This article records Netsujo's operating system as of June 2026. Since then, we have separated state, authority, evidence, exact-head verification, independent QC, and recovery into Agent OS, Controller, and Orchestrator responsibilities. We keep this article as a primary record of that transition rather than silently rewriting the earlier operating model as if it were today's standard.
Key takeaways
- Netsujo is a small team where humans hold judgment while AI agents (Claude Code) handle implementation and verification across multiple businesses and site operations.
- Quality is enforced through mechanisms, not carefulness: third-party AI fact-checking (an operating rule), CI that automatically blocks factual errors before anything goes live, and source-required proof verification.
- Several SEO and site audits are registered in GitHub Actions for Monday runs. Schedule registration, successful execution, and Discord notification are separate states, so we check run history before claiming that an audit ran.
- Positioning, publication decisions, first-hand fact verification, and ethical boundaries remain in human hands.
Why we built an operating system
Netsujo was founded in June 2023 as a small, Kyoto-based startup working in Web3 and AI. As a small team took on more businesses and more sites to run, the amount that needed checking grew faster than headcount. Relying on human attentiveness alone was not sustainable: we experienced real errors, including a misstated job title and a mixed-up date. So we changed our approach. Rather than asking people to try harder to be careful, we built errors-cannot-happen-or-get-stopped mechanisms into daily operations itself. We call the resulting collection of mechanisms our operating system internally: AI agents handle implementation and verification, automated gates enforce quality, and humans hold judgment and responsibility.
| Aspect | Traditional operations | Operating system |
|---|---|---|
| Quality assurance | Staff try to "be careful" | Built into CI, verification scripts, and third-party AI |
| Catching factual errors | Left until someone notices in review | CI fails automatically on a banned-pattern match |
| SEO/site monitoring | Addressed after a problem occurs | Weekly audit schedules are registered; execution and notification are verified from run records |
| Work records | Scattered across individual memory and chat | Kept in markdown/ledgers, verifiable later |
| Implementation speed | Proportional to headcount | Repetitive, routine work delegated to AI and automation |
Three operating principles
- Humans decide, AI implements: positioning, publication approval, and ethical judgment stay with people; research, code changes, analysis, and routine work go to AI agents.
- Facts are enforced by mechanism, not vigilance: relying on human attention always lets some errors through, so we require sources for claimed results, run automated checks for banned phrasing, and add third-party AI fact-checking.
- Everything gets recorded: decisions, work results, and audit outcomes are kept in markdown or Discord so they can be verified later, avoiding reliance on what one person happens to remember.
Three layers of quality gates
We do not publish what AI writes as is. Every piece of work is checked by three mechanisms of different character: one is CI, automatically enforced before anything reaches production; the other two are an operating rule and a verification script. Layering mechanisms with different levels of enforcement makes errors easier to catch.
- Third-party fact-checking via Codex (operating rule)
- We have work read by an AI other than the one that produced it, to surface errors, exaggeration, and unverifiable claims stated as fact. When it runs depends on the type of artefact. A diff inside the repository goes through one review after the machine checks — lint, types, tests, and build — have passed. An artefact whose truth the machine checks cannot judge, such as research, a proposal, or the body of an externally published article, goes through one review before it is reported as complete. The reviewer is granted read access only: it does not implement, auto-fix, commit, or push.
- CI that blocks factual errors (automated)
- Errors that have actually happened before, such as a mistaken title, a fabricated credential, an old service name, or a wrong date, are registered as banned patterns in CI. If matching text appears, the CI check fails automatically before it reaches production. Because CI fails automatically, this layer cannot be skipped by human judgement.
- Proof verification requiring a source (script)
- Claimed achievements, such as speaking engagements, awards, or media coverage, must be registered with a verifiable source URL and a verification date; if either is missing, the verification script fails with an error. A human still checks the source content itself; the script only guarantees that a source and a verification date are always attached.
As an example: if someone's title were written incorrectly, under the old process the mistake could sit until a reviewer happened to notice it. Under this system it matches a banned pattern in the fact-error-blocking CI, and the check fails automatically before release. Turning a past mistake into a registered pattern lets the same error be caught automatically the next time.
Scheduled weekly audits and execution checks
Quality does not hold up under a one-time check. Several audits are registered to run every Monday morning via GitHub Actions, and the workflows include a step configured to notify their results. The point is to avoid checking only when someone asks whether there is a problem. Schedule registration, start, completion, result, and notification are separate states: we do not report that something “runs every week” or that a notification was delivered without checking the corresponding execution and notification records.
- Full GSC audit: combines Search Console and GA4 to check sitemaps, indexing status, CTR underperformance, and canonical mismatches
- Core Web Vitals audit: measures mobile and desktop load speed via PageSpeed Insights
- Funnel effectiveness: analyzes traffic and results for local-government and Kansai Web3 funnels using GA4 and GSC
- IndexNow ping: automatically notifies Bing IndexNow of key URLs to encourage crawling
- llms.txt consistency check: verifies the URL consistency of the llms.txt file provided for AI search
- Originality audit: scores article content for originality and flags thin content
In addition to these, a fact-error-blocking CI check and a check for unresolved merge conflicts both run automatically before code reaches production. Serious issues found are fixed on the spot; minor ones are logged and addressed in the next cycle. We try to put as much of the path from detection to fix onto a system as possible.
Before saying the audits are running, separate five states
| State | What it confirms | What it alone does not show |
|---|---|---|
| Registered | A schedule definition exists in the repository | That it started |
| Started | There is a record that execution began at the intended time | That processing ran to the end |
| Processing completed | There is a record that it finished without failing partway | That the judgement inside was correct |
| Result | The judgement and the scope it covered can be read | That it reached the person responsible |
| Notified | There is a record that the result was sent to its recipient | That the recipient acted on it |
Check them in that order: do not report a lower row when the row above it has not been established. Not writing “this runs every week” without looking at the execution record is part of using this table.
Separate the read-only role from the role that changes things
Once there are several agents, the next problem is not capability but state management. If it is not written down who may do what right now, and how something that has stopped gets resumed, then checking progress itself turns into human work. For each role we write down the permissions, what remains possible while it is stopped, and the conditions for resuming. What follows is our design example.
| State | What is possible | What is not possible | Conditions for resuming |
|---|---|---|---|
| Running | Change files inside the assigned unit of work, and commit and push to a branch | Extend the change beyond the unit of work; decide on a production release | — |
| Read-only | Read the diff it was given and the existing records, and return findings | File changes, automatic fixes, commits, pushes | — |
| Stopped | Read records and artifacts that already exist | Starting a session, execution, automatic review, re-runs, delegated invocation | Denied by default. It succeeds only when the authorization identifier, the exact scope, the target commit, the caps, and the expiry are all present at once |
- Do not let the role that implemented something also stand as the independent review of it. An implementer reading their own change, tracing the cause, and writing tests is necessary work and is not what gets separated. What gets separated is counting that check as an independent review: if the premise of the implementation was wrong, the implementer’s own check passes with the same premise intact.
- Do not treat stopping as revoking capability. Stopping is a temporary state; the role keeps its standing as a reviewer. Managing state and standing in the same field makes it impossible to restore afterwards.
- Make the default for resuming a refusal. In our own setup, resuming succeeds only when the authorization identifier, the exact scope, the target commit, the caps, and the expiry are all present at once, and if even one is missing it does not resume. Do not fill in values you cannot observe: a budget you cannot see is not the same as a budget that is sufficient. That particular set of five is our setting for a configuration that can hit an external service’s usage cap, not a prerequisite for every AI operation.
What this aims at is not reducing human judgement but reducing human relaying. When state and resume conditions are readable, nobody has to go around asking why something is stopped. We have not measured how much relaying it removes, so we do not report that as a result.
What is written here is a design example of the state categories and resume conditions we use. Which constraints the runtime actually enforces, and how much is an operating convention, differs by role. We record what was designed separately from what a machine can actually refuse. The right divisions change with the size of the organisation, the data involved, and the cost of stopping, so this is not written to be adopted as-is.
What stays with humans
The more we delegate to AI, the clearer it becomes what humans must keep holding. The following decisions are never handed to a system; a person carries the responsibility.
- Deciding positioning and messaging
- The final call on whether something is fit to publish
- First-hand verification of facts, including proper nouns and track record
- Where the ethical line sits
- Deciding what to build next, and what to stop doing
This operating system does not aim to reduce the number of people involved. It aims to free people to spend their time on the highest-value work there is: judgment. The more we hand implementation and verification load to AI and to the system, the more time we have to think about what to build, who it reaches, and how to stay honest.
What we learned from delegating to AI
- Strength: repetitive work, such as bulk fixes, audits, and applications, can be delegated to AI and automation. We have not measured the reduction, so we do not report it as a percentage saved.
- Strength: quality checks built into the system are not affected by anyone's workload or condition on a given day.
- Strength: because records persist, past decisions can be verified and reused later.
- Limit: AI still makes mistakes through exaggeration or unfounded assumptions; third-party checks and sourced verification remain essential.
- Limit: decisions about what to build and whether to publish cannot be handed to a system; they still require human judgment.
- Limit: building the system itself carries real upfront cost. Starting small and growing it is the realistic path.
Summary
Running a company with AI agents does not mean replacing people with a system. It means holding firmly to a division of labor: humans decide, AI implements and verifies, and mechanisms stop errors. Even a small startup can reach stable operations with fewer oversights once quality is built into the system this way. Netsujo tests this operating system daily on its own site: building the tools we use ourselves, and enforcing our own integrity through mechanisms. We believe that discipline feeds directly into the quality of the development and analysis work we provide clients.