What Makes an AI System Worth Using a Month After Deployment?
Netsujo Inc. ·
The proof of concept produced the expected answers. The system connected successfully. Yet the assessment can change when the people responsible for the work begin using it every day.
Editing the answers takes too long. The system cannot handle exceptions during busy periods. Nobody knows whom to contact when something goes wrong. If these conditions remain unresolved, returning to the previous way of working is understandable.
In AI deployment, Netsujo focuses on turning technical validation into conditions that let people keep using the system. To do that, we design the decision to move into production separately from the decision about its effect on the work after deployment.
Take One Real Case Through the Entire Workflow
Consider an illustrative workflow in which AI drafts replies to inquiries.
Whether the reply reads naturally is one thing to check. But the person handling an actual inquiry also has to read it, find the necessary information, edit the draft, decide whether to send it, and record the interaction. Even if generating the draft is fast, the complete case—including checking and editing—may not become easier.
Before production, walk through this entire sequence with the people who will actually use it. Alongside ordinary inquiries, test cases with missing information, questions the system cannot answer, and incorrect references. Check that anything requiring human attention reaches the responsible person in a form they can understand.
The people who know the receiving team's work and the engineers responsible for integration and evaluation need to examine the results using the same unit of work.
Separate Implementation Acceptance from Business Impact
When the required functions work and the agreed evaluation conditions are met, there is a basis for accepting the implementation. What comes next is a different observation: how much the system is used in ordinary work, and what changes as a result.
OpenAI's Forward Deployed Engineer job description likewise identifies production use, measurable business impact, and evaluation-based product feedback as success measures. When considering customer deployments, Netsujo also distinguishes use from impact.
For the inquiry-drafting example, observations can be separated as follows:
- Use: how many of the inquiries in scope actually used AI, and the reasons it was not used for the others.
- Quality: what needed correction, errors that must not be missed, and exceptions returned to a person.
- Workload: the time for the complete case, including checks, edits, and exception handling.
- Cost: model and service charges, along with the work required for maintenance and checking.
Greater use can still increase the burden of human review. Cases where AI was not used also have reasons behind them. If many inquiries are a poor fit, the evidence may support narrowing the scope.
A Speed Improvement We Chose Not to Adopt
A faster result does not always justify moving a candidate into production. Netsujo's published account of a 24-hour period of actual operation on September 8–9, 2026 includes an experiment to speed up the code checkout process before automated checks. We did not adopt it.
| Stage | What the record shows |
|---|---|
| Change tested | A faster code checkout process |
| Observed duration | From approximately 8 seconds to approximately 7 seconds |
| Independent review | A high-severity finding that the change could compromise the correctness of change detection under certain conditions |
| Adoption decision | The risk outweighed an improvement of approximately 1 second, so the candidate was rejected |
We do not accept a candidate that compromises required correctness simply because one metric—speed—improves. This record illustrates a decision about adopting a technical candidate in our own operation. Whether a customer's team continues to use a system, or experiences less work, must be measured separately in that team's subsequent workflow.
| Observation | What is examined | Basis for a decision |
|---|---|---|
| Technical acceptance | The specific version, functions, connections, and evaluation conditions | Tests and independent review support the adoption decision |
| Use and impact | Use in ordinary work and the burden of a complete case | Observe use counts, quality, time, and cost in that workflow |
In deployment support, we decide in advance what improvement would justify adoption and what unresolved problem would require us to stop. That puts the desired change and the quality that must be preserved into the same decision.
Close the Gap Between “It Runs” and “We Can Adopt It”
One lesson we can carry from our own experience into practical work is to avoid collapsing adoption into a single pass indicator. A passing test has a scope: what was checked, on which candidate, and in which environment. When a revision changes the candidate, we need to reconsider whether the previous pass is still sufficient for the decision.
In the inquiry-drafting example, producing a natural answer to an ordinary question and being manageable for the operator during a busy period are separate things to verify. Even when a connection test passes, problems can remain: consulting outdated material, confusing conditions that differ by customer, or allowing the operator to miss content that should not be sent.
Before moving into production, bring at least the following questions together around one complete case. Who checks that the answer is correct? Is the supporting information needed for that check visible? What reaches the operator when a question falls outside the system's scope? Where can a process resume after stopping midway? If these points remain unclear, agreeing on operating conditions may take priority over adding features.
Not every implementation acceptance condition can necessarily be finalized at the start of development. Share exceptions discovered through testing with the customer. If necessary, narrow the scope and return to a candidate that can be evaluated. Recording the reason for that decision also makes clear what additional checks will be needed when the scope expands later.
When Automation Still Needed a Human Instruction to Continue
The same operational record also documents a problem in which continuation stopped because of how waiting states were handled. A person had to instruct the process to resume, even though a mechanism for deciding the next step existed.
The lesson is that a resumption procedure alone does not complete the operating workflow. Which states should wait? What must become available before work proceeds? Which states require a human decision? Unless those distinctions connect to the running process, users still have to intervene even when the mechanism exists. This historical record alone does not establish that the entire current operation has become autonomous.
To examine the same issue in a customer's workflow, test what happens if a reference source becomes unavailable while an inquiry is being processed. Does the operator receive a clear indication of failure? Could a retry send the same reply twice? Does the system avoid automatically resending a case when it is uncertain whether the reply was already sent? Checking what happens after a stop reveals burdens that a successful demonstration can leave hidden.
Returning work to a person can itself be a designed part of the operation. The person receiving it needs to know which case is involved, what has already been completed, and what remains uncertain, so they can make the next decision.
Use the Post-deployment Comparison to Decide What Comes Next
One approach is to set an initial observation period of 30 days to examine continued use. This is a planning example for work performed every day, where sufficient cases can be collected. For a monthly task or a workflow with few cases, choose a different period that will produce the examples needed for a decision.
Before comparing results, record the number of cases in scope, the time required, and quality under the existing process. After deployment, measure using the same unit of work as far as possible. Also record differences in inquiry difficulty, the person handling the work, or how busy the team was.
A shorter duration alone may not isolate AI's effect. If only a few cases have been observed, there is no need to rush to a conclusion. Use the records to decide whether to continue, expand, narrow, or stop the deployment.
The purpose is to collect evidence sufficient to decide whether this way of working should continue. Adhering to a 30-day duration is secondary to that purpose.
Divide the 30 Days into Stages That Produce Useful Records
An observation period does not necessarily run under identical conditions from the first day of deployment. People need time to learn the controls during the first few days. If settings or scope change partway through, combining the periods before and after that change into a single result makes it harder to see what helped.
For a workflow with a reasonably consistent number of cases every day, the following is one possible plan. This is an unexecuted planning example. The actual observation period and number of cases should fit the work.
| Example period | What to do | Records used for the decision |
|---|---|---|
| Before deployment | Measure an existing case from beginning to end | Case count, difficulty, operator, time, and errors |
| Days 1–7 | Test with a limited scope | Reasons the system could not be used, edits required, and points of confusion in operating it |
| Days 8–21 | Use it in ordinary work under the agreed conditions | Cases in scope, actual uses, editing burden, exceptions, and usage charges |
| Days 22–30 | Compare while accounting for differences in conditions | The decision to continue, expand, narrow, or stop, and its rationale |
For example, recording only how many drafts were adopted hides inquiries that were excluded before anyone tried the system. Distinguishing the total inquiries an operator handled, the subset considered in scope, and the subset where AI was actually used helps show whether the scope was realistic.
Time records should cover more than waiting for generation. Include checking the supporting information, editing, and recording the interaction. If someone abandons AI for a case, record the time spent returning to manual work as well. Averaging only the cases where AI was used biases the result toward the successful cases.
The same applies to cost. Alongside usage charges, examine where the necessary checking and maintenance work occurs. If the sales team's time decreases while another person's checking work increases, that additional work needs to be included in the decision.
Investigate the Reasons Behind Low Use
Attributing non-use only to the operator's motivation can bring improvement to a halt. Were there few inquiries in scope? Was the necessary material unavailable? Was leaving the existing screen too cumbersome? The observation “it is not being used” can point to different next steps.
Trace a case where the system was used and a case where it was not, and identify the point at which the decisions diverged. If input is too burdensome, revisit the user flow. If answers lack supporting information, revisit retrieval and references. If most cases are out of scope, reconsider the choice of workflow. Decide what development is necessary after identifying what needs to change.
More uses alone also do not establish success. Decide whether to continue by combining use with acceptable levels of checking work and errors, and clarity about the work that remains with the team. Measurement should make the next action easier to choose, rather than merely supply numbers to justify the deployment.
Make the Operation Work When the Responsible Person Changes
Production use requires more than settings and instructions. It also needs a contact for problems and someone who owns the decisions. Decide who does what when reference information stops being updated, answer quality declines, or the service stops.
During handover, going beyond delivering a manual can reveal gaps. Have the next operator process an actual case and check whether they can return an exception to the appropriate person. That also clarifies which situations the customer can handle and where external support is needed.
Netsujo's support connects the division of responsibilities among the customer's operators, existing development team, and specialist engineers with PoC evaluation, production acceptance conditions, and the observations needed after deployment. We agree on the scope of implementation and ongoing operation according to the workflow and the team involved.
If you already have a PoC, show us the working screen and the steps an operator follows to complete one case. We can identify what still needs to be decided for people to keep using it.
Explore support from PoC to production / Discuss the conditions for production use