A problem for people who use AI
How to check and ship development handed to AI
Developers who proceed with implementation using AI, operators using several AIs, and makers of verification tools
Fixes in public PRs; overall effect unconfirmed
Posts and updates from July to October, and operations records whose publication date is unknown. Publication, update and experience times are shown separately.
Editorial check: 2026-10-23
What people are paying attention to
While implementation from assignments to AI and from requests advances, developers who use it heavily are also building mechanisms for review and for checking actual behavior.
What is hard
Even after implementation, checking the original requirements and behavior on failure remains. There is also a request that the edited scope is not linked to related tests and that checking is slow.
Workarounds being tried
They combine review by a separate model, runtime probes with code analysis, and a person's preview check. A proposal to link to related tests with hooks after edits and at stop is also presented.
- Source: Even when several AIs share the work, the judgment of completion is made separately
- Source: Run AI-written code and check its behavior down to failure cases
- Source: The AI implements a request, and a person checks it in a preview
- Source: Connecting AI edits to the tests that relate to the change
What the sources confirm
In DoltLite's public PR, we confirmed fixes and added tests. The maintainer also added verification for the native path that could not be run at first. The related-test hook proposal has no result after adoption.
- Source: Run AI-written code and check its behavior down to failure cases
- Source: Even when several AIs share the work, the judgment of completion is made separately
- Source: The AI implements a request, and a person checks it in a preview
- Source: Connecting AI edits to the tests that relate to the change
What is still unclear
The miss rate and the reduction in overall checking effort are unknown. The related-test proposal has the issues of changes missed through Bash and a retry cap. A merge also does not show that all problems are resolved.
- Source: Run AI-written code and check its behavior down to failure cases
- Source: Even when several AIs share the work, the judgment of completion is made separately
- Source: The AI implements a request, and a person checks it in a preview
- Source: Connecting AI edits to the tests that relate to the change
Workarounds and results in more detail
Run it, check the failure conditions, and check again after fixing
Fixes and added tests confirmed in a public PR
A runtime probe and code analysis pointed out recovery handling. The maintainer fixed it and reportedly added fault-injection tests for the native path that could not be run at first.
Scope and limits: The course of one PR. It is not a detection rate for the whole product or a comparison with other methods.
A model separate from the implementer reviews against the original request
Carried out by the maker; comparison of effect unconfirmed
In Orka's later tally, 100 of 111 first reviews were fix requests. This counts review decisions, not a code defect rate.
Scope and limits: The same operator's six projects, September 28 to October 10, 2026.
Automate from request to draft PR, with a person operating and judging
Run by the person; success rate not stated
Andrew Lisowski explains that after the AI's implementation they check the diff and preview, request needed fixes, and then merge.
Scope and limits: One practice on their own app. Operation counts and effort reduction are unknown.
Record the edited scope and run related tests before stopping
Published remedy; result of adoption unconfirmed
A proposal using two existing hooks. Detecting changes made through Bash and a cap that prevents endless retries remain.
Scope and limits: A reply to one feature request. The example counts and seconds are not measurement results.
Read with care
- Provider explanations, operators' self-reports and public fix records are different kinds of basis.
- We do not count the Product Hunt, GitHub, social posts and article of the same practice as separate users or success stories.
What to check next
We look for sources that keep recording, for the same kind of change, checking time, missed defects, send-backs and recurrence after fixes.
Connect groups of edits to related tests and the return of failures
In development that fixes several files, the asker explains that running all tests is heavy and gets skipped, while the AI guesses a single test and considers the check done. Checking on every file edit also means trying the work-in-progress state many times. The replier's proposal accumulates changed files on edit and runs the related tests together when the response ends.
On success the list is cleared, and on failure the results are returned to the AI to lead to a fix. However, changes made through the shell or code generation are not recorded as they are, and the reply explains that endless retries on failures that cannot be fixed also need a cap. This is a proposed remedy, and a report of adoption or improvement by the asker is unconfirmed. The seconds and test counts shown are also usage examples and are not treated as actual results.
Questions to think with
- Are changed files and check targets captured, including shell operations and code generation outside the editing tools?
- Can you leave in the completion report the conditions related tests cannot check, and the stop cap for when failures cannot be fixed?
A user's request and a replier's proposal using existing hooks. The confirming question is an editorial suggestion.