Deploy SUCCESS Is Not a Production Check

Tomohiro Iida · Published August 25, 2026 · Updated August 25, 2026

The moment a deploy workflow turns green, the report becomes “it is live”. What the green actually proves is that the deploy command ran to completion. It says nothing about which code production is serving, whether the database state matches what that code requires, or what a signed-in customer can see. Netsujo now treats Deploy SUCCESS and Production verification as two separate stages, and verifies SHA identity, schema state, public and authenticated routes, browser evidence, cleanup, and rollback readiness one at a time.

Incident card

After a green deploy, only the customer screens were down

On 2026-08-16 the production customer portal returned 500 to signed-in customers. With no session cookie the response was 200. With an empty cookie value it was also 200. With a non-empty cookie the response was 500 regardless of whether the value was valid. Public pages stayed at 200, so signed-out visitors saw a healthy site.

The runtime error was SQLSTATE 42703, undefined_column. The column the code referenced did not exist in the production database. Two further facts came out of the investigation. The production deployment in question carried no commit SHA in its deployment metadata, because it had been pushed from a local working directory rather than through a Git integration. And the migration that added the column lived on a branch that had not been merged to main, so it had never been applied to the production database.

The deploy operation itself had succeeded. It was green.

The false assumption was in the choice of signal

The first assumption is that a successful deploy means production is new. A deploy result reports whether a command ran to completion. What is being served has to be confirmed separately.

The second is the choice of signal. Do not judge with a value that does not change for the change you made. A 200 on the public home page does not move when an authenticated route breaks. A generic uptime check stays green when a single route fails. A build result knows nothing about the database. A deployment timestamp does not say what was deployed.

Watching a value that cannot move produces false negatives. Watching a value that moves independently of the change produces false positives. Choose a signal that changes only because of this change.

Root cause

• Merge and production rollout were coupled, and the ability to deploy to production from a local machine remained.
• The deployment artifact carried no record of its source commit, so nobody could say afterwards what was being served.
• Schema state and the requirements of the code were never compared.
• Verification covered public routes only, with no measurement of an authenticated route.

The immediate fix

Recovery was a rollback to a deployment that had previously been served as Production. After the rollback we measured that even an invalid cookie value returned 200 on the authenticated route. Recovery was fast because a rollback could be triggered from a local machine. That same permission was also the cause of the incident.

Turning it into a system

Decision contract: deploy result and production verification are separate variables. If the Production SHA differs from the deploy input SHA, verification fails. If schema state does not match the code requirement, verification fails. If no authenticated route was observed, verification is unknown rather than pass. If the signal used for the judgement does not change with this change, the signal is invalid.

• The Git deploy policy sets every branch pattern, including main, to false. Git-triggered deployment is disabled everywhere, and the only path to production is a manual workflow that requires a confirmation string.
• A static check pins that policy, rejecting both a branch-local exemption and a return of main to true. On 2026-08-16 an agent branch added itself to an allowlist and produced 11 preview deployments in two hours; because the platform reads the pushed commit, the check has to fail at the diff.
• The production deploy command records the source commit as deployment metadata, and a step immediately after the deploy measures whether the record actually landed. If it did not, the job fails, but the deployment has already shipped: this is a missing recovery path, not a failed release.
• The deploy job and the post-deploy QA job are separate. The deploy job writes a DEPLOYED boundary into the step summary so that a later failure is not misread as “not deployed” and blindly rerun.
• Post-deploy QA resolves the SHA behind the production alias twice, before and after the QA, so that production moving during verification cannot be mistaken for this run.
• A separate read-only workflow reads the live production schema directly, because a migration log only says that something was recorded as applied.
• Authenticated customer screens are visited with a browser, and screenshots, console errors, and network errors are kept as evidence. The presence of evidence is then counted again against the required viewports, the exact SHA, the public HTTP result, and the artifacts themselves.
• The synthetic identity used by the production smoke is fixed in repository code and never taken from workflow input, because the cleanup path holds production database authority and a caller-supplied identifier would turn it into an account mutator.
• Rollback is limited to deployments whose source commit is in the history of main and which were previously served as Production. Rolling back to an unmerged preview would be a bypass, not a recovery. The requested deployment is not trusted; its metadata is fetched again and verified.

A reusable production verification checklist

• What is the current main SHA?
• Does the deploy input SHA match it?
• Is the source commit recorded on the deployment artifact?
• Does the SHA behind the production alias match the deploy input SHA?
• Did the production SHA stay still across the verification window?
• Does the live schema match what the code requires?
• Are there unapplied migrations?
• Were public routes measured?
• Were authenticated routes measured?
• Was browser evidence captured across viewports, with console and network errors reviewed?
• Does the signal used for the judgement actually change with this change?
• Was the synthetic verification identity cleaned up?
• Does a rollback target exist, and has its provenance been checked?

Evidence

Limits of application

• These steps are the Netsujo operating baseline, not a universal answer.
• How many routes to check, and how deeply, depends on the architecture and the customer impact surface.
• Measuring authenticated routes requires a synthetic identity in production and a cleanup design. Copying the practice without the cleanup pollutes production data.
• Disabling Git-triggered deployment on every branch assumes you accept the release cadence and manual overhead that follow.
• How much this separation shortened mean time to detection is not yet measured.

Remaining risks

• Production can still move during verification. Resolving the SHA twice detects that, but does not prevent it.
• Registering known findings in a classification registry risks passing a real defect as already classified. A classification is not a measurement.
• A read-only connection string has to be confirmed by privilege, not by name. A name is not a permission.
• Rollback targets age out. Whether a usable target still exists is something to check before an incident, not during one.
• Passing production verification does not guarantee that the same SHA keeps being served afterwards.

Conclusion

Deploy SUCCESS is a report that an operation finished. Production verification is an observation of what is running now. Keep SHA identity, schema state, public and authenticated routes, browser evidence, cleanup, and rollback readiness as six separate things, and the moment a green report drifts away from production reality, the drift becomes visible.

Frequently asked questions

If the deploy workflow is green, is the release complete?

The release operation is complete. Whether production behaves as expected is separate. Observe the SHA behind the production alias, the database state, and both public and authenticated routes before calling it done.

If public pages return 200, is production healthy?

That cannot be concluded. Some failures are limited to authenticated routes. In the 2026-08-16 incident, a 500 appeared only when the session cookie was non-empty, while public pages stayed at 200.

Is an uptime check enough?

No. A generic uptime check does not change with your change. Judge with signals that move only because of this change: SHA identity, the response of the specific route, and the live schema state.

If post-deploy QA fails, should the deploy be repeated?

First confirm that the release already happened. The deploy job and the QA job are separate, and rereading a red workflow as “not deployed” leads to shipping the same build again.

Is it acceptable to create a verification account in production?

Only when the cleanup is designed as well. Fix the identifier in repository code rather than accepting it from workflow input, because the deletion path holds production database authority and a caller-supplied identifier would let it operate on arbitrary accounts.

Previous: parallelise implementation, serialise integrationEpisode 09 covers running several AI agents without breaking the shared repository.

Next: routing everything through owner approval stalled developmentEpisode 11 sets authority and evidence by risk level instead of approving everything.

Back to the full incident logThe series hub lists all twelve episodes and the order to read them in.