A Red CI Run Is Not a Reason to Press Rerun

Tomohiro Iida · Published August 25, 2026 · Updated August 25, 2026

The shortest possible action after a CI run turns red is to press rerun. Some failures really do go green on a second attempt. The problem is moving forward without being able to explain why it went green. Netsujo now classifies a CI failure before rerunning it: save the first failure evidence, extract a failure signature, and assign it to deterministic, flaky, infrastructure, rate-limit, or unknown. Only then do we decide whether a rerun is allowed, whether code has to change, or whether a human has to judge.

Incident card

A red run was being erased by pressing the button again

The failures we observed had completely different natures and all appeared in the same colour. On 2026-08-23, the primary rate limit on the GitHub Actions installation token stopped the workflow that verifies pull request head identity with HTTP 403. Measured that same day, the SIGNAL PR Build Evidence workflow started 100 runs in 95 minutes: 39 succeeded and 60 failed with 403. That failure had nothing to do with the content of the pull request, yet the commit status remained a failure. A different failure ended with exit 134, a heap out-of-memory in tsc that exceeded the roughly 2GB Node default; and when several Git worktrees were placed under a shared checkout, raising the Node heap to 8GB did not stop the out-of-memory condition, because complete copies of the same repository had entered the analysis scope of ESLint and TypeScript. Over the same period we also saw lint errors, type errors, and assertion failures, which reproduce every time for the same input. Treating all of these with a generic rerun makes the pipeline pass sometimes and fail other times, and only the passing attempts stay in the record.

Two false assumptions were stacked

The first is that a green rerun proves the original failure was harmless. A rerun changes the execution environment and the clock, not the code. Green only says the failure did not reproduce this time. The second is that the diff you touched last is the cause, which is especially dangerous for resource-limit failures: out-of-memory conditions and rate limits cross a threshold when input volume or call frequency changes slightly, so if reverting one file makes the run pass, the cause may be total volume rather than that file. To separate the two, move both the diff and the resource setting and compare the results. One observation in which only one variable moved is not a basis for a causal claim.

Root cause

The root cause is that a failure was treated as a one-off event. Concretely, the following were missing. Without the last of them, flaky and deterministic cannot be told apart: a first red and a fifth red look the same and mean different things.

• The evidence of the first failure was not saved.
• No failure signature was extracted from the exit code and message.
• There was no definition for classifying a signature.
• Rerun had no precondition, so anyone could press it any number of times.
• Nobody counted how many times the same signature had appeared.

The immediate fix

• Save the failing workflow, job, step, exit code, and the first 20 and last 50 lines of the log.
• Write the failure signature in one line, from the exit code and the error message.
• Pick one classification and write one line explaining why that classification was chosen.

Take those three before the red run disappears, because depending on the concurrency configuration a newer commit cancels the older run and the original failure log becomes unreadable. Saying “I looked at the log” also does not let anyone decide later whether a failure was the same one, so decide the fields in advance: workflow, run id, run attempt, job, step, exit code, subject SHA, start time, runner, and the head and tail of the log. Run attempt and subject SHA are mandatory, because without those two nobody can tell afterwards whether the result varied on the same SHA or whether a different SHA produced a different failure, and the flaky judgement depends entirely on that distinction. Comparing raw logs then makes the same failure a different string every time, so drop the parts that change on every run: timestamps, run ids, temporary directory paths, elapsed time, and line numbers. What remains is the signature, and it holds three elements.

• The exit code, or the terminating signal.
• The word that determines the error kind, such as heap out of memory, rate limit, not ok, or a type error code.
• The names of the job and the step where it occurred.

Those three joined into one line are the signature. Too fine a granularity counts one failure as several. Too coarse a granularity puts unrelated failures into one bucket. The test is whether you can choose the next operation by looking at the signature; if you cannot, it is too coarse.

Turning it into a system

• Deterministic: the same input always produces the same result. The deciding evidence is that two runs on the same SHA fail at the same step with the same exit code.
• Flaky: the result varies for the same SHA and the same configuration. The deciding evidence is a success and a failure with different run attempts under one signature. A single observation cannot decide it.
• Infrastructure: runner startup, network, disk, or memory constraints. Exit 134 is the representative case, a heap out-of-memory beyond the roughly 2GB Node default. The deciding evidence is that the failure depends on the amount of a resource rather than on the meaning of the code.
• Rate-limit: an external API call budget was exhausted. The deciding evidence is HTTP 403, HTTP 429, and the words rate limit in the response body. Another workflow consuming the same budget at the same time makes the class more certain.
• Unknown: none of the above. Having no deciding evidence is itself the result. Do not force it into one of the other four.

Preconditions for an allowed rerun

• The failure evidence has been saved.
• The signature can be written in one line.
• The classification is flaky or rate-limit.
• One line of justification for that classification has been written.
• A fix task will be opened regardless of the rerun result.
• No code was changed immediately before.

The last precondition matters most. If code changed, this is not a rerun but a new verification. The previous green and the previous red both close as results belonging to the previous SHA. Decision contract: deterministic forbids rerun; a code change requires a fresh exact-head CI and makes the previous green stale; unknown escalates to a human.

• The CI concurrency group separates ordinary pull request runs from manual reruns, because a rerun reuses the older event payload and would otherwise let a stale SHA cancel the run for the newest SHA.
• A “Reject stale PR head rerun” step compares the event head SHA with the live branch ref over the Git protocol and fails when the two 40-character SHAs differ.
• A separate exact-head identity workflow confirms the pull request head through three paths: local checkout, event head, and remote branch ref.
• The gh API retry helper retries only rate-limit errors, at 5, 10, 20, and 30 seconds. Errors such as 401, 404, and 422 return after one attempt, and an exhausted retry budget returns the last non-zero status rather than a fabricated success.
• The typecheck heap contract is pinned at 4096MB and a regression test detects the removal of that setting, so a failure classified as infrastructure cannot be closed by raising a limit alone.

Path-aware CI branching, what to do about a check that was never executed, and how the classifier itself is kept out of the hands of the change being classified are covered by the episode on CI execution branching. How a green deploy is separated from a confirmation that production behaves as expected is covered by the episode on production verification. Both are linked at the end of this article.

A reusable checklist

• Was the failure evidence saved, including the run attempt and the subject SHA?
• Can the signature be written in one line, with the variable parts dropped?
• Was it counted how many times that signature has appeared before?
• Was one classification chosen, with a reason?
• For deterministic failures, are you changing code instead of rerunning?
• For flaky failures, was a success and a failure confirmed on the same SHA before registering the test and opening a fix task?
• For infrastructure failures, did you measure the analysis scope, file count, duplicated repositories, and caches?
• For rate-limit failures, is the retry bounded and free of fabricated success?
• Are you resisting the urge to force an unknown into another class?
• After the code change, did you run a fresh exact-head CI?
• Does the verified SHA match the SHA you are about to operate on?

Evidence

Limits of application

• These classes and steps are the Netsujo operating baseline, not a universal answer.
• Retry delays and counts have to match the timeout of the calling job, so the values cannot be copied directly.
• The heap value depends on the memory available to the runner.
• Allowing one rerun for flaky failures depends on how safety-critical the test is; some checks should allow none.
• The effect of this classification is not yet measured. Recurrence rate and time spent red are the metrics we plan to track.

Remaining risks

• A wrong classification lets automation proceed on the wrong assumption, and confusing infrastructure with deterministic produces green while the cause remains.
• Forcing unknown into another class defeats the mechanism. Unknown is a legitimate result.
• Automatic retry does not reduce the call volume that causes the rate limit.
• A fresh exact-head CI still leaves integration with a moving base branch as a separate question.
• Logs expire. Evidence has to be captured while the run still exists.

Conclusion

The first decision after a red CI run is not whether to press the button. It is which class the failure belongs to. Saved failure evidence, an extracted signature, one of five classifications, the operations that classification permits, and a fresh exact-head CI after any code change: together these turn a rerun from a hopeful gesture into a decided operation.

Frequently asked questions

Should rerun be banned?

No. What is banned is a rerun you cannot explain. Transient external causes such as a rate limit are handled with a bounded retry, while deterministic failures do not change on rerun.

Reverting one file made it pass. Is that file the cause?

One observation cannot decide that. Resource-limit failures depend on whether total volume crosses a threshold, so move both the diff and the resource setting before concluding.

How fine should the signature be?

Fine enough that you can choose the next operation by looking at it. Drop the variable parts such as timestamps, run ids, temporary paths, and line numbers, then fold what remains into the exit code, the word that determines the error kind, and the job and step names.

Can the previous green be reused after a code change?

No. The previous green belongs to the previous SHA. After a code change, run CI against the exact head you are about to operate on.

How should a known flaky test be handled?

One rerun may move the work forward, but the work does not stop there. Register the test and open a fix task, otherwise the same test keeps consuming one rerun at a time.

Previous: CLOSED is not MERGEDEpisode 06 checks a claimed fix through a five-layer evidence ladder.

Next: Cutting the Actions bill nearly cut the safety checksEpisode 08 covers path-aware CI branching that keeps the required checks.

Back to the full incident logThe series hub lists all twelve episodes and the order to read them in.