I Expected Cutting GitHub Actions to Cost Us Quality. The Opposite Happened.

Tomohiro Iida · Published August 19, 2026 · Updated August 23, 2026

When the idea of cutting GitHub Actions came up, it made me nervous. I believed more CI meant more safety: lint, type checking, the full test suite, browser tests, database checks, security checks, and a production build on every change. Then Claude Code and Codex raised our development speed, and a different problem appeared. An AI agent commits, pushes, and updates a pull request at much shorter intervals than a person does, and every one of those events started a workflow. Before long, CI was more congested than the code.

Believing that more CI means more safety

A green CI run is reassuring. Watching only the colour is where it becomes dangerous. What deserves attention is what was checked, which SHA was checked, whether the conditions matched production, whether any path swallows a failure, and whether the check has anything to do with the current change. In our environment the AI agents had pushed GitHub Actions into overuse. The normal CI and the exact-head verification ran the same heavy suite. A docs-only change still started Postgres. Chromium was installed when no browser was needed. And by the time the change had been classified, the dependency install had already finished. That is not safety. That is routing everyone through the same inspection hall every time.

Faster agents break the assumptions behind CI

When a person pushes once every few hours, running a broad check every time keeps the run count small. AI agents are different. Builder A commits and pushes, Builder B commits and pushes, Builder C commits and pushes, then review fixes and test fixes each push again. At that point making a single test 20 percent faster is not enough. If the trigger count doubles or triples, internal speed-ups do not clear the queue. The first question was not how fast the test code runs. It was whether this check should run at all right now. We had scaled up the agents without designing that, and GitHub Actions became the new bottleneck.

From 9 minutes 31 seconds to 2 minutes 9 seconds

After introducing path-aware CI, we ran a canary on a real pull request that changed documentation only.

The classifier reported needs_code false, needs_browser false, and needs_database false. That does not mean nothing was verified. Checks that a Markdown change can still break stayed in place: consistency of public claims and contract wording, workflow safety, deploy policy, CI coverage, agent operating rules, and static row-level-security coverage. A docs-only change was not waved through. We simply did not start runtime infrastructure the change could not affect.

We stopped Draft runners first

Pouring a full CI run into a Draft pull request on every push produces results that the next push makes stale. Previously each Draft push started CI, and Ready started it again. Now Draft pushes start no hosted runner at all, and CI runs when the pull request becomes Ready. We did not abandon checking during the work itself. Scope-appropriate checks run locally or in a dedicated environment, and the GitHub-hosted runner is reserved for the integration evidence produced at Ready.

Replacing “run everything” with “classify the change”

A lightweight classify job now sits at the very start of CI and looks only at the changed paths. It decides three things: needs_code, needs_browser, and needs_database. The important part is that this decision happens before the dependency install. We already had a mechanism to skip later tests, but by the time the skip was decided, the runner had started, Postgres was up, and the dependencies were installed. The work was “not run” while most of the cost and waiting time had already been spent. Moving classification to the entrance made it possible to start only the runtime infrastructure a change actually needs.

The classifier does not run from the pull request side

Path-aware CI has a real attack surface. If a pull request can modify the classifier itself, it can declare itself safe and delete the heavy checks. So the classifier runs from the trusted version on the base SHA, not from the pull request head. A pull request that changes CI or the classifier cannot exempt itself and falls to the full path. Unknown paths, a failure to fetch the changed-file list, an unclassifiable diff, changes to package manifests or the lockfile, and browser test configuration all fall to the safe side as well. The rule is short: if it can be proven safe, it may be skipped; if it is unclear, it runs. Speed is never bought by inverting the safe default.

Exact-head verification is not a copy of full CI

Confirming that the current pull request head and the SHA under verification are the same matters. Confirming it does not require running the entire heavy suite a second time. Previously the same head carried two heavy checks that produced the same evidence twice. We split the roles instead. A Ready pull request gets an integration CI run plus an exact-head identity check. The Final Candidate gets one exact-SHA full gate. The same evidence is not manufactured twice, the strong check happens once at the stage that needs it, and it is obvious which result the final decision uses.

A skip is not a PASS

The evidence semantics mattered as much as the technical optimisation. For a pull request where the browser suite was skipped, we do not record browser as PASS, because it did not run. The correct value is NOT_RUN, together with separate evidence explaining why the omission was allowed. The same applies to a pull request that never started Postgres: the database check did not succeed, it did not happen.

Reducing execution volume and maintaining evidence strength are managed separately. The Final Candidate holds its own full exact-SHA gate so the required evidence lines up on one SHA.

Fewer checks made defects easier to see

When there is too much CI, a failure stops being information and becomes noise. An unrelated flaky test fails, an older run is cancelled, several workflows hit the same SHA, and separating which red belongs to the current change turns into its own job. Time goes into returning CI to green rather than into the defect. Running a narrow, reliable check close to the change has the opposite effect: the failure means something. One concrete example is a Next.js route module with an extra export, which only breaks the production build. Instead of adding a heavy production build to every pull request, we built a static gate that detects that one class of defect in seconds. The criterion is not how long a check takes. It is whether a defect that actually occurred before will reliably be stopped next time. Measured that way, a small gate shaped around a known failure mode can be stronger than a broad, heavy suite that runs every time.

Stop runners that cannot change any state

Lightening the pull request CI does not remove fixed consumption if neighbouring workflows react to the same events. Our Agent Integration Controller is the single writer that evaluates pull requests produced by multiple AI agents and sends the integrable ones to main. It used to start immediately after a push to a Ready pull request, at a moment when CI was still running and no merge was possible even if the Controller behaved perfectly. Normal integration decisions now hang off the successful CI completion event. The Controller runner does not start when a Draft pull request skipped CI, when CI ended in failure or cancellation, or when CI has not finished for the new head of a Ready pull request. Stopping the normal CI runner on the Draft side is not enough on its own: if another Controller starts on the skip notification, the fixed consumption survives. The whole event chain has to be examined.

The 10-minute cron became a recovery fallback

The Controller also had a schedule for re-evaluating state after a missed event or a transient failure. It ran every 10 minutes, which is 144 times a day and 4,320 times over 30 days. Normal integration can start immediately from pull request events and CI completion events, so the schedule is a recovery path, not the normal path. Moving it to hourly gives 24 runs a day and 720 over 30 days, an 83.3 percent reduction in scheduled starts. Normal integration waiting time did not increase. Event-driven starts remain the normal route, and the cron is limited to re-evaluating state within roughly an hour. The principle is simple: ask not only whether this job is necessary, but whether starting it right now can change any state.

Post-merge workflows burn the same fixed cost

Optimisation stops halfway if a separate post-merge workflow unconditionally begins heavy preparation every time something lands on main. The main post-merge check also moved classification ahead of dependency installation. For docs-only and README-only changes it skips the dependency install, the fact-consistency check, the lint and type check cache restore, lint, and type checking. The conflict marker check needs no dependencies, so it always runs. Unknown paths, an unavailable diff, and an unclassifiable change fail closed to the code-change side. We did not build a separate docs classifier for that workflow. The same classifier used by the pull request CI is the single source. Duplicating classification logic per workflow is how one copy quietly becomes more permissive than the other. Across the whole lifecycle, from Draft to Ready to merge to Controller to post-merge, the rule is to avoid starting expensive preparation before it is needed.

Quality is not measured in minutes spent

A large GitHub Actions bill looks like thorough checking. Runner minutes are not what protects quality. What matters is which defect a check stops, whether it is checking the current SHA, whether it ran in the required environment, whether a failure always reaches a gate, and whether the reason for an omission can be explained. On that basis, removing duplicate runs was not a reduction in quality. It improved the freshness and the meaning of the evidence.

What we removed was ritual, not quality

The dangerous version of this work is deleting checks because you want speed. That is just cutting corners. The safe version requires being able to answer five questions about each check.

• Which defect does it stop?
• For which kind of change is it needed?
• At which stage is its evidence required?
• If it is omitted, does another gate cover the same ground?
• If the decision itself fails, does it fall to the safe side?

A workflow that cannot answer those tends to survive as a reassurance ritual. I used to treat running a lot of CI as quality management. I now treat quality management as stopping the necessary defect, in the necessary place, for the change in front of me.

Design the CI that does not run

• Do not start GitHub-hosted runners for Draft pull requests.
• Classify changed paths before the dependency install once the pull request is Ready.
• Keep code, browser, and database as separate axes.
• Separate exact-head identity confirmation from full validation.
• Send unknown changes and CI self-modifications to the full path.
• Turn a classification failure into a failing required check.
• Never treat a skip as a PASS.
• Assemble the required exact-SHA evidence at the Final Candidate stage.
• Do not start the Controller runner at moments when it cannot change state.
• Keep cron as an event-driven recovery fallback rather than the normal route.
• Classify before installing dependencies in post-merge workflows too.

The first question for CI is not how to run it faster. It is whether this check should run at all right now. Assemble the required evidence completely, but only at the moment it is required. Turning AI agent speed into business speed rather than runner queues and invoices means treating that as part of the design.

Evidence

Limits and remaining risk

• The measured numbers come from one docs-only canary in our repository. They are not a general benchmark.
• Path-aware skipping depends on paths correlating with risk. Where that correlation is weak, the safe range has to widen.
• A change to the classifier patterns themselves is the highest-leverage place for a mistake, which is why it is forced to the full path.
• Reducing runner minutes says nothing about the detection power of the remaining checks.

Frequently asked questions

Does skipping checks on a docs-only change let defects through?

The checks that a Markdown change can break were kept: public claim consistency, contract wording, workflow safety, deploy policy, CI coverage, agent operating rules, and static row-level-security coverage. Only runtime infrastructure the change cannot affect was not started.

Why classify before installing dependencies rather than skipping tests later?

A later skip still pays for the runner start, the database start, and the dependency install. Most of the cost and the waiting time is already spent by then.

What stops a pull request from marking itself as safe?

The classifier executes from the trusted base SHA rather than the pull request head, and any pull request that touches CI or the classifier is sent to the full path.

Does the hourly cron slow integration down?

Normal integration runs on pull request and CI completion events, so it is not waiting for the schedule. The cron only re-evaluates state after a missed event, within roughly an hour.

AI agent development and operations incident logThe series hub lists every incident in this record and the order to read them in.

Parallelize implementation, serialize integrationThe separate article covers running several AI agents in one repository without breaking it.

Separate the roles before adding more AIThe separate article covers how design, implementation, audit, and human responsibility are split.

Continue the series

Previous: #07 Classify CI failures before you press rerun

Next: #09 Parallelize implementation, serialize integration

Back to all 12 episodes of the AI Agent Development Incident Log