CI test gating: deciding what a red build is allowed to stop

Connecting an automated suite to a pipeline is an afternoon of configuration. Deciding what that suite is permitted to stop is the part that takes months to get right, and it is not really an engineering decision. It is an agreement between everyone who will one day be blocked by it, encoded in a file.

Teams get this wrong in both directions. Some gate on nothing, so failures are advisory and quietly stop being read. Others gate on everything at once, which produces bypasses within a fortnight. The mechanics of wiring the jobs up are the easy half.

What a gate is actually promising

A required check makes a specific claim to everyone in the repository: if this is red, your change is wrong. That claim is the entire value of the mechanism, and it is spent every time the check is red for a reason that has nothing to do with the change — a leaked record, a slow runner, an expired certificate, a third party having a difficult morning.

People do not usually respond to a false block by complaining. They respond by learning a workaround, and the workaround outlives the cause by years.

A gate claims that red means the change is wrong. Every false block spends that claim down, and an apology does not put it back.

Four tiers, defined by what each may stop

The useful way to organise this is not by test type but by trigger, budget and consequence. Four tiers cover almost every product.

On every push, in seconds

Type checking, linting, unit tests, a build. This tier blocks unconditionally and should never need a conversation, because a failure here is unambiguous and the feedback arrives before the author has switched context. If it takes longer than the time it takes to read a notification, something has been put in the wrong tier.

On a pull request, in minutes

Service-level checks and a small number of browser journeys covering the paths where a silent regression would be expensive. This tier blocks the merge, and it is the one that has to be earned rather than assumed. The right size is set by the wait a reviewer will actually sit through rather than by how much coverage exists.

After merge, before release

The broader browser suite, the cross-browser matrix, anything too slow for a pull request. This tier blocks a deployment rather than a merge, which is a meaningful distinction: it keeps the main line moving while still standing between a defect and a customer. Failures here need to page someone, or the tier is decorative.

On a schedule, blocking nothing

Long-running suites, real third-party sandboxes, performance runs, exhaustive matrices. This tier is where tests go to die. It blocks nothing, so nobody is obliged to look, and a scheduled job with no named reader accumulates failures until the accumulated failure is the normal state. If you cannot name the person who reads it on a Monday morning, delete the job rather than pretending.

The order to switch gating on

Turning a new suite into a required check on its first day is the single fastest route to somebody adding an override. The sequence that works is slower and holds:

1. 01Run the tier in reporting mode against real changes for long enough to build a pass history. Days of data, not one impressive demonstration.
2. 02Agree the threshold in writing before anything blocks. What failure rate against unchanged code is acceptable, and who decides that it has been met.
3. 03Promote in increments. Individual tests, or one tier, never the entire suite in a single commit. A gate that arrives gradually is argued about gradually.
4. 04Only then set it as required in branch protection, and record the date. When somebody proposes relaxing it in six months, that date is the beginning of the conversation.

A quarantine route has to exist before the first gate closes, not after the first emergency. One person should be able to pull a single misbehaving test out of the blocking set without convening anybody, provided that act is recorded and carries a deadline — which is the same discipline that stops flakiness compounding.

The bypass problem

Every organisation ends up with a way round the gate: an administrative merge, a magic phrase in a commit message, a label that skips a job. The instinct is to remove them all. That instinct is wrong, and teams that act on it get worse behaviour rather than none — releases held hostage by an unrelated failure, or changes split into pieces small enough to slip past.

Keep the escape hatch and make it expensive to use quietly. Three properties are enough:

• Attributable. The bypass records who used it and on which change, without anyone having to volunteer that information.
• Visible. It announces itself somewhere the team already reads, at the moment it happens rather than in a monthly report.
• Reviewed. Somebody looks at the list on a fixed cadence and asks what made each one necessary. A bypass used four times for the same reason is a defect in the gate, not in the person.

An override nobody can see is not a gate with an exception. It is an advisory check with extra steps.

Green on the branch, red on the main line

Two changes can each pass every check in isolation and break the product when combined, because neither test run ever saw the other branch. This is the failure mode that makes people distrust a pipeline they otherwise built correctly, and it is not a test quality problem at all.

The remedies are ordered by cost. Requiring a branch to be current before merging catches most of it and costs a rebase. A merge queue that tests the combined result before landing anything catches the rest, at the price of serialising merges, which is a fair trade above a certain team size and pure overhead below it. Choose deliberately rather than adopting whichever one the platform advertises.

When the budget is blown

Every gating tier needs a wall-clock number agreed with the people who will be waiting for it, and that number will eventually be exceeded. The responses have a correct order.

1. 01Move checks down a layer. Most of what a slow suite verifies through a browser is a statement about data, and it runs in a fraction of the time one level below.
2. 02Shard across more workers, having first confirmed that the tests can honestly run at the same time.
3. 03Split the tier, moving the slowest journeys to the post-merge stage and accepting that they now protect a release rather than a merge.
4. 04Delete. There is always coverage nobody would write today, and removing it is the cheapest minute you will ever buy.

Renting larger runners appears on nobody’s list because it works once and then stops working. It buys a fixed improvement against a suite that grows, and it removes the pressure that would otherwise have produced the layering decision that was needed anyway.

Who owns a broken main line

This is the question that decides whether any of the above survives contact with a busy quarter. A named role, rotating if you like, is responsible for a red main line inside an agreed window. The default remedy is to revert the change rather than to fix forward under pressure, because a revert is reversible and a hurried fix at the end of the day is how a second defect gets in.

Write the rule down and give it an owner. Pipeline conventions held only as shared understanding erode within two releases, and the erosion is never noticed until the next time something red is ignored.