Diagnosing flaky tests: a procedure, not a retry

Every automated suite eventually arrives at the same fork. A run goes red, somebody presses the button again, the second run is green, and the change merges. That re-run is a decision about the value of the whole suite, and it is almost never discussed as one.

Flakiness is not a single defect with a single remedy, which is why generic advice about it is so unsatisfying. There are four distinct families, each with a diagnostic tell, and telling them apart takes far less effort than most teams assume. What follows is the order we work through them when we are brought in to stabilise a suite somebody else built.

Measure before you guess

You cannot triage this from memory, and memory is what most teams use. The engineer fixes the test that annoyed them on Tuesday, which is rarely the one costing the most time. Before anything else, start keeping per-test results across runs, keyed by a stable test identifier rather than by a line number.

Two numbers then fall out of that history, and they are the ones worth acting on:

• The failure rate on runs where the relevant code did not change. A test that goes red against an untouched area is either unreliable or is telling you something about a shared dependency, and both are worth knowing.
• That rate multiplied by how often the test executes. A weekly test failing one time in five wastes less of the team’s week than a per-commit test failing one time in fifty. Ranking by rate alone sends people to the wrong end of the list.

Until failures are counted per test across runs, every conversation about flakiness is a conversation about whose irritation was most recent.

Four families, and the tell for each

Leaked state between tests

The tell is positional. The test passes when executed on its own and fails when the whole suite runs; or it fails only in one shard; or it started failing the week somebody enabled parallel workers. The cause is a record, a session, a cache entry or a configuration change that one test leaves behind and another silently depends on.

Two commands usually settle it: run the single test in isolation, then run the suite with execution order deliberately randomised. If randomising the order changes the failure pattern at all, stop investigating anything else — you have found the family.

Timing assumptions

The tell is load sensitivity. Failures increase when the machine is busy, when the worker count goes up, or when the pipeline runs during the working day rather than overnight. A test that waits a fixed number of seconds is the obvious version; the subtler version is a test that waits for an element to be visible and then acts on the assumption that the data behind it has also arrived.

Modern runners have made this family less common and more insidious. Automatic waiting absorbs the first several hundred milliseconds of every race, so the fragility only reveals itself later, on a slower runner or under contention. That is a known cost of the convenience rather than a reason to give it up — it is written down among the genuine drawbacks of the tooling.

The environment underneath

The tell is clustering. Unrelated tests fail together, the failures group in time rather than by feature, and the pattern correlates with a deployment, a certificate rotation or somebody else’s outage rather than with any commit of yours. Suites that call third-party services without intercepting them live permanently in this family.

The diagnostic question is whether failures track the clock or the code. If a chart of failures over time shows bands, the tests are reporting on infrastructure and were never really unstable.

A genuine race in the application

The tell is the absence of the other three. Order is irrelevant, the machine is idle, nothing else failed, and the test still goes red one run in thirty. This is the good outcome, and it is routinely thrown away. A test that intermittently catches a real concurrency defect is doing exactly the job it was written for, and a retry setting is how that finding gets deleted before anyone reads it.

The procedure

1. 01Quarantine on the first occurrence. Remove the test from the merge gate, keep it executing on a separate schedule so the data keeps arriving, and attach a named owner and a date. A quarantine with neither is a deletion nobody had to argue for.
2. 02Make it fail on demand. Loop the test in isolation, then in the full suite, then with randomised order, then inside the container image the pipeline actually uses, then with the machine deliberately loaded. Until you can produce the failure when you want it, you are not debugging, you are waiting.
3. 03Bisect the conditions rather than the code. Change one variable per run: worker count, order, environment, the presence of a neighbouring test. The variable that flips the outcome names the family, and the family names the fix.
4. 04Fix it structurally. Every durable repair is boring: the test creates its own data, the clock is injected, randomness is seeded, external calls are intercepted, the sign-in state is established once and reused. Clever fixes here have a short shelf life.
5. 05Prove the repair with the same loop that produced the failure, at the same worker count, and only then return the test to the gate.
6. 06Close the class, not the instance. If one test leaked a record, look at what else uses that fixture, because the pattern was copied from whichever test was written first.

Four responses that make it worse

• A blanket retry count in the runner configuration. It buys a green pipeline at the price of the signal you were paying for, it triples the run time of a bad day, and it permanently hides the fourth family.
• Fixed sleeps. They convert a fast intermittent failure into a slow intermittent failure, and they accumulate, because the number that worked last month is always slightly too small this month.
• Muting the test without recording that you did. Six months later nobody can say which journeys are actually covered, which is worse than knowing you have a gap.
• Changing runner. Almost every flaky suite we have looked at would have been flaky in any tool, because the causes sit in data and environment rather than in the driver. Migration is an expensive way to postpone the architectural work that was always the answer.

The properties that make flakes rare

Prevention is not vigilance, it is a short list of constraints imposed early enough to be cheap. Each test builds the state it needs and disposes of it. Nothing asserts against the real current time. Random values come from a seed recorded in the report. Calls leaving your system are intercepted by default and allowed through by exception. The suite runs with parallelism switched on from the first week, so the assumptions that only hold in single-file execution never get written.

Add one operational rule to that list: the gate holds only tests that have earned it. Promoting a new test straight into a blocking check, before it has run repeatedly against unchanged code, is how a team learns to reach for the bypass — which is a pipeline design question rather than a testing one.

When to delete the test

A test that has been quarantined and repaired twice is not unlucky. It is telling you that what it checks cannot be observed reliably at the level it is written, and the useful move is to ask what it was defending. Often the same risk can be covered by a service-level check that answers in milliseconds and has nothing to race against. Occasionally the honest answer is that the risk was never worth a maintained test, and the suite gets shorter. A smaller suite that is believed outperforms a larger one that is re-run.