Flaky test

A flaky test is one whose outcome varies while the code under test stays the same. Run it repeatedly against a single commit and it fails on some of those runs. The variation belongs to the test, its environment or its timing rather than to the product, which means a red result has stopped being informative: nobody can tell a genuine regression from noise without running it again and seeing what happens the second time.

The damage lands on trust rather than on wasted minutes. Once a suite fails intermittently, people retry until the pipeline turns green, and that reflex will eventually be applied to a real failure on a busy afternoon. This is how a genuine defect ships past a passing build. A handful of unreliable tests is therefore worse than it looks on a dashboard, because their real effect is not the runs they waste but the authority they remove from every other test running beside them. The cost is also hard to see in a report, since a suite carrying a retry habit still publishes a pass rate that looks perfectly healthy.

The usual causes, roughly in order of frequency

1. 01Waiting on elapsed time instead of on state. A fixed sleep is a bet about how fast a machine will be, and a shared build runner under load is not as fast as the laptop the test was written on.
2. 02Shared data. Two tests writing to the same record pass in isolation and fail together, which is why a suite starts misbehaving on the day somebody enables parallel execution.
3. 03Order dependence. A test relying on something an earlier test left behind will fail the moment the runner shuffles or splits the execution.
4. 04The environment itself. Animations, clock and time zone differences, calls out to real third-party services, and containers competing for resources all produce failures nobody wrote.

A test that has to be run twice to pass has already failed, whatever the second run happens to say.