Skip to content
steelabs

Checklist

Flaky-test investigation flowchart and log

Follow a repeatable investigation from a failing run to a supported cause. Preserve evidence before rerunning and track quarantine decisions explicitly.

Download editable Markdown

How to use this

Useful for: Teams whose release pipeline fails inconsistently and whose reruns hide the original signal.

  1. Save the first failing trace, log and configuration before rerunning. Record the candidate build and test seed so a later green result does not erase the conditions of the failure.

  2. Compare an isolated execution with the suite and parallel execution. A difference points toward shared state or ordering; it does not by itself prove the application is correct.

  3. Investigate waits, external dependencies and environment contention separately. Replace a timing guess with an observable condition only after confirming the required behaviour.

  4. If quarantine is necessary, name an owner and expiry. Keep the missing release signal visible and document the alternative check until the test can return to the gate.

Scope and limitations: This workflow narrows hypotheses. A green rerun cannot establish the cause of a failure, and retries alone do not repair an unreliable test.

Document preview

# Flaky-test investigation

## Decision flow
1. Preserve failing evidence: build, environment, worker, seed, trace, logs and test data.
2. Can the failure repeat in isolation with the same starting state?
   - Yes: compare expected behaviour with actual responses; investigate the product or the test expectation.
   - No: run alongside the neighbouring tests with the same data arrangement.
3. Does the failure depend on order or parallel execution?
   - Yes: inspect shared accounts, mutable fixtures, cleanup and resource limits.
   - No: inspect explicit waits, asynchronous completion and external dependencies.
4. Does the application violate the agreed requirement in the captured evidence?
   - Yes: open a product defect with a reproducible state.
   - Unknown: obtain the missing observation or clarify the requirement.
   - No: repair the test or environment and demonstrate the corrected condition.
5. Repeat under the conditions that previously failed. Document the result and remaining uncertainty before restoring the gate.

## Investigation log
| Run / build | Isolation or suite | Workers / seed | Data state | Result | Evidence |
| --- | --- | --- | --- | --- | --- |
| Original failure | | | | | |

## Hypothesis register
| Hypothesis | Observation supporting it | Experiment | Outcome | Next action |
| --- | --- | --- | --- | --- |
| Shared account mutation | | | | |
| Missing completion condition | | | | |
| External provider variability | | | | |

## Quarantine record
- Affected check and release risk:
- Alternative evidence until repair:
- Owner / expiry / issue:
- Conditions required for restoration:

Example — fictional: a test passes alone but fails when another worker deletes its shared account. The repair creates an account per test and cleans up only that account. Verify the previously conflicting tests together; increasing a timeout would not address the observed data collision.

Read alongside this worksheet

Get help applying itAll resources