A test automation strategy that survives contact with a roadmap

Automation encodes an expectation. If the expectation is still changing weekly, encoding it produces work that gets thrown away, and the team learns that automation is expensive rather than that they automated too early.

Signals that it is too early

• The flow under test has changed shape more than once this quarter, as opposed to changing content.
• Nobody can state the expected behaviour without asking someone else, which means the first output should be a written expectation rather than a script.
• There is no environment that can be brought to a known state, so every run starts from whatever the last person left behind.
• The team has no capacity to fix a failing test within a day, which is the point at which a red build starts being ignored.

None of these are permanent. They are sequencing problems, and the useful move is to spend a cycle of manual and exploratory testing establishing what matters and stabilising it, then automate the part that has settled.

The single most useful property of a test is that its failure tells you where to look. A browser test that goes red tells you a page is broken. A service-level check tells you which component broke it. That difference compounds across hundreds of runs and thousands of triage minutes.

The practical rule is to push each check to the lowest level that can still answer the question honestly. A login test at the browser level is often really an API-level authorisation question wearing a slower costume.

Not the newest feature, and not the easiest flow. The first tranche should be the journeys where a regression is both plausible and expensive, because those are the ones that justify the maintenance commitment you are taking on.

1. 01

   The revenue path

   Whatever sequence ends in money changing hands or a customer being created. It is the flow with the clearest cost of failure and usually the one with the most integration surface underneath it.

2. 02

   The flows your support queue already complains about

   Defect history is a better predictor of future defects than any risk workshop, and it is free. Whatever broke twice will break again.

3. 03

   Authorisation boundaries

   Horizontal checks — one account attempting another account’s resources. Cheap to write once the request scaffolding exists, and the failures are severe when they occur.

4. 04

   Everything else, slowly, on evidence

   Coverage added because something broke, rather than coverage added because a percentage looked low.

Almost every property teams want from a suite — parallel execution, deterministic results, the ability to run against any environment — is downstream of one decision: whether each test creates and destroys the data it needs, or whether tests share a fixture set that someone maintains by hand.

Shared fixtures are faster to start with and are the most common cause of a suite that cannot be trusted. Tests begin depending on state left by earlier tests, order becomes significant, parallelism becomes impossible, and a failure late in the run tells you nothing about what actually broke.

Isolated test data is not a refinement you add later. It is the thing that decides whether a suite can run in parallel, and retrofitting it means rewriting every test that assumed otherwise.

Where realistic data is genuinely required, the constraint becomes a data-protection one rather than an engineering one, and the options narrow. That is covered separately in testing with production-like data without breaking GDPR.

A suite that runs but blocks nothing is a report. It has some value, and considerably less than the cost of maintaining it. The moment automation starts paying back is the moment a failure prevents something.

That does not mean everything should gate. The sensible arrangement is tiered: fast, deterministic checks gate every pull request; slower browser journeys gate the merge to the release branch; long-running suites run on a schedule and gate nothing, because a two-hour job in the critical path teaches people to bypass it.

• Per pull request: unit and service-level checks, finishing in a few minutes.
• Per release candidate: the browser journeys, with a defined owner for triage when they fail.
• Nightly: anything slow, exploratory or environment-dependent, reviewed rather than enforced.

Wiring this properly is most of what a CI test integration engagement consists of, and it is usually the step that turns an existing suite from a report into a gate.

The deciding question for framework choice is rarely which tool is technically best. It is whether the people who will still be here in a year can read and extend what gets written. A framework only one supplier understands has a half-life measured in months after that supplier leaves.

This is the argument for building in the language your product team already uses, for keeping the abstraction shallow enough to read, and for treating handover as a deliverable rather than an email. It is also why we build frameworks to be extended by product engineers rather than by specialists, and why the in-house versus outsourced question usually resolves to a blend rather than to either pole.