When to automate a test, and when the answer is no

The decision is taken one case at a time, and most teams take it with one input: how important is this flow. Importance on its own is the least reliable signal available. It reliably produces a fragile automated check guarding the part of the product that matters most, which is close to the worst arrangement obtainable.

A more useful starting position is that the default answer for any individual candidate is no, and the case for yes has to be made. That sounds obstructive. In practice it is what keeps a suite small enough to be believed.

What a test costs after the day it is written

Writing it is the deposit. The recurring charges are what decide whether it was worth it: seconds added to every pipeline run, an edit every time the area is refactored, triage attention every time it goes red for a reason that is not a defect, and one more thing a new engineer has to understand before they can change anything nearby.

A test is not a purchase, it is a standing order. The question is never whether it is worth writing but whether it is worth keeping for two years.

Framed that way, the size of a healthy suite stops being an ambition and becomes a budget, which is also how the time cost of a regression pass should be reasoned about on the manual side.

Five signals

The last two are the ones teams skip, and they are the ones that most often overturn the verdict.

Cost to automate is not a fixed property of the case, it is a property of the case and the layer together. The same rule about discount eligibility might take a day to automate through the interface and twenty minutes as a service-level check. Deciding "is this worth automating" without first asking "at which layer" is how teams conclude that a cheap check is expensive.

Judgeability is the harder one. A machine can confirm that a confirmation email was queued. It cannot confirm that the email reads sensibly, arrives in the right folder, or renders acceptably in a mail client nobody on the team uses. Automating the part that is checkable and quietly pretending it covers the rest is a common and expensive self-deception.

Four verdicts, not two

The framing of automate-or-not hides the two answers that are most often correct.

Automate it here

Repetitive, settled, silently damaging, cheap at this layer, and mechanically judgeable. Authentication and the primary revenue path usually qualify, and so do a surprising number of unglamorous data rules that nobody would nominate.

Automate it one layer down

The most common correct verdict, and the one that requires someone to ask what is actually being checked. If the assertion is really about a calculation, a permission or a stored value, the browser is a slow and unreliable way to observe it. Moving the check to the service layer typically makes it faster by two orders of magnitude and removes every cause of instability that came from rendering.

Keep it manual, deliberately

A written manual case is a legitimate destination, not a holding pattern. Anything requiring judgement, anything on unfinished behaviour, anything run twice a year: these belong in a documented pass with a named owner. The failure is not keeping cases manual, it is keeping them manual by accident and calling it a backlog.

Stop checking it at all

Every mature suite contains cases nobody would write today, guarding features that were deprecated, edge conditions that can no longer occur, or behaviour that changed and was patched into passing rather than reconsidered. Deleting them is real work with a real return, and no team ever schedules it.

A useful habit is to attach the verdict to the case in writing at the moment it is taken, along with the reason. Six months later the reason is what allows somebody to revisit the decision instead of re-deriving it, and it prevents the same case being nominated for automation twice by two different people who each assumed nobody had considered it.

Cases that look automatable and are not

• Anything whose expected result is "the layout looks right". Pixel comparison can be made to work, and it arrives with a maintenance bill that should be agreed in advance rather than discovered.
• End-to-end email or SMS delivery. The queueing is checkable. Whether a message reaches a real inbox depends on reputation and filtering, which are not properties of your build.
• One-off migrations and cutovers. High stakes, zero repetition, and by the time the automation is trustworthy the event has happened.
• Flows requiring a code from a real handset. There are workable routes around this, and each one is a decision about your non-production environments rather than a trick.
• Anything whose behaviour is still under argument. Automating it now buys maintenance rather than coverage, which is the sequencing mistake behind most abandoned suites.

The reverse mistake

All of the above argues for restraint, so it is worth naming the failure in the other direction, which is at least as common and much quieter. A team runs the same manual checklist before every release, has done for a year, and has never once totalled the hours. Nobody proposes automating it because no single instance feels expensive enough to justify the project.

The test for this is arithmetic rather than opinion: multiply the length of the pass by the number of releases in a year, then compare that to a fortnight of engineering. The cases that survive that comparison are the ones an automation engagement should start with, and they are almost never the ones a team nominates from memory.

The rule, compressed

Automate what you already repeat, at the lowest layer that can honestly answer the question, only once the behaviour has stopped moving, and only where a machine can tell right from wrong without a person squinting at the result. Everything else is a manual case, a deletion, or a conversation about what you were really trying to find out.