Building a test automation framework from scratch: the first 90 days

Most failed automation efforts fail the same way. Someone records a few scripts, they work, everyone is encouraged, and three months later the suite takes forty minutes, fails unpredictably, and one person understands it. The problem was not the tooling. It was that scripts were written before a framework existed.

This is the sequence we use, and roughly what falls where in the first ninety days.

Weeks 1–2: decide what you are automating and why

No code yet. This fortnight produces a list of user journeys ranked by frequency of use and cost of failure, and a written statement of what the suite is for.

That statement matters more than it sounds. "Block a merge that breaks checkout" and "produce a nightly report on overall product health" are different systems with different speed requirements and different tolerances for flakiness. Teams that skip this build something that half-serves both.

A suite that gates merges must be fast and near-zero-flake. A suite that reports nightly can be slower and broader. Deciding which you are building is the first architectural decision.

Also settled here: which layer each check belongs to. Anything verifiable through an API should be tested through the API — it is faster and dramatically more stable. Reserve browser-level tests for things that genuinely require a browser.

Weeks 3–4: framework architecture, before any test cases

This is the part that record-and-playback skips and the reason those suites collapse. What gets built:

• Page objects or an equivalent abstraction, so a changed selector is fixed in one place rather than forty.
• A test-data strategy. Tests must create the state they need and clean up after themselves. Suites that depend on a shared, hand-maintained database are the single most common source of flakiness we are called in to fix.
• Environment configuration, so the same suite runs against local, staging and CI without edited code.
• Isolation and parallelism from the start. Retrofitting parallel execution onto tests that assume they are alone means rewriting them.
• Reporting that says what broke and shows a trace or screenshot, not just a red mark.

At the end of this period there may be only one or two actual tests. That is the correct amount. The framework is the deliverable; tests are what it makes cheap.

Weeks 5–8: the critical path, and CI gating

Now the highest-value journeys get automated — typically authentication first, then the primary revenue or conversion flow. Each one is written, run repeatedly to prove it is stable, and only then considered done. A test that passes once is not finished.

The important milestone in this window is not test count. It is wiring the suite into the pipeline so it blocks a merge. Until that happens the suite is advisory, and advisory suites decay — a failure that stops nothing gets ignored on a busy day, and then permanently.

A flake policy is agreed here too, and it needs to be uncomfortable to be effective: a test that fails intermittently is quarantined immediately and either fixed or deleted within a fixed window. Tolerating a flaky test is how a team learns to disbelieve the whole suite.

Weeks 9–12: breadth, and handover

With the critical path protected and gating in place, coverage extends outward to secondary flows. Execution time is watched closely — a gating suite that grows past roughly ten minutes starts changing developer behaviour, and not for the better.

This is also when handover happens, which should mean:

1. 01Written documentation covering how to run the suite, how to add a test, and how the data strategy works.
2. 02At least one working session where your engineers add a test themselves while we watch rather than help.
3. 03The suite running in your CI, in your repository, under your ownership.

If an automation engagement ends and only the supplier can extend the suite, the engagement failed regardless of coverage numbers. You have rented a capability rather than acquired one.

Four things not to do

• Do not set a coverage percentage as the goal. It drives volume of low-value tests. Rank by cost of failure instead.
• Do not automate through the browser what an API call can verify. It is slower and far more fragile.
• Do not let the suite depend on a hand-maintained shared dataset. Someone will change it and nobody will know why the suite went red.
• Do not automate a flow whose intended behaviour is still being argued about. Wait for it to settle; you are not ready.

What ninety days realistically produces

A framework your team owns and can extend, the critical journeys covered and gating every merge, regression feedback in minutes rather than a manual pass measured in days, and documentation good enough that the next engineer does not need us.

What it does not produce is total coverage. Anyone promising that in a quarter is either automating trivia or planning to hand you something you cannot maintain.