Test data management: five strategies and what each one costs

Ask a team why their automated suite is unreliable and the answer will usually be about the framework. It is usually about the data. How a test gets hold of the state it needs is the decision that determines whether the suite can run in parallel, whether it can run twice in a row, and whether a failure means anything — and it is normally made in the first fortnight by whoever wrote the first test, without anyone treating it as a decision.

There are five workable strategies. Most products need two of them at different layers, and the structural choice between them is very hard to revisit once a few hundred tests have been written against the wrong one.

What a good strategy has to deliver

Three properties, and they are more demanding than they look. A test must be able to run on its own, without any other test having run first. It must produce the same result when run twice in succession. And it must tolerate a copy of itself executing at the same moment against the same environment, because that is what parallelism means once a suite is sharded.

The honest measure of a data strategy is whether one test can run concurrently with itself against the same environment and both copies pass.

Almost every strategy below satisfies the first two. The third is what separates them.

One: a shared seeded dataset

A known set of accounts and records is loaded into the environment, and tests refer to them by name. It is the fastest thing to build and by far the most common thing we are asked to repair.

It degrades in a predictable sequence. A test mutates a record another test reads. Someone edits a row by hand to unblock a demonstration. The seed script and the actual environment diverge, and nobody can say when. Tests acquire an implicit execution order that nothing enforces and nobody documented, so the day the runner shards them the suite falls apart in a way that looks like a tooling problem.

It is legitimate for genuinely immutable reference data: currency codes, tax bands, country lists, a product catalogue nothing writes to. The rule is that a seeded row is only safe if no test can change it, and that constraint should be enforced by permissions rather than by good manners.

Two: each test creates what it needs

The test builds its own account, its own order, its own document, usually by calling the same API a real integrator would call, and disposes of it afterwards. This is the default we recommend, and it is not free.

Three things have to exist for it to work:

• Builders with sensible defaults, so a test that only cares about a customer’s country does not have to specify twenty other fields. Without them, every schema change edits four hundred test files.
• Uniqueness derived from the run rather than from a counter. Email addresses, references and slugs need a component unique per worker and per execution, otherwise two shards collide the first time they run at the same second.
• Teardown that executes even when the test fails halfway through. Cleanup written as the last line of a test body is cleanup that skips precisely when it is needed, and a fortnight later the environment holds ninety thousand orphaned records.

The unadvertised benefit is that the creation path itself ends up exercised on every run, which surfaces a category of defect that browser tests reach only by accident.

Three: an isolated tenant per run

Where the product is already multi-tenant, the cheapest isolation available is the one the application enforces for its customers. Each run provisions an organisation, does everything inside it, and discards it at the end. For multi-tenant platforms this removes an entire class of interference for very little engineering.

It is not total isolation, and teams are routinely surprised by what leaks across the boundary: feature flags evaluated globally, rate limits counted per address, a shared search index, background jobs queued in one worker pool, and anything cached without a tenant in the key. Those are worth enumerating before the strategy is adopted, because each of them reappears later as an unexplained flake.

Four: snapshot and restore

Wrap each test in a transaction and roll it back, or restore a database image between runs. At the unit and integration layers this is excellent: it is fast, the isolation is genuine, and there is no cleanup code to forget.

It stops being viable the moment the system under test is larger than one database. Rolling back a transaction does not un-send a message from a queue, un-index a document, un-write an object into storage or un-invalidate a cache, and a rollback that only covers the relational store leaves the rest of the system holding state from a test that officially never happened. Treat this strategy as belonging below the service boundary rather than across it.

Five: stub the source

For data that arrives from somewhere you do not control — a payment provider, a credit reference, a partner feed — the answer is usually to intercept the call and return a fixed response. It makes tests deterministic and stops a supplier’s sandbox outage from turning your pipeline red.

The cost is drift. A stub records what the third party did on the day it was written, and it will keep confirming that forever. The mitigation is not to abandon stubbing but to run a small, separate set of checks against the real sandbox on a schedule, accept that those may be unreliable, and keep them well away from anything that blocks a merge.

The problems nobody puts in the estimate

• Data with a checksum. Tax numbers, bank identifiers and national insurance formats will not accept a random string, so a valid-value generator is needed before the first test can be written, not after.
• Records that age. A subscription seeded as expiring in thirty days is a test that starts failing in a month, and diagnosing it is unreasonably hard because nothing changed.
• Volume. Pagination, sorting and performance behaviour cannot be exercised with three rows, and generating a realistic tail is a separate piece of work from generating a valid record.
• Ownership. A seed script with no named owner rots. Someone should be responsible for it in the same way someone is responsible for a migration.

How to combine them

The layered answer is close to universal. Unit and integration tests get transactional rollback. Service-level tests create and destroy their own records through the API. Browser journeys reuse those same builders rather than clicking through the interface to arrange preconditions, because arranging state through the UI is slow and makes every setup step a potential cause of failure in a test about something else. Reference data is seeded once and made read-only. Third parties are stubbed by default.

One question sits underneath all of this and is worth settling before any of it: whether the data is personal. Realistic data with real people in it changes the strategy from an engineering choice into an obligation with a regulator attached, and the answer is much cheaper to arrive at now than after a database has already been copied.