Regression testing

Regression testing is verification aimed backwards. Rather than confirming that a new feature does what it was asked to do, it confirms that everything which worked before the change still works after it. The name refers to the failure being hunted: a regression is functionality that has gone backwards, usually as an indirect consequence of an edit somebody made in a place they believed was unrelated to it. The distinction is worth holding onto because the two activities are budgeted differently: testing a new feature scales with what was built this month, while regression scales with everything the team has ever shipped.

The commercial question is never whether to do this but how much of it to do, because the population of things that used to work only ever grows. Covering all of it before every release eventually becomes the constraint on how often a team can release at all. Selecting from it by risk keeps the cost roughly flat as the product grows, at the price of admitting in advance that some areas will go unchecked, which is a decision far better made deliberately than by running out of afternoon.

How the scope gets chosen

• By blast radius. What does the changed code touch, what shares its data, and what calls into it from somewhere nobody has read in a year.
• By history. Areas that have broken before tend to break again, and the defect record is a better guide to fragility than the architecture diagram is.
• By consequence. Anything that takes money, writes to a system of record or sends a message to a customer deserves more attention than a preferences screen.
• By suitability for automation. The stable repetitive part belongs in a suite running on every commit, while anything still changing shape stays with a person until it settles.

A suite that only ever grows will eventually cost more to maintain than the defects it catches are worth, so retiring cases has to be somebody’s explicit job.