Regression Testing Strategy for Weekly Releases

Weekly releases break regression testing in a specific way. A full regression pass that takes three days was fine when releases were monthly. Run it weekly and it eats 60 percent of the cycle, so teams quietly stop running all of it, then stop knowing which parts they skipped, then get surprised in production.

The answer is not a faster version of the full pass. It is a different shape of regression altogether, built around the fact that you now ship every week and most of what shipped last week is still fine.

This is the strategy we put in place for teams moving to a weekly cadence. It assumes you have some automation already. If you are starting from zero, our best practices for automated testing is the place to begin.

The principle: regression is tiered, not total

A single regression suite that either runs or does not is the wrong unit. Split it into tiers by how often each runs and how long each is allowed to take.

Tier 0, smoke. Does the product start and can a user complete the one journey that pays the bills? Login, the core action, checkout or equivalent. Under five minutes. Runs on every commit to the main branch and after every deploy to every environment, including production.

Tier 1, core regression. The critical paths and the areas that have broken before. Under thirty minutes wall-clock, parallelised. Runs on every pull request before merge. This is the tier that decides whether a change gets in.

Tier 2, extended regression. Everything else that is automated: edge cases, secondary flows, cross-browser, integration with third parties. Up to a few hours. Runs nightly and on the release candidate.

Tier 3, exploratory. Human testing of the change itself. Not scripted, time-boxed, targeted at what changed this week and what it touches. Runs once per release, on the candidate, by someone who did not build the feature.

Everything in your current regression suite belongs in exactly one tier. Sorting it is the first week's work and it is mostly a tagging exercise.

Deciding what goes in Tier 1

Tier 1 is the tier that matters, because it is the gate. Too small and it does not catch anything. Too large and it gets slow, people stop waiting for it, and the gate becomes decorative.

Three questions pick the contents:

1. If this broke, would we roll back? If yes, it is Tier 1. If you would ship anyway and fix forward, it is Tier 2.
2. Has this area produced a production defect in the last six months? If yes, Tier 1 until it has gone six months clean. Your defect tracker is the best regression risk model you have.
3. Did this change in the last two releases? Recently changed code is where defects live. Promote tests for recently changed areas into Tier 1 for a few weeks, then demote them.

That third rule is the one most teams miss. The Tier 1 suite should move. A test that was essential in March because the payment flow was rewritten does not need to block every merge in September.

Keep the gate under thirty minutes

The thirty-minute budget is not arbitrary. It is roughly the longest a developer will wait for a result before switching to something else, and once they switch, a red build gets noticed hours later instead of minutes.

Ways to hold the line:

• Parallelise early. A suite of 640 tests across three browsers runs in two and a half minutes on twelve workers. Serial, it is over an hour. Most frameworks make sharding a one-line change.
• Run API tests before UI tests. An API suite covers more logic per second than any browser test. Put the fast tier first so the pipeline fails fast.
• Cap test duration. Any single test over sixty seconds is a candidate for Tier 2 or for a rewrite.
• Set a budget alert. When Tier 1 passes 25 minutes, someone owns bringing it back down that sprint. Treat suite time as a non-functional requirement with a threshold.

Quarantine flakes, do not retry them

A weekly cadence has no room for a suite that is red for reasons nobody trusts. The first time a developer merges past a red build because "that test is always flaky", the gate is gone.

The rule we use: a test that fails and then passes on retry, twice in a week, is moved to a quarantine tag automatically. Quarantined tests still run but do not block. Someone owns the quarantine list and it must be shorter at the end of each sprint than at the start. Tests that cannot be stabilised in two sprints are deleted. A deleted test is honest. A flaky test is a lie about coverage.

Retrying on failure hides this. Use retries to classify, never to pass.

The weekly rhythm

Here is what the calendar looks like for a team releasing every Thursday.

Monday. Sprint work. Tier 0 on every commit, Tier 1 on every PR. Nightly Tier 2 runs overnight and its failures are triaged first thing Tuesday.

Tuesday. Same. QA pairs with developers on the features landing this week, writing the exploratory charter for Thursday while the feature is still fresh.

Wednesday. Code freeze for this week's release, usually midday. Release candidate is cut. Full Tier 2 runs against the candidate on the staging environment. Exploratory session (Tier 3) in the afternoon, two hours, time-boxed, against the candidate.

Thursday morning. Go or no-go. The decision is made against written exit criteria: Tier 1 and Tier 2 green on the candidate, quarantine list reviewed, no open severity-1 or severity-2 defects from the exploratory session. If the criteria are met, deploy. Tier 0 runs against production immediately after.

Thursday afternoon and Friday. Monitoring, hotfixes if needed, and the regression suite review: what escaped, which tier should have caught it, and what to promote or demote for next week.

The review is the part that makes the system improve. Every production defect gets one question: which tier should have caught this? The answer either adds a test, promotes an existing one, or reveals an area with no automation at all.

What to measure

Four numbers, reviewed weekly, on a dashboard everyone can see.

• Escape rate. Defects found in production divided by all defects found. The number this whole strategy exists to push down.
• Tier 1 duration. Must stay under budget. Rising duration is the early warning that the gate is about to be ignored.
• Flake rate. Share of Tier 1 runs that needed a retry to go green. Above 5 percent, stop adding tests and fix the suite.
• Time to green after a red. How long a broken main branch stays broken. This is a team behaviour metric disguised as a test metric.

If you want to see what the escapes are costing in money rather than counts, the bug cost calculator will convert your escape rate into an annual figure. It tends to be a persuasive number in the conversation about whether to invest in Tier 1.

We built Watchtower because keeping these four numbers visible across tiers and runs turned out to be the hard part of the whole system. Whatever you use, make them visible.

Where teams get stuck

No environment for the candidate. Tier 2 runs against staging, staging is shared, and somebody deployed a half-finished feature to it on Wednesday afternoon. The fix is an environment per release candidate, which is easier than it sounds on most modern platforms. Until then, lock staging from the Wednesday freeze.

Exploratory testing gets skipped. It is the only tier that is not automated, so it is the first thing dropped when Wednesday is busy. Protect it in the calendar. It is also the tier that finds the defects automation cannot, which is most of the embarrassing ones.

The suite belongs to one person. If one QA engineer owns regression, the strategy dies when they are on leave. Tiers, tags and exit criteria all need to live in the repository and the pipeline, not in someone's head.

If you are moving to weekly releases and the regression suite is the thing in the way, book a 30-minute QA audit. We will look at your current suite and sketch the tiers with you on the call.