Test Strategy and the Test Pyramid

What a test strategy is

Exhaustive testing is impossible (one of the seven testing principles), so every team chooses what to test and what to leave. A test strategy makes that choice explicit: a short description of how a team or organization tests its software. It answers questions such as:

• Which risks matter most, and how much testing does each deserve?
• At which test levels do we test what, and which tests are automated?
• Which non-functional qualities do we test (performance, security, accessibility), and how?
• Which environments and test data do we use?
• Who does what, and how do we decide that something is ready to release?

A good strategy is short enough that the whole team knows it, and specific enough to settle arguments: "do we need an end-to-end test for this?" should have an answer that follows from the strategy, not from whoever argues longest.

Test strategy and test plan

A test plan applies the strategy to one specific project, release or feature: its scope and objectives, the approach, the schedule and people, the environments, the risks, and the entry and exit criteria. The strategy changes rarely; a plan is written for each piece of work. In an agile team the plan is often light: the acceptance criteria of the stories, a few lines in the Sprint Backlog and the Definition of Done together do the job of a separate document.

The international standard ISO/IEC/IEEE 29119 (part 3, 2021 edition) describes test documents such as a test policy, organizational test practices (called the organizational test strategy in the 2013 edition), test plans and test completion reports. Regulated industries, such as medical devices or aviation, often need that level of documentation. The standard is not universally liked: in 2014 testing associations including the Association for Software Testing called for its withdrawal, arguing that it was documentation-heavy and at odds with context-driven testing. Use it as a checklist of things to think about, not as a template to fill.

The test pyramid, revisited

The test pyramid is the best-known picture of a test strategy: many fast unit tests at the bottom, fewer tests in the middle, a few slow tests through the user interface at the top. The Git & CI/CD pages explain the basics and how a pipeline runs each layer. Three details are worth knowing before you use the picture to make decisions.

• Its age and its layers. Mike Cohn popularized the pyramid in his book Succeeding with Agile (2009), but he drew it years earlier, around 2003–2004, according to Martin Fowler's account. Cohn's layers were unit, service and UI (user interface): "service" meant testing through the application's API (application programming interface) below the screens, not "integration" in today's sense.
• Its reasoning. Tests higher up are slower, more expensive to write and maintain, more often flaky, and point less precisely at the cause of a failure. Ham Vocke's article "The Practical Test Pyramid" (2018) sums up the advice: "Push your tests as far down the test pyramid as you can." If a unit test can find a defect, do not use a UI test to find it.
• Its limits. The pyramid says nothing about exploratory testing, non-functional testing or production. It was drawn for applications with a lot of logic of their own. For software that mostly connects other systems, a different shape may fit better.

Other shapes: trophy, honeycomb and cone

Several alternatives to the pyramid have become well known. They are all ways of saying where most of the testing effort should go.

The testing trophy

The testing trophy was drawn by Kent C. Dodds in 2018, building on Guillermo Rauch's 2016 advice "Write tests. Not too many. Mostly integration." From the bottom up it has static analysis (type checkers and linters, which catch typos and type errors for almost nothing), unit tests, a thick layer of integration tests, and a small top of end-to-end tests. The argument: in a web application most bugs are in how parts fit together, and integration tests give the most confidence per test. Modern tools have also made integration tests much faster than they were when the pyramid was drawn.

The testing honeycomb

Spotify's engineers described a similar shape for microservices in 2018, the testing honeycomb: few tests of the implementation details inside a service, most tests on the integration of a service with its direct neighbors (its database, the APIs it calls), and few "integrated" tests that need many services running at once. A small service has little logic of its own, so there is little to unit-test.

The ice-cream cone

The ice-cream cone is the pyramid upside down, a name credited to Alister Scott: a scoop of manual testing on top, many automated tests through the user interface, some integration tests and hardly any unit tests. It grows by accident, when a team automates what testers used to click through by hand. The result is a suite that takes hours, fails for reasons that have nothing to do with the change, and points at a screen instead of a line of code. Every shape agrees that this one is a problem.

Choosing a shape

The shapes disagree less than they seem. All of them say: most tests should be fast and reliable, and only a few should need the whole system. They differ on where "most" is, and the right answer depends on where your defects come from.

The Kestrel portal shows both cases in one product. Pricing rules, rounding and the night-hour rule are pure logic, full of boundaries and easy to get wrong: they deserve many unit tests. Most of the rest of the portal moves data: read readings from meter-api, read contracts from the database, show them. Bugs there are wrong field names, missing data and changed API responses, which unit tests with stand-ins cannot see. So the team settled on a mix:

• Static analysis on every commit (the portal is PHP; see static analysis).
• Many unit tests for pricing and validation logic.
• Integration tests for every page's data access and for the meter-api client, against a real database and a contract with meter-api (see integration and contract testing).
• A handful of end-to-end tests for the journeys that must never break: log in, see this month's costs, download an invoice.
• Exploratory sessions for each new feature, and monitoring in production.

Risk-based testing

A shape says how to test; it does not say what deserves the most attention. In risk-based testing, product risks decide that: the team lists what could go wrong, estimates for each risk how likely it is and how much harm it would do, and spends testing effort in proportion. The same likelihood and impact reasoning is used in security, and a risk matrix shows it at a glance.

In refinement for PBI-450 ("Show this month's costs split by day and night"), the Portal team scored the risks of the cost page on a scale of 1 to 3 and sorted them:

# Product risks for PBI-450, scored 1 (low) to 3 (high).
risks = [
    ("Slow cost page at the evening peak", 2, 2),
    ("Wrong rate at the 07:00 and 23:00 boundaries", 3, 3),
    ("Hourly prices missing for dynamic contracts", 2, 3),
    ("Wrong daily total on 25 October (100 quarter-hours)", 3, 3),
    ("Chart unusable with a screen reader", 2, 3),
    ("Typo in the page footer", 2, 1),
]


def depth(score):
    """How much testing a risk score deserves."""
    if score >= 7:
        return "extensive"
    if score >= 4:
        return "thorough"
    return "light"


ranked = sorted(risks, key=lambda risk: risk[1] * risk[2], reverse=True)
print(f"{'Risk':52} {'L':>2} {'I':>2} {'Score':>5}  Testing")
for name, likelihood, impact in ranked:
    score = likelihood * impact
    print(f"{name:52} {likelihood:>2} {impact:>2} {score:>5}  {depth(score)}")

Risk                                                  L  I Score  Testing
Wrong rate at the 07:00 and 23:00 boundaries          3  3     9  extensive
Wrong daily total on 25 October (100 quarter-hours)   3  3     9  extensive
Hourly prices missing for dynamic contracts           2  3     6  thorough
Chart unusable with a screen reader                   2  3     6  thorough
Slow cost page at the evening peak                    2  2     4  thorough
Typo in the page footer                               2  1     2  light

The scores are judgments, not measurements, and that is fine: the value is in the conversation. Billing errors have a high impact because they reach every affected customer's bill and Kestrel's reputation; anything involving time is likely to go wrong because time rules have many edge cases (and defects cluster together). "Extensive" here means boundary tests at unit level, a check against KBS in acceptance testing and an exploratory session; "light" means a glance during review.

Risk analysis also decides the order: test the highest risks first, so that if time runs out, what is left untested matters least. And it is repeated: when INC-22047 on 6 October 2026 showed that a meter-api release could leak database connections under real traffic and slow the portal down, "slow cost page at the evening peak" deserved a second look before the winter (see performance and load testing).

Shift-left and shift-right

Draw the software lifecycle as a line from idea (left) to production (right). Traditional testing sat in one block near the right, after development. Two movements have spread it out in both directions.

Shift-left testing moves testing and quality activities as early as possible: reviewing requirements and user stories, discussing examples before coding (see the three amigos on quality in agile teams), writing tests with or before the code, running static analysis in the editor and unit tests on every commit. Defects found early are cheaper to fix and do not block anyone downstream. Shift-left does not mean that later testing can be dropped; it means fewer surprises reach it. (In IT service management, shift-left means something else: moving support work to a lower support tier.)

Shift-right testing continues testing after release, in production, where the real data, real load and real users are: a canary release to a small group first, feature flags to switch a feature off without a deployment, synthetic checks that log in every few minutes, and monitoring of errors and response times. Some problems only appear in production, such as a meter vendor that sends readings a little differently, so the question is not whether to test there, but how to do it safely. The page on testing in production goes into the details.

Entry and exit criteria

Testing could go on forever, so a strategy needs a stop rule. Entry criteria say when testing can start (the build installs, the test environment has data). More important are exit criteria: the conditions, agreed in advance, under which testing is done enough to release. Typical exit criteria:

• All planned tests have run, and the highest risks are covered.
• No open defects of severity critical or major (or each one has an agreed workaround and the Product Owner accepted it).
• The automated regression suite is green, without retries.
• Non-functional targets are met, such as 99% of cost pages within 300 ms in the load test.

Agreeing on them beforehand matters, because at the end of a sprint or before a deadline everybody wants to release. Exit criteria turn "are we done?" into a check instead of a debate. In Scrum the Definition of Done plays this role for each increment, and a pipeline's quality gate enforces the automated part. When an exit criterion is not met, releasing anyway is a business decision to take openly, with the risk written down, not something that happens by default.

A test strategy on one page

The Portal team wrote its strategy for the cost page on one page, kept next to the code in the repository so that it changes with the code:

Test strategy: portal cost page (PBI-450)            Portal team, 7 Oct 2026

Highest risks   wrong price at 07:00/23:00; wrong total on DST days (92/100
                quarter-hours); missing dynamic prices; screen reader access
Unit            every pricing and validation rule, with boundary values;
                runs on every commit, under 30 seconds
Integration     data access against PostgreSQL; meter-api client against
                the agreed contract; runs on every pull request
End-to-end      3 journeys: log in, this month's costs, download invoice
Acceptance      Isabel Santos and Lucía Romero: 20 test customers, portal
                costs equal KBS invoices to the cent
Non-functional  load test before the winter peak (p95 < 300 ms);
                axe check and keyboard check per page; security tests per
                release
Exploratory     one 90-minute session per new feature, notes in the story
Production      canary for each release; synthetic login + cost page check
                every 5 minutes
Test data       synthetic customers only, never copies of production
Exit criteria   all of the above green; no open critical or major bugs;
                Product Owner accepts open minor bugs

Notice what is missing: tool names, test counts and a coverage percentage. Tools change; the decisions above do not. Each line links to a page in this topic, and the team reviews the page in a retrospective when an escaped defect shows that the strategy had a gap.