Test Levels and Test Types

Levels and types

"Did you test it?" is not a useful question until you know what was tested and for what. Testers use two separate ideas to say that precisely.

A test level is a group of test activities aimed at one stage of the system: a single component, components working together, the complete system, or the system being accepted by the people who will use it. Each level has its own goal, its own test basis (the documents or knowledge that say what correct behavior is), its own typical defects and, usually, its own people and environment.

A test type is a group of test activities aimed at one quality characteristic: does it calculate the right price (functional), is it fast enough (performance), can a screen reader user operate it (accessibility). Any type can be tested at any level.

The ISTQB (International Software Testing Qualifications Board) Foundation Level syllabus names five levels. Many teams use four, with a single "integration" level; ISTQB splits it in two because integrating components inside one system and integrating with other systems are different jobs.

Component testing

Component testing, usually called unit testing, checks one component on its own: a function, a class, a module. The site defines the unit test on the programming pages. Developers write unit tests with the code, often before it (see test-driven development), and run them on every change. Because they need no database, network or browser, thousands run in seconds.

For the cost page the smallest units are pure calculations, such as the cost of one reading:

from decimal import Decimal, ROUND_HALF_UP


def energy_cost(kwh, price_per_kwh):
    """Cost in euros of one reading, rounded to the cent."""
    cost = Decimal(str(kwh)) * price_per_kwh
    return cost.quantize(Decimal("0.01"), rounding=ROUND_HALF_UP)


def test_night_reading():
    assert energy_cost(1.5, Decimal("0.2240")) == Decimal("0.34")


def test_day_reading():
    assert energy_cost(1.5, Decimal("0.3120")) == Decimal("0.47")


def test_no_consumption_costs_nothing():
    assert energy_cost(0, Decimal("0.3120")) == Decimal("0.00")

...                                                                      [100%]
3 passed in 0.16s

A unit test finds defects in logic: a wrong formula, a wrong rounding rule, a > that should be >=. It cannot find a defect in how components talk to each other, because it never lets them talk. How to write unit tests in Python is the subject of unit testing with pytest.

Component integration testing

Component integration testing checks the interfaces and interactions between components of one system: does the validator hand its result to the repository in the form the repository expects, does the SQL (Structured Query Language) the repository writes match the real table? It is one kind of integration test. The test below lets two real components work together, with an in-memory SQLite database standing in for PostgreSQL:

import sqlite3

import pytest


def check_reading(kwh):
    """Component 1: the validator. Returns a list of problems."""
    problems = []
    if kwh < 0:
        problems.append("negative")
    if kwh > 25:
        problems.append("too high")
    return problems


class ReadingRepository:
    """Component 2: stores readings in a database."""

    def __init__(self, conn):
        self.conn = conn
        conn.execute(
            "CREATE TABLE readings (meter_id TEXT, kwh REAL, status TEXT)"
        )

    def save(self, meter_id, kwh, status):
        self.conn.execute(
            "INSERT INTO readings VALUES (?, ?, ?)", (meter_id, kwh, status)
        )

    def count(self, status):
        sql = "SELECT COUNT(*) FROM readings WHERE status = ?"
        return self.conn.execute(sql, (status,)).fetchone()[0]


def receive(meter_id, kwh, repo):
    """The integration: validate, then store with the right status."""
    status = "rejected" if check_reading(kwh) else "validated"
    repo.save(meter_id, kwh, status)
    return status


@pytest.fixture
def repo():
    conn = sqlite3.connect(":memory:")
    yield ReadingRepository(conn)
    conn.close()


def test_valid_and_invalid_readings_are_stored_apart(repo):
    receive("M-30001", 0.25, repo)
    receive("M-30002", -4.21, repo)
    receive("M-30003", 91.3, repo)
    assert repo.count("validated") == 1
    assert repo.count("rejected") == 2

.                                                                        [100%]
1 passed in 0.16s

The repo fixture gives each test a fresh, empty database and closes it afterward. SQLite is quick and needs no server, but it is not the database Kestrel runs: column types, date functions and locking differ. Whether that difference matters enough to test against a real PostgreSQL is discussed on integration and contract testing.

System testing

System testing checks the complete, integrated system against its requirements: the whole portal, deployed as it will run in production, used through its real interfaces. It looks at end-to-end tasks ("a customer logs in, opens the cost page and sees September split into day and night") and at qualities that only exist for the whole system, such as response time and security.

System tests need a test environment that resembles production as closely as is affordable: the same operating system and versions, realistic configuration and realistic (but never real) customer data. Kestrel's environments follow the release process described in release and deployment management. Automated system tests that drive the user interface are end-to-end tests; the page on end-to-end and UI testing shows how to write them.

System integration testing

System integration testing checks the interfaces between the system and other systems: the portal and meter-api, the nightly KBS (Kestrel Billing System) export, an external payment provider. These tests are hard to automate fully, because the other systems belong to other teams or companies and may not have a test version. Common answers are shared test environments, simulated services and contract tests.

Acceptance testing

Acceptance testing checks whether the system is ready to be used: does it meet the needs of the business and its users, so that it can be accepted and released? It is mainly validation, and it is done by or with the people who will use or own the system. Its goal is confidence, not finding as many defects as possible; serious defects found here mean the earlier levels missed something. ISTQB lists several forms:

• User acceptance testing (UAT): intended users check that they can do their work with it. For the cost page, Product Owner Isabel Santos and Lucía Romero from billing take 20 test customers and compare the portal's monthly costs with the invoices KBS calculates for the same readings. Every difference of even one cent is explained or fixed.
• Operational acceptance testing: the people who will run the system check that they can. Sanne de Vries, the portal's service owner, wants to see that the cost page is monitored, that backups and restores work, and that the runbook tells the service desk what to do when the page shows no data.
• Contractual and regulatory acceptance testing: checks against a contract or a law, for example the information an energy bill must show, or the accessibility requirements in accessibility testing.
• Alpha and beta testing: real users try the software before general release, at the supplier's site (alpha) or in their own environment (beta). A pilot of the cost page for Kestrel employees who are also customers would be an alpha test.

Acceptance criteria written with the user story, before development, turn acceptance testing from a final hurdle into a shared target; see acceptance criteria. In IT service management, the same idea at release level is service validation and testing.

Functional and non-functional testing

Functional testing checks what the system does: the functions it must perform and the results they give. Is the 23:15 reading priced at the night rate? Does the cost page add up to the monthly total? Functional tests are written from requirements, user stories and business rules, and every example on this page so far is one.

Non-functional testing checks how well the system does it: speed, capacity, security, usability, reliability and similar quality attributes. The usual checklist is the product quality model of ISO/IEC 25010 (2023 edition). Its first characteristic, functional suitability, is the target of functional testing; the other eight are non-functional:

Non-functional requirements are easy to forget because nobody asks for them until they fail: no user story says "the page must not take eight seconds". Write them down with numbers, so they can be tested. A non-functional test often needs a special environment (a load generator, a screen reader, a security scanner) and is usually run less often than functional tests.

Black-box and white-box testing

ISTQB counts two more test types, named after how the tests are designed rather than what they check: black-box testing derives tests from the specification without looking at the code, and white-box testing derives them from the code's structure, for example to make every branch run. Both are explained, with their techniques, on test design techniques.

Testing after a change

Every change, whether a bug fix, a new feature, a library update or a configuration change, raises two questions: did the change do what it should, and did it break anything else? Three kinds of testing answer them.

Confirmation and regression testing

Confirmation testing (sometimes called re-testing) runs again the test that revealed a defect, after the fix, to confirm that the failure no longer happens. If the defect was found by hand, the confirmation test is the steps from the bug report.

Regression testing checks that a change has not broken anything that worked before: that it caused no regression. Because a change in one place can affect code far away, a regression suite covers much more than the changed code, and because it runs after every change, it is the first candidate for automation. When the team fixed BUG-1041, the test that exposed the bug first served as the confirmation test, and then stayed in the suite as one more regression test:

from datetime import time


def is_night(at):
    """Night rate applies before 07:00 and from 23:00 (fixed for BUG-1041)."""
    return at.hour < 7 or at.hour >= 23


# Confirmation test: the exact case from the bug report, now passing.
# From now on it also guards against the bug coming back.
def test_bug_1041_quarter_past_eleven_is_night():
    assert is_night(time(23, 15))


# Regression tests: behavior that worked before the fix must still work.
def test_early_morning_is_night():
    assert is_night(time(2, 30))


def test_seven_oclock_is_day():
    assert not is_night(time(7, 0))


def test_afternoon_is_day():
    assert not is_night(time(14, 0))


def test_one_minute_before_eleven_is_day():
    assert not is_night(time(22, 59))

============================= test session starts ==============================
collecting ... collected 5 items

test_example.py::test_bug_1041_quarter_past_eleven_is_night PASSED       [ 20%]
test_example.py::test_early_morning_is_night PASSED                      [ 40%]
test_example.py::test_seven_oclock_is_day PASSED                         [ 60%]
test_example.py::test_afternoon_is_day PASSED                            [ 80%]
test_example.py::test_one_minute_before_eleven_is_day PASSED             [100%]

============================== 5 passed in 0.16s ===============================

The last test was added with the fix: a fix for 23:00 must not move the problem to 22:59. Naming the test after the bug number makes the history easy to find later. Over time the regression suite grows with every bug and every feature, so it needs maintenance: remove duplicates, keep it fast, and choose which tests to run for which change by looking at what the change can affect (impact analysis).

Smoke tests

A smoke test is a small, fast set of tests of the most important functions, run to decide whether a build or a deployment is worth testing further, or worth keeping. It answers "is it basically alive?", not "is it correct?". The name is often explained with hardware: switch the new device on and see whether smoke comes out. That story is folklore, but the idea fits.

After every deployment of the portal, the pipeline runs a short script against the new version. If any check fails, the deployment is rolled back before customers notice:

#!/usr/bin/env bash
# smoke.sh: run after every deploy of the portal; stops at the first failure
set -euo pipefail
BASE_URL=${1:?usage: smoke.sh BASE_URL}

check() {
  local path=$1 expected=$2 status
  status=$(curl -s -o /dev/null -w '%{http_code}' --max-time 10 "$BASE_URL$path" || true)
  if [[ $status == "$expected" ]]; then
    echo "ok    $path ($status)"
  else
    echo "FAIL  $path: expected $expected, got $status"
    exit 1
  fi
}

check /health 200
check /login 200
check /costs 302    # not logged in: redirected to the login page

$ ./smoke.sh https://portal.kestrel.example.com
ok    /health (200)
ok    /login (200)
ok    /costs (302)

A real smoke test for the cost page also logs in as a dedicated test customer and opens one cost page, which needs a test account and a secret kept in the pipeline. The same checks, run every few minutes in production, become synthetic monitoring (see testing in production). Smoke tests are not a replacement for the regression suite: they are the gate in front of it.