Testing and Quality in Pipelines

Why pipelines test

A CI/CD (continuous integration and continuous delivery) pipeline is only as useful as the checks it runs. Building the code proves that it compiles or installs; it says nothing about whether it works. The checks on this page are what turn "the build is green" into a statement you can trust. They do two jobs:

• Fast feedback. The developer who pushed a change learns within minutes that it broke something, while the change is still fresh in their mind and small enough to fix easily. A bug found a month later, in production, costs far more: someone has to find it, understand code they did not write, and repair data along the way.
• A safety net. Tests make it safe to change code. When the team refactors a function, upgrades a library or merges three branches on the same day, the pipeline checks that everything that used to work still works. Without that net, people stop improving code because they are afraid to touch it.

The key property is that the pipeline runs the same checks, the same way, on every change. A colleague who "ran the tests locally" may have skipped the slow ones, used a different Python version, or had an extra file lying around. The pipeline starts from a clean checkout each time, so its answer does not depend on anyone's laptop.

The test pyramid

Tests differ in how much of the system they exercise. The more they cover in one go, the more realistic they are, but also the slower, more expensive and more fragile. The test pyramid is a rule of thumb for the mix, popularized by Mike Cohn (Succeeding with Agile, 2009): a wide base of fast, isolated unit tests, a smaller middle layer of integration tests, and a narrow top of end-to-end tests.

The pyramid is a guideline, not a law. Some teams prefer a "trophy" with a thicker integration layer, because for a web service that mostly moves data between a database and an API (application programming interface), integration tests catch the bugs that matter. What every variant agrees on is the anti-pattern: an "ice-cream cone" with hundreds of slow end-to-end tests and almost no unit tests, which makes the pipeline slow and every failure hard to locate. These shapes, and how to choose one for a team, are compared on Test Strategy and the Test Pyramid.

Unit tests

A unit test checks one small piece of code, typically a function or a class, in isolation: no database, no network, no files. That is what makes it fast (thousands run in seconds) and precise: when one fails, it points at a single function. The programming page explains how to write them with unittest or pytest; in a pipeline they run first, on every push.

Integration tests

An integration test checks that several parts work together through their real interfaces: your code against a real database, two services talking over HTTP (Hypertext Transfer Protocol), a script against the real file layout. Unit tests replace those neighbors with stand-ins (test doubles), so they cannot catch a wrong SQL (Structured Query Language) column name or a changed API field; integration tests can. The price is setup: the pipeline must start the database or service first. In GitHub Actions, service containers do this: the job declares, for example, a PostgreSQL container, and the runner starts it next to the job (see the workflow below). Real databases in tests with Testcontainers: Integration and Contract Testing.

End-to-end tests

An end-to-end test (E2E test) drives the whole system the way a user would: a browser-automation tool such as Playwright, Cypress or Selenium opens the web shop, logs in, puts an item in the cart and checks the order confirmation. For a network team, the equivalent is a test that brings up a lab topology and checks that a host in one VLAN (virtual local area network) can reach a server in another. E2E tests catch what nothing else catches (a broken login page, a missing firewall rule), but they are slow, need a complete environment, and break for reasons unrelated to the change. Keep them few and aimed at the paths that must never break.

Flaky tests

A flaky test passes and fails on the same code without any change: green on one run, red on the next. Common causes:

• Timing: a fixed sleep(2) that is usually long enough, a race between two threads, a page that sometimes loads slowly.
• Order dependence: one test leaves data behind that another test relies on, so the result depends on which runs first.
• Shared or external state: a real network service, a shared test database, the current date or time zone, random data without a fixed seed.
• Resource limits: the runner has fewer processor (CPU) cores or less memory than a developer laptop.

Flaky tests are dangerous because they teach people to ignore red. Once "just re-run it" becomes the habit, a real failure gets re-run too, and the safety net is gone. Treat a flaky test as a bug: record it, quarantine it (move it to a separate, non-blocking job or mark it so a failure does not block merges) so the main pipeline stays trustworthy, and fix the cause with a deadline. Automatic retries can hide flakiness for a while, but a test that needs three attempts to pass is telling you something. Causes, reruns and quarantine in a large suite are covered on Flaky Tests.

Linting and formatting

The cheapest checks do not run the code at all. A linter reads source files and reports likely bugs and style problems: an unused import, a variable that may be undefined, a comparison that is always true. A formatter rewrites code into one consistent layout, so reviews are about content and not about spaces. In a pipeline, the formatter runs in "check" mode: it fails if a file is not formatted, but does not change anything.

Run the same tools locally (many teams use a pre-commit hook) so the pipeline rarely has to tell you. Pin the tool versions in a requirements file or lock file: a new linter release with a new rule should arrive as a deliberate pull request, not as a surprise red build on Monday morning.

Code coverage

Code coverage measures which parts of the code ran while the tests ran, usually as a percentage of lines (line coverage) or of decision outcomes (branch coverage: did each if go both ways?). Tools such as coverage.py (with the pytest plugin pytest-cov) do this by asking the Python interpreter to report every line it executes. The example below does the same thing by hand, for one function and one test, so you can see what the number means:

import dis
import inspect
import sys


def shipping_cost(weight_kg, express):
    if weight_kg <= 0:
        raise ValueError("weight must be positive")
    cost = 4.95
    if weight_kg > 10:
        cost += 7.50
    if express:
        cost *= 2
    return round(cost, 2)


executed = set()


def tracer(frame, event, arg):
    # Called by the interpreter: once per function call, then once per line
    if frame.f_code is shipping_cost.__code__ and event == "line":
        executed.add(frame.f_lineno)
    return tracer


sys.settrace(tracer)
assert shipping_cost(2, express=False) == 4.95  # the only test
sys.settrace(None)

code = shipping_cost.__code__
body = {line for _, line in dis.findlinestarts(code)
        if line is not None and line != code.co_firstlineno}
source, first = inspect.getsourcelines(shipping_cost)
print(f"Line coverage: {len(executed)} of {len(body)} lines "
      f"({len(executed) / len(body):.0%})")
print("Never ran:")
for line in sorted(body - executed):
    print(f"  line {line}: {source[line - first].strip()}")

Line coverage: 5 of 8 lines (62%)
Never ran:
  line 8: raise ValueError("weight must be positive")
  line 11: cost += 7.50
  line 13: cost *= 2

The test passed, and the report says exactly what it did not exercise: invalid weights, heavy parcels and express delivery. That is the real value of coverage: it points at untested code, so you know where tests are missing. What it does not measure:

• Whether anything was checked. A test that calls shipping_cost(20, True) without an assert would push coverage to 88% and verify nothing.
• Combinations. 100% line coverage can still miss the case "heavy and express", where each line ran in some test but never together.
• Missing code. If the function forgets to handle a weight given in grams, there is no line to be uncovered.

In practice, a pipeline runs pytest --cov=app --cov-report=term-missing and can fail the build below a threshold with --cov-fail-under=80. A sensible threshold stops coverage from quietly dropping; chasing 100% tends to produce tests that execute code without checking it. Many teams look at coverage of the changed lines in a pull request rather than the whole project. Branch coverage with coverage.py, and mutation testing, which checks whether the tests would notice a wrong line, are on Code Coverage and Mutation Testing.

Security testing

Security checks are just more checks, and they belong in the same pipeline, on every change. This is often called "shifting left": finding a vulnerability while it is a line in a pull request instead of a finding in a penetration test months later. The main kinds:

SAST: scanning your own code

Static application security testing (SAST) analyzes source code without running it, looking for patterns that lead to vulnerabilities: user input that flows into an SQL query or a shell command, a disabled certificate check, a hard-coded key. CodeQL is GitHub's own engine (it powers GitHub code scanning); Semgrep is a widely used alternative with rules you can read and write yourself, and language-specific tools exist too, such as Bandit for Python. SAST produces false positives, so findings need triage; it is good at the classic injection bugs described on the web application security page.

SCA: scanning your dependencies

Most of the code you ship was written by someone else. Software composition analysis (SCA), or dependency scanning, lists the third-party packages your project uses (direct and indirect) and compares their versions with databases of known vulnerabilities, each usually identified by a CVE number. Examples: Dependabot alerts and security updates on GitHub, pip-audit for Python, npm audit for JavaScript, and OSV-Scanner, which uses the open OSV (Open Source Vulnerabilities) database. SCA tools also report licenses, which matters when a dependency's license does not fit how you distribute your software.

Secret scanning

Secret scanning looks for passwords, API keys and tokens in commits. On GitHub it runs on the platform itself for public repositories (private organization repositories need GitHub Secret Protection), and push protection can stop the push before the secret ever reaches the server; see Securing Repositories. A pipeline step adds little on top of that, but tools such as gitleaks can scan a repository's history in other environments. If a scan does find a real secret: rotate it first, because it has already been exposed.

Container image scanning and DAST

Container image scanning (with tools such as Trivy or Grype) applies SCA to a built container image: it checks the operating-system packages and libraries inside every layer, including those in the base image you did not choose package by package. DAST (dynamic application security testing) works from the outside: it runs against a deployed test instance, sends crafted requests the way an attacker would and looks at the responses. ZAP is a well-known open-source DAST tool. Because DAST needs a running application, it usually runs later in the pipeline, against a staging environment, and often on a schedule instead of on every push.

Quality gates

A check that nobody has to pass is a suggestion. A quality gate is a rule that decides whether a change may move on: "all tests pass", "no new high-severity findings", "coverage does not drop below 80%". In a pipeline, the mechanism is simple: a step that finds a problem exits with a non-zero exit status, the job fails, and everything that depends on it does not run.

On GitHub, the gate in front of the main branch is a required status check, set in a branch protection rule or ruleset: the pull request cannot be merged until the named checks have succeeded (see Pull Requests and Code Review). Two details matter in practice:

• The required check is identified by the job's name as it appears on the pull request, for example lint or test (3.12) for a matrix job. Renaming a job silently breaks the rule; the merge then waits for a check that never reports.
• Tools that report findings (code scanning, coverage services) can post their own check, so "no new high-severity alerts" can be required too.

A pipeline in miniature

Every CI platform builds on the same idea as this small shell script: run the stages in order and stop at the first failure. set -e makes Bash exit as soon as a command fails (see safer scripts), which is how a failing stage stops the rest. The "lint" stage looks for a forbidden pattern (a debugging print(), the "test" stage runs an assertion:

cat > prices.py <<'EOF'
def with_vat(amount):
    return round(amount * 1.21, 2)
EOF

cat > pipeline.sh <<'EOF'
set -e
echo "== lint"
if grep -n 'print(' prices.py; then
    echo "lint failed: remove debugging print() calls"
    exit 1
fi
echo "== test"
python3 -c "from prices import with_vat; assert with_vat(100) == 121.0, with_vat(100)"
echo "== package"
tar -cf release.tar prices.py
echo "pipeline passed"
EOF

bash pipeline.sh
echo "---- someone changes the VAT rate by mistake ----"
sed -i 's/1.21/1.2/' prices.py
bash pipeline.sh

== lint
== test
== package
pipeline passed
---- someone changes the VAT rate by mistake ----
== lint
== test
Traceback (most recent call last):
  File "<string>", line 1, in <module>
AssertionError: 120.0

The second run never reaches "== package": the test stage failed, Python exited with a non-zero status, and set -e stopped the script. The script's own exit status is non-zero too, which is what a CI platform reads to mark the job as failed and keep the gate closed.

A complete GitHub Actions workflow

The workflow below puts the page together for a small Python service (the syntax itself is explained on the GitHub Actions page). It lints first, then runs unit and integration tests and a dependency scan in parallel:

# .github/workflows/ci.yml
name: ci

on:
  pull_request:
  push:
    branches: [main]
  merge_group:            # only needed if the repository uses a merge queue

permissions:
  contents: read          # least privilege for GITHUB_TOKEN in every job

concurrency:
  group: ci-${{ github.ref }}
  cancel-in-progress: true   # a new push cancels the run for the old commit

jobs:
  lint:
    runs-on: ubuntu-24.04
    timeout-minutes: 10
    steps:
      - uses: actions/checkout@v7
        with:
          persist-credentials: false
      - uses: actions/setup-python@v7
        with:
          python-version: "3.12"
          cache: pip
      - run: pip install -r requirements-dev.txt   # pinned tool versions
      - run: ruff check .
      - run: ruff format --check .
      - run: yamllint .

  test:
    needs: lint
    runs-on: ubuntu-24.04
    timeout-minutes: 15
    strategy:
      matrix:
        python-version: ["3.12", "3.13"]
    services:
      postgres:
        image: postgres:16
        env:
          POSTGRES_HOST_AUTH_METHOD: trust   # throwaway test database, no password needed
        ports:
          - "5432:5432"
        options: --health-cmd pg_isready --health-interval 5s --health-retries 10
    steps:
      - uses: actions/checkout@v7
        with:
          persist-credentials: false
      - uses: actions/setup-python@v7
        with:
          python-version: ${{ matrix.python-version }}
          cache: pip
      - run: pip install -r requirements-dev.txt
      - name: Unit tests with coverage
        run: pytest tests/unit --cov=app --cov-report=term-missing --cov-fail-under=80
      - name: Integration tests
        run: pytest tests/integration
        env:
          DATABASE_URL: postgresql://postgres@localhost:5432/postgres

  scan:
    needs: lint
    runs-on: ubuntu-24.04
    timeout-minutes: 10
    steps:
      - uses: actions/checkout@v7
        with:
          persist-credentials: false
      - uses: actions/setup-python@v7
        with:
          python-version: "3.12"
          cache: pip
      - run: pip install -r requirements-dev.txt
      - name: Dependency scan (SCA)
        run: pip-audit -r requirements.txt

• permissions: contents: read at the top limits the automatic GITHUB_TOKEN to reading the repository; no job here needs more.
• persist-credentials: false stops actions/checkout from storing that token on the runner for later Git commands, where later steps (and the tools they install) could read it.
• needs: lint makes test and scan wait for the cheap check, then run side by side. The matrix runs the tests once per Python version, so this workflow reports four checks: lint, test (3.12), test (3.13) and scan. Those are the names to make required.
• The postgres service container gives the integration tests a real database. Its only setting is a trust mode that is acceptable for a disposable container on the runner and nowhere else. Real credentials belong in encrypted secrets, never in the file.
• SAST is not in this file: GitHub's CodeQL code scanning can be switched on in the repository's security settings ("default setup"; free for public repositories, while private organization repositories need GitHub Code Security) or added as its own workflow.
• The action versions (@v7) are the current major versions as of 2026. A major-version tag can be moved by the action's owner; for actions from outside GitHub's own actions/ organization, pin a full commit SHA (the Secure Hash Algorithm hash that names a commit), as below.

# Pattern for a third-party action (illustrative owner, name and SHA):
- uses: example-org/report-action@0123456789abcdef0123456789abcdef01234567  # v2.4.1

A full 40-character commit SHA cannot be re-pointed to other code, unlike a tag; the comment keeps the version readable, and Dependabot can update both together. Why this matters, with the March 2025 tj-actions/changed-files incident as the example, is covered on the GitHub Actions page.

Keeping pipelines fast

A pipeline that takes 40 minutes changes behavior: people batch up changes, stop waiting for results, and merge on red "because it was probably flaky". Aim for minutes. The usual levers:

• Fail fast, then go wide. Run the cheap checks (lint, formatting) first, then fan out into parallel jobs. Jobs without needs run at the same time on separate runners.
• Cache dependencies. Downloading the same packages on every run is pure waste. actions/setup-python and actions/setup-node have a built-in cache input; actions/cache caches any folder by a key such as the hash of a lock file. A cache is a speed-up, never a source of truth: the job must still work when the cache is empty.
• Split slow test suites. A matrix or a test runner's own sharding option runs one large suite in several parallel parts.
• Run only what changed. A paths filter on the trigger skips a workflow when no relevant file changed (documentation-only edits do not need the integration tests). Be careful with required checks: GitHub's documentation warns that a workflow skipped by a path filter leaves its checks pending, which can block the merge. Common answers are to filter inside the job instead, or to make only an always-running summary job required.
• Cancel superseded runs. The concurrency block with cancel-in-progress: true stops the run for an older commit when a newer one is pushed to the same branch.
• Move slow checks off the critical path. Full end-to-end suites, DAST and long performance tests can run nightly or before a release, as long as someone reads the results.

on:
  pull_request:
    paths:
      - "app/**"
      - "tests/**"
      - "requirements*.txt"